The map shows 2-metre soil moisture trends in Asia during March-April-May for the period 1982-present. Some areas show strong drying e.g. Mongolia, Caspian Sea ~ -10 g m ^-2 y ^-1. The desert provinces of western China appear to be getting wetter, while most of the fertile eastern provinces are getting drier.

Is China drying out?

To see what is happening in China more clearly, soil moisture trends can be summarised in terms of an average for each province (which can be displayed using Google Chart Tools thanks to the googleVis package in R). Mouse-over to see the wetting or drying trend (in g m ^-2 y ^-1) for each province.

The chart below shows the annual data along with the linear regression lines. Changes in soil moisture levels over the past 30 years are comparable to or exceed the annual variability in several cases.

data: http://nomads.ncdc.noaa.gov/data.php#cfsr

David Lobell (Stanford University) and co-workers have related changes in crop yields to observed climatic shifts over the past thirty years. They found that long-term temperature changes are having a greater impact on yields in major agricultural areas than shifts in rainfall patterns. A grain trader might find this fact surprising. There is a well-known rule of thumb that crops benefit from plenty of rain during the vegetative (growth) stages and drier conditions during the fruiting/harvest stages. Low yields or poor quality occur if these conditions are not met. Therefore yield variability is often associated with rainfall variability. The finding by Lobell and co-workers is an example of how short-term variability can mask a longer-term underlying trend.

Nowadays climatic forecasts are made using state-of-the-art physics based models such as CFSv2. Where there is sufficient historical data, these models can also be used to recreate the state of the atmosphere in the past. The maps below show the trends in 2m temperature over land during 1982-2010. Trends were extracted from CFSv2 Reanalysis data using R‘s raster package and plotted using ggplot2 (“trends” means linear regression coefficients although there is markedly non-linearity in many places.)

The upper map is for December-January-February (DJF) and the lower for June-July-August (JJA). Globally, DJF warmed by ~0.36^oC/decade over land while JJA warmed somewhat less ~0.28^oC/decade. Winter in North America and Scandinavia warmed by several degrees during the period 1982-2010, whereas summer warming was much weaker. There are even some places where winter and summer trends are reversed. The US breadbasket states show little warming (even some cooling) during the summer  growing season, but this is far from the case in Central Europe or Russia.

Corresponding maps for rainfall look quite different:

While regional trends can be large ~ ±50mm/decade, there was no significant net wetting or drying trend over land.

The spatial and temporal structure of climatic shifts during 1982-2010 is much more complex than headline global warming numbers suggest. For agriculture, the details matter a great deal.

Blocking and the Arctic Oscillation

Extreme conditions can occur when the normal variable circulation of the atmosphere gets stuck into a stationary “blocking” pattern. A block can extend over 1000km’s and persist for days or weeks. In 2010, the  block which brought a heat wave to Russia also caused floods in Pakistan.

Blocking anti-cyclones bring severe winter cold to NW Europe. Mild moist air is kept away and replaced by cold dry air. Under clear skies and with long winter nights, the land surface radiates away more heat than it receives. Lower and lower temperatures are reached and relief only comes when the blocking pattern breaks down.

The formation of Northern hemisphere winter blocks is associated with the negative phase of the Arctic Oscillation (AO). The AO index describes the strength of a ring or vortex of air which circulates around the pole, shown in (a) below. The origin of this westerly wind or “jet” is the North-South temperature gradient and the rotation of the earth (coriolis force). In winter, when the temperature gradient is large and the AO is positive, the jet helps confine very cold Arctic night air to polar latitudes.

A negative AO index means that the polar vortex is weaker than normal. In this case, the situation shown in (c) is likely to occur. The difference between high and low AO is analogous to an ice-skater who is very stable when spinning fast, but more likely to wobble and fall over when spinning slowly. The earth-scale meanderings of the jet stream are called Rossby waves. Larger amplitude meanderings such as (c) favour the formation of atmospheric blocks, which is why severe winter weather is associated with negative AO.

The above chart  of daily AO index since 1950 shows the recent period of negative index. The chart also shows that Arctic Oscillation is highly variable and irregular. Most of this derives from internal variability of the earth’s climate system. However researchers have known for some time that there is a connection between solar activity and AO e.g. the Little Ice Age (or Maunder Minimum ~1645-1715) was a period low solar activity and severe winters in NW Europe and the US. This has been a puzzle, because the brightness of the sun is nearly constant (variation ~ 0.1%).

Arctic Oscillation and Solar UV

Although only a small part of the sun’s output is in the ultra-violet (UV), variations in this part of the solar spectrum are now known to be fairly large (~8%).The above figure (from the UK Met Office) describes research linking the strength of the polar vortex to solar UV variability.^[1] The proposed mechanism works as follows. Solar UV heats the stratosphere at ~ 25km where most of it is absorbed by ozone. Heating is greatest over the tropics where sunlight is most intense. Lower UV radiation means less heating, which means a lower North-South temperature gradient, which means weaker thermal winds in the stratosphere. If this effect is transmitted down to low levels (troposphere), it might weaken the entire polar vortex and push AO negative.

While this is an area of active research, it is probably a good idea to keep an eye on solar UV. Here is a plot of the total UV radation in the range 115-310nm using satellite data from SORCE. (This R script will download the latest data from SORCE and reproduce the plot.)

While total solar radiation has picked up this year, UV radiation is still weak. If the new research is right, the odds are shifted in favour of another severe winter.

[1] Solar forcing of winter climate variability in the Northern Hemisphere,  S. Ineson, A.Scaife, J. Knight, J. Manners, N. Dunstone, L. Gray, J. Haigh Nature Geoscience 4, 753–757 (2011)

The map uses an equal area projection (Albers) with horizontal and vertical scales in meters.

R aficionados will recognize use of the ggplot2 R package. We are experimenting with ggplot2 for weather map-making at Biospherica™ .

Data from CFSv2 Analysis.

ggplot2 : Elegant Graphics for Data Analysis, Hadley Wickham

Code

Daily 2m temperature maps are available from operational CFSv2 here (tmp2m.l.gdas.201107.grib2). These correspond to initial conditions used by the CFSv2. To find a mean temperature for July, an average was taken over  4xdaily 6h forecast records from this grib2 file. There are probably many good ways to do this.  One way is to first interpolate from gaussian to latlon grid, then use the wgrib2 -netcdf to produce files which can be read directly by R’s raster package. To get a temperature anomaly the monthly 2m temperature climatology needs to be subtracted. This is available here (flxf06.cfsr.fclm.m07.1982.2010.grb2).

The temperature anomaly raster map can be projected from latlon to another PROJ4 map projection using raster::projectRaster. The resulting July global temperature anomaly raster object is called tmp2m.ras. USA country maps (level 0 & 1) are from gadm. These are reprojected and called USA.poly & USA.poly1. Finally, with tmp2m.ras, USA.poly and USA.poly1 objects in R memory,  ggplot2 was used to produce the map as follows:

library(raster)
library(RColorBrewer)
library(ggplot2)
#crop tmp2m.ras
x.min <- bbox(USA.poly)[1,1]
x.max <- bbox(USA.poly)[1,2]
y.min <- bbox(USA.poly)[2,1]
y.max <- bbox(USA.poly)[2,2]
tmp2m.ras <- crop(tmp2m.ras, extent(x.min,x.max,y.min,y.max))
#mask tmp2m.grd using country outline
USA.ras <- rasterize(USA.poly, tmp2m.grd)
tmp2m.ras <- mask(tmp2m.ras,USA.ras)
#convert data to dataframe for ggplot
tmp2m.ps <- rasterToPoints(tmp2m.ras)
df <- data.frame(tmp2m.ps)
colnames(df) <- c("easting","northing","tmp2m")
USA.poly <- fortify(USA.poly,region="ISO")
USA.poly1 <- fortify(USA.poly1,region="ID_1")
#now the pretty stuff
g <- ggplot(df,aes(x=easting,y=northing)) + scale_x_continuous(limits=c(x.min,x.max)) + scale_y_continuous(limits=c(y.min,y.max))
g <- g+geom_tile(aes(fill=tmp2m),alpha=0.6)+scale_fill_gradientn(name=expression(paste(degree,"C",sep="")),colours=rev(brewer.pal(9,"RdYlBu")))
g <- g+coord_equal()
g <- g+stat_contour(aes(z=tmp2m,colour=..level..),size=0.5, bins=4)+scale_colour_gradient(name=expression(paste(degree,"C",sep="")))
#country outline
g <- g + geom_path(data=US.poly, aes(x=long,y=lat,group=group), colour="grey40", alpha=1.0,size=0.5)
#state outline
g <- g + geom_path(data=USA.poly1, aes(x=long,y=lat,group=group), colour="grey80", alpha=1.0,size=0.3)
g <- g + opts(title = "USA Temperature July 2011")
print(g)

For example, suppose you want to know the relation between spatially averaged Irish and Scottish wind speeds. The result from ECMWF reanalysis is shown below. Clearly Scottish and Irish wind speeds are correlated. This might have implications for trading of wind power generation between the UK and Ireland, for example.

This type of analysis is easy to do in R using Robert Hijman’s superb raster package. The first step is to download horizontal(U) and vertical(V) wind speed data from ECMWF Data Server for ERA-interim. To decode this file, the wgrib utility or similar must be installed on your system. The following code creates a raster ”stack” object consisting of 1342 U-component wind speed raster maps for NW Europe: 
library(raster)
region <- extent(-40,40,0,80) #subset selection -40W 40E 0N 80N
Ucomp.stk <- stack()
for (i in seq(1,2*1342,by=2) ){
system(paste("wgrib ERAdailyWind.grb -d ", i, " -text -o temp.txt",sep=""))
temp <- scan("temp.txt",skip=1)
temp <- t(matrix(temp,240,121))
temp.ras <- rotate(raster(temp,xmn=0,xmx=360,ymn=-90,ymx=90))
temp.ras <- crop(temp.ras,region)
Ucomp.stk <- addLayer(Ucomp.stk,temp.ras)
}
....
wind.stk <- calc(calc(Ucomp.stk,function(x) x*x)+calc(Vcomp.stk,function(x) x*x),sqrt)

The last line calculates absolute wind speeds from Ucomp.stk and Vcomp.stk. rotate() optionally shifts to Greenwich centred maps. The code for Vcomp.stk is the same as Ucomp.stk and omitted.

A rasterized map of Scotland can be created from a level 1 GADM SpatialPolygons object as follows:
UK <- getData("GADM",country="GBR",level=1)
scotland <- SpatialPolygons(UK@polygons[3])
scotland.ras <- rasterize(scotland, wind.stk,getCover=T)
scotland.ras <- scotland.ras/cellStats(scotland.ras, sum)
The option getCover=T in rasterize() means that any reanalysis cells which partially overlap Scotland are included with appropriate weight. The average wind speeds for Scotland are then:

scottish.wind.stk <- scotland.ras * wind.stk
scottish.speeds <- cellStats(scottish.wind.stk, sum)


Irish average wind speeds are found in the same way.

Finally the high density scatterplot shown above was produced using the hexbin package:


library(hexbin)
wind.bin <- hexbin(irish.speeds,scottish.speeds)
count.cols <- colorRampPalette(c("azure2","yellow","orange","red"),space="Lab")
plot(bin, colramp=function(n) count.cols(n), main="12hr Wind Speeds 2009-2010",xlab="Ireland m/s",ylab="Scotland m/s",legend=0)


How severe are conditions in 2010 compared to previous Russian droughts and heat waves?

The plot^[2] shows monthly growing season precipitation versus temperature anomalies for wheat croplands 1948-2010. A positive index value indicates high cropland precipitation or temperature. June and July this year are clear outliers.

Gridded 200km climate data (NCEP Reanalysis) were re-projected^[1] to an equal area projection and standardized^[4]. Then a weighted average over the harvested area for the year 2000 (data from C. Monfreda et al) was carried out.

__________________________________________________

[1] USDA World Agriculture Report, August 2010 http://www.fas.usda.gov/psdonline/circulars/production.pdf

[2] made using smoothScatter() base R graphics.

[3] raster map calculations using raster package by Robert J. Hijmans & Jacob van Etten.

[4] Monthly standardized precipitation and standardized temperature indices.

Quantifying the relationship between weather and vegetation on a regional scale is a statistical problem which requires long time-series of NDVI. GIMMS is a 25 year (1981-2006) NDVI dataset based on inputs from a sequence of NOAA weather satellites carrying AVHRR radiometers. The GIMMS team removed satellite drift, calibration effects etc.[1] Of course NDVI has a complex spatial variation which is related to variations in vegetation cover type, not to weather variations. To isolate the impact of weather variations, it makes sense to “standardize” NDVI. Vegetation Condition Index (VCI) is:

For example, NDVI_min might refer to the minimum local May NDVI value observed during 1981-2006. A tropical rainforest and a semi-arid region have very different NDVI, but both may have VCI close to 1 during a favourable period.

The maps below show the correlation between VCI and 3-month Standardized Precipitation Index (SPI3) computed for Africa by calendar month. GIMMS 8km NDVI was upscaled to 200km to match the scale of gridded monthly climate datasets for precipitation (GPCP).[2] The red areas correspond to areas of high correlation between VCI and SPI3. Vegetation in these areas responsive to rainfall variations on a 3-month timescale. A striking feature  is the line high correlation in the Sahel region. Vegetation in the Sahel is highly sensitive to drought. Southern Africa is also highly rainfall sensitive during Dec-May. It is impressive that the combination of datasets of very different origins (GPCP and GIMMS) produces spatially and temporally consistent information.

Sensitivity of Vegetation to Precipitation 1981-2006

R code snippets

Most of the effort is in pre-processing the climate and NDVI data so that they share the same projection, grid, time coordinate etc. GDAL and the interp() function from the akima package were used for this purpose. Eventually we end up with SPI3 and VCI data represented by N × T matrices where N = Nx ×Ny is the dimension of the spatial maps and T is the number of observations. From there,  SPI3 and VCI maps were organized by calendar month,

vci.m <- lapply(seq(1:12),function(m) vci[,seq(from=m, to=T, by=12)])
spi3.m <- lapply(seq(1:12), function(m) spi3[,seq(from=m, to=T, by=12)])

Correlation maps between VCI and SPI3  for each calendar month are obtained from,

corr.maps <- lapply(seq(1:12),function(m) sapply(seq(1:N), function(i) cor( vci.m[[m]][i,],spi3.m[[m]][i,]) ))

Finally the above map table was produced using spplot() from the sp package with overlays from the maps package.

[1]  Tucker, C.J., J.E. Pinzon, and M.E. Brown (2004), Global Inventory Modeling and Mapping Studies, NA94apr15b.n11-VIg, 2.0, Global Land Cover Facility, University of Maryland, College Park, Maryland, 04/15/1994.

[2] SG: Adler, R.F., G.J. Huffman, A. Chang, R. Ferraro, P. Xie, J. Janowiak, B. Rudolf, U. Schneider, S. Curtis, D. Bolvin, A. Gruber, J. Susskind, P. Arkin, 2003: The Version 2 Global Precipitation Climatology Project (GPCP) Monthly Precipitation Analysis (1979-Present). J. Hydrometeor., 4,1147-1167.

Standardized Precipitation Index (SPI) is a widely used measure of drought which can be defined for any time-scale of interest. For any location, SPI is normally distributed with zero mean and unit standard deviation. Index values > 2 indicate exceptionally wet conditions for that location, values < -2 indicate exceptionally dry conditions for that location, etc. Historical precipitation is the only input needed to compute SPI.

Australia experienced drought between 2002 and 2007. The image below shows SPI computed for a location in the drought-prone Murray-Darling basin of New South Wales. The time-series run from Jan 1948 to Jan 2010 and the index was calculated for time-scales from 1 to 12 months. Precipitation data is from NCEP Reanalysis [1] in a 1.875° × 1.875° grid cell centred at 30°S 145°E.

The drought of 2002 to 2007 shows up very clearly. It was preceeded by a wet period between 2005 and 2001. While 2009 showed an episode of severe drought at short time-scales, SPI at was normal/wet at longer time-scales during 2009. Agricultural yields recovered.

Calculating SPI-M

Empirical rainfall probability distributions are far from normal (gaussian) and often approximate a shifted gamma distribution. The empirical cumulative probability distributions are used to transform the rainfall time-series into time-series of percentile probabilities. A normally distributed precipitation index is found by pretending that these percentile probabilities derive from a standard cumulative normal distribution and inverting to find the index values.

This is simple in R. If the vector data contains rainfall infall data, then:

fit.cdf <- ecdf(data)
cdfs <- sapply(data,fit.cdf)
SPI <- qnorm(cdfs)


Tha rainfall data are M-month moving averages (current and previous months). A separate index is calculated for each calendar month to remove seasonality. The R code used to compute SPI values (based in NCEP Reanalysis or other data sets such as GCPC) is here.

[1] The NCEP/NCAR 40-year reanalysis project, Bull. Amer. Meteor. Soc., 77, 437-470, 1996

Noted Added 11 October 2011: I have uploaded a slightly improved SPI R script here. The function getPrecOnTimescale(precipitation,k) takes a vector of monthly precipitation values and returns a k-month average (i.e current month and prior k-1 months). getSPIfromPrec(precip.k) takes k-month precipitation values and returns the corresponding vector of SPI values.

The approach taken by organisations such as ECMWF or NCEP is to re-run numerical forecast models with a range of carefully chosen initial conditions. The collection of runs is called the ensemble. Ensemble prediction systems (EPS) give probabilistic forecasts for variables such as rainfall, temperature etc. Current operational EPS have 20 (GFS)  or 51 (ECMWF) ensemble members from which the probability distributions are derived. ECMWF give an overview of their system here. The probability distributions capture part of the intrinsic uncertainty in weather or climate.

The graph below shows histograms of 20 ensemble member temperatures near some major cities. The data were extracted from NCEP GENS 16-day 2m temperature forecast produced at 00UTC 2 Feb 2010 (i.e GFS forecasts for 18 Feb).

The maps below show some corresponding ensemble statistics for the entire globe (1° resolution, equal area cylindrical projection).

The upper map indicates that forecast uncertainty (standard error) is high between 40° and 60° in both hemispheres (related to the chaotic behaviour of  jet streams.) Currently, 16 day temperatures north of Lake Baikal in Siberia are very uncertain, for example. The contours indicate ensemble median temperatures.

Skewness in ensemble temperatures is shown in the lower map. For example, large negative skewness is found in north central US, eastern mediterranean, and Paraguay/Mato Grosso. This suggests tail risk of low temperatures relative to ensemble mean in these areas.

EPS is the future of weather and climate forecasting. These systems produce huge amounts of data. Building useful applications of EPS is both a challenge and an opportunity.

For anyone interested, the R code used to produce these graphs is given here.

The above wind speed maps are based on NCEP GFS analysis at 300mb pressure level (equivalent to ≈ 10km). The speed scale is in m/s.

There are many interesting things to notice about these maps. Firstly, there are several (5 or 6) planetary scale meanders of the jet stream. The meanders are called planetary or Rossby waves. Secondly, the enclosed area is larger and wind speeds higher in the northern hemisphere. This situation is reversed during the southern winter. Notice the closed loop of clockwise circulating air close to New Zealand. In this case a jet stream meander has grown large, become unstable and broken off. The loop encloses a pocket of cold air. Such detached loops can persist and remain in the same location for days.

The graphic below shows the GFS 300mb analysis wind speeds for ooUTC 19 Jan 2010 and the forecast wind speeds for 00UTC 20 & 21 January.  Rossby wave propogation can be seen clearly (the ridges and troughs advance anticlockwise in the Northern hemisphere, clockwise in the Southern Hemisphere)

code

300mb wind velocities (zonal wind component Uvel and meridonal wind component Vvel) at 300mb were extracted from 0.5° GFS grib files. In an earlier post complete GFS forecasts in grib2 format were downloaded and relevant fields extracted using the wgrib2 utility. In fact, it is possible to download only the required fields. This much faster partial-http option uses cURL and an easy to use Perl script called get_gfs.pl from NCEP.

jet.R is the script which produced the above graphics. Here Roger Bivand’s sp and rgdal packages are used to transform the latitude-longitude GFS projection to  a polar projection. For example, wind speeds in the northern hemisphere are contained in the SpatialGridDataFrame object sg_north. It is transformed into Universal Polar Stereographic (ups in the Proj.4 library) using
sg_north <- spTransform(sg, CRS("+proj=ups +north"))
sg_north@bbox <- polar_north@bbox
Unfortunately spTransform() produces a SpatialPoints object, because the grid is non-uniform after transformation. To recover a SpatialGrid object, the interp() function from package akima was used to resample back onto a regular grid. This is the slowest part of jet.R.