Validation of short-term climate prediction 4.5
4.5.2 Validation of MLR in short-term prediction
(vrf2-scheme). Although the predictions of maximum and minimum temperatures and precipitation are mostly optimal for the ARIMAex+Hi-Res- GCM combination model, second-to-best results are obtained with the ARIMAex+SST- model. For precipitation, the latter model is even better than the former which, in turn, shows the high degree of teleconnections between ocean state indices and the rainfall pattern in the study area.
Table 4.17. Average performance, as measured by the RMSE and the NS, of the various autoregressive models (AR, ARIMA, ARIMAex) in predicting 12-month monthly temperature and precipitation time-series at all climate sites for the vrf1- and vrf2- validation schemes.
predictand model
vrf1
cal 1971-1985/vrf 1986 vrf2
cal 1971-1999/vrf 2000 calibration verification calibration verification RMSE NS RMSE NS RMSE NS RMSE NS Tmax
(°C)
AR 0.67 0.61 0.54 0.41 0.68 0.67 0.89 -1.01
ARIMA 0.71 0.5
1 0.47 0.57 0.68 0.6
5 1.00 -1.81 ARIMAex+SSTs 0.62 0.80 0.44 0.62 0.58 0.84 0.76 -0.47 ARIMAex+ECHO
-G
0.68 0.5
5 0.42 0.66 0.67 0.6
6 0.76 -0.44 ARIMAex+HiRes 0.55 0.70 0.31 0.80 0.52 0.79 0.58 0.06 Tmin
(°C)
AR 0.79 0.87 0.92 0.82 0.78 0.86 0.76 0.69
ARIMA 0.81 0.8
4 0.83 0.85 0.78 0.8
5 0.86 0.61
ARIMAex+SSTs 0.76 0.92 0.64 0.91 0.65 0.93 0.70 0.74 ARIMAex+ECHO
-G
0.78 0.8
5 0.79 0.87 0.73 0.8
7 0.76 0.70
ARIMAex+HiRes 0.54 0.93 0.58 0.92 0.50 0.94 0.49 0.87 PCP
(mm/day)
AR 2.42 0.49 2.17 0.51 2.56 0.49 2.54 0.32
ARIMA 2.37 0.4
5 2.07 0.55 2.57 0.4
6 2.58 0.27
ARIMAex+SSTs 2.31 0.68 1.79 0.68 2.35 0.71 2.31 0.44 ARIMAex+ECHO
-G
2.32 0.4
8 1.92 0.61 2.47 0.5
0 2.40 0.36
ARIMAex+HiRes 1.93 0.64 1.78 0.65 2.17 0.61 2.23 0.45
*the best models with the highest NS for Tmax, Tmin, and PCP are highlighted in bold italics.
Table 4.18. Best predictor-sets used in the MLR- models with number of stations for forecasting the 12-month-ahead climate for the vrf1- and vrf2- calibration/verification schemes.
predictand predictor set1
number of stations2 vrf1
cal 1971-1985, vrf 1986
vrf2
cal 1971-1999, vrf 2000 Tmax
HiRes - 1
GCMs+HiRes - 2
HiRes+SSTs 3 -
GCMs+HiRes+SSTs 1 1
Tmin
HiRes - 1
GCMs+HiRes 1 1
HiRes+SSTs 2 2
SSTs 1 -
PCP
HiRes 2 3
GCMs 5 5
GCMs+HiRes 3 6
HiRes+SSTs 5 6
GCMs+SSTs 1 -
GCMs+HiRes+SSTs 1 -
SSTs 7 4
1models associated with SST- teleconnections are highlighted in bold italics
2 number of stations add up to 4 for Tmin and Tmax and to 24 for PCP
(group (1) in the table), very good calibration and verification performances (NS > 0.5) for predicting the climate in the study area are obtained for both vrf- schemes, albeit with some caveat for the maximum temperature (Tmax) forecasting for year 2000 (vrf2), similar to what has already been found with the autoregressive methods of the previous section. As the table shows, this precarious situation for the year-2000 Tmax-series cannot be remedied by neither the GCMs nor the most complete GCMs + SSTs variants of the MLR- method, all of which indicates some peculiar behavior of the observed Tmax during that time period.
When going further down Table 4.19, one may notice that for the minimum temperature and the precipitation the group (2) of the MLR-model variant, which uses pure GCMs-predictors, namely, from the Hi-Res- and GCMs + Hi-Res - predictor set, results also in good predicting performances for some of the four calibration/verification periods.
Finally, adding the ocean SSTs to the GCM- predictor set, i.e. the use of the full MLR- model (Eq. (4.11)) (group (3) in the table) improves the prediction skills in most of the cases further.
This holds in particular for the most complete GCMs+HiRes+SSTs model variant, although it is not able neither to fix the prediction problem with Tmax for year 2000 (scheme vrf2). In any case, the results of Table 4.19 provide some more evidence for the general, well-known fact that the prediction power of a model increases with increasing complexity of the model.
The optimally validated MLR- models for each climate station in the study region, with individual performance results, similar to that listed in Table 4.19 for the average, and using the appropriate optimal predictor-sets from Table 4.18, are subsequently used to forecast the monthly climate in the study area over the short (1 year) and long term, i.e. 14 years for the calibration/verification scheme vrf2.
The two panels of Figure 4.9 show predicted and observed maximum/minimum and precipitation time series at pilot station(48459), with the corresponding sets of MLR- equations
Table 4.19. Average performances of the various MLR- model variants with combinations of GCM- predictor sets and ocean indices (SSTs) to predict the monthly temperature and precipitation series for the vrf1- and vrf2- calibration/verification-schemes.
model predictand
vrf1
cal 1971-1985 /vrf 1986 vrf2
cal 1971-1999 / vrf 2000 calibration verification calibration Verification RMSE NS* RMSE NS* RMSE NS* RMSE NS*
(1) SSTs
SSTs Tmax (°C) Tmin (°C) 0.64 0.86 0.77 0.91 0.50 0.60 0.68 0.91 0.79 0.89 0.66 0.85 1.16 0.68 -2.54 0.73 PCP (mm/day) 2.21 0.79 1.72 0.65 2.47 0.67 2.36 0.42
(2) GCMs
ECHO-G Tmax (°C) Tmin (°C) 0.81 0.91 0.35 0.80 0.64 0.91 0.17 0.83 0.91 0.88 0.38 0.81 1.02 0.87 -1.74 0.58 PCP (mm/day) 2.47 0.40 2.13 0.48 2.66 0.42 2.59 0.27 GCMs Tmax (°C) Tmin (°C) 0.70 0.80 0.52 0.84 0.54 0.85 0.40 0.85 0.80 0.78 0.52 0.84 0.82 0.78 -0.70 0.68
PCP (mm/day) 2.06 0.58 1.56 0.70 2.15 0.62 2.10 0.55 HiRes Tmax (°C) Tmin (°C) 0.47 0.52 0.76 0.93 0.35 0.45 0.75 0.96 0.61 0.54 0.71 0.93 0.87 0.43 -1.29 0.89
PCP (mm/day) 1.81 0.68 1.69 0.61 2.04 0.66 2.11 0.51
GCMs +HiRes
Tmax (°C) 0.48 0.77 0.37 0.73 0.57 0.75 0.53 0.30 Tmin (°C) 0.52 0.93 0.47 0.95 0.51 0.94 0.37 0.93 PCP (mm/day) 1.71 0.71 1.55 0.65 1.91 0.70 1.95 0.60
(3) GCMs+SSTs
ECHO-G + SST Tmax (°C) Tmin (°C) 0.74 0.88 0.54 0.82 0.52 0.91 0.56 0.83 0.86 0.86 0.52 0.82 1.02 0.87 -1.69 0.58 PCP (mm/day) 2.41 0.44 2.08 0.53 2.57 0.46 2.46 0.36 GCMs + SSTs Tmax (°C) Tmin (°C) 0.69 0.80 0.54 0.85 0.52 0.84 0.46 0.85 0.79 0.77 0.54 0.85 0.83 0.77 -0.78 0.69
PCP (mm/day) 2.04 0.60 1.55 0.71 2.14 0.62 2.09 0.55 HiRes+SSTs Tmax (°C) Tmin (°C) 0.49 0.52 0.82 0.94 0.32 0.43 0.80 0.96 0.64 0.52 0.71 0.93 0.85 0.44 -1.12 0.88
PCP (mm/day) 1.71 0.76 1.56 0.77 1.98 0.70 2.07 0.54 GCMs+HiRes
+SSTs
Tmax (°C) 0.46 0.79 0.33 0.80 0.55 0.76 0.55 0.20 Tmin (°C) 0.51 0.94 0.46 0.96 0.50 0.94 0.41 0.91 PCP (mm/day) 1.70 0.72 1.54 0.65 1.91 0.70 1.95 0.60
*in each calibration/verification column the highest NS for Tmax, Tmin and PCP are highlighted in bold italics
listed at the top of each panel. Note, that the optimal MLR-model structure for the climate series at this particular station may not necessarily agree with the average optimal model structure found in Table 4.19. As indicated in the upper panel of Figure 4.9, the MLR-model applies HiRes+SSTs – and SSTs- predictor sets for predicting the maximum and minimum temperatures, respectively, by using the 3-season scheme, with the corresponding regression equations as listed. For the precipitation prediction (lower panel), on the other hand, the HiRes- predictor-set is used exclusively, in conjunction with the 4-season scheme. The courses of the time-series drawn show that, in general, the predicted climate series fit the observed ones rather well, however, with larger discrepancies usually at the peaks of the climate series.
The average predicting performance of all optimal MLR-models for all climate series in the study region are summarized in Table 4.20. One may notice that for all three climate series and both calibration/verification schemes, the prediction accuracies are rather high (NS > 0.5), with the - by now accepted – exception of the poor forecasts of the maximum temperatures in year 2000 (vrf2), where the NS is only 0.36, i.e. a value which, after all, may still be acceptable.
Figure 4.9. Observed and predicted monthly maximum and minimum temperature (upper panel) and precipitation (lower pane) at climate station 48459 between years 1986 and 1999, for the calibration/verification case vrf1, using different MLR- model variants, as indicated by the corresponding regression predictor equations on top of the charts. For the two temperatures the 3-season- , and for the precipitation the 4-season scheme is used.
Table 4.20. Average performance, as measured by the ME, RMSE and NS, of the optimal MLR-models in predicting the one-year-ahead monthly temperature and precipitation for the vrf1- and vrf2- calibration/verification schemes.
calibration/
verification period predictand calibration verification (one year)
ME RMSE NS ME RMSE NS
cal: 1971-1985 vrf: 1986
Tmax 0.09 0.43 0.87 0.09 0.30 0.84
Tmin 0.04 0.49 0.95 0.01 0.39 0.96
PCP 0.20 1.62 0.82 0.44 1.28 0.84
cal: 1971-1999 vrf: 2000
Tmax 0.02 0.53 0.79 -0.56 0.49 0.36
Tmin 0.01 0.49 0.94 -0.10 0.33 0.94
PCP 0.22 1.88 0.73 0.66 1.78 0.67