Validation of short-term climate prediction 4.5
4.5.1 Validation of the autoregressive models
The group of autoregressive techniques consists of the classical data-driven AR- and ARIMA- models and the ARIMAex- model, which uses external regressors from GCMs and/or SSTs.
Validation of the AR- and ARIMA- models 4.5.1.1
The orders of the optimal AR- models for the different climate series and stations in the study region and the two verification periods were listed earlier in Table 4.5. Figure 4.5 shows the continuous forecasts at climate station 48459 for the 14 years 1986-1999, following the vrf1- calibration/verification scheme, i.e. calibration during 1971-1985. The two NS- coefficients listed at the top of the charts indicate the predicting performance of the short-term 12-months-ahead forecast and of the longer-term, 1986-1999 (14-years-ahead) forecast. These NS-values demonstrate clearly that the predicting performances of the former are naturally all better than of the latter. The courses of the various AR-forecast curves show also that while these are more or less able to simulate the seasonal fluctuation of the observed series, the cyclic pattern of the forecast-series get continuously compressed, while the confidence intervals are getting wider with increasing forecasting time, i.e. as the prediction time is moving more and more into the future.
The forecasts with the seasonal optimal ARIMA(p,d,q)(P,D,Q)-models, whose structures for the corresponding time series have been given in Table 4.7 and Table 4.8, are done in a similar manner. They are shown in Figure 4.6. Similar to the AR-forecast above, the performances of the ARIMA- temperature forecasts get worse, when forecasting over a longer term (14 years) than for the short term (12 months). However, this does not hold for the rainfall forecasting which is even better in the long- than in the short run, i.e. the 1-year forecast for 1986.
From Figure 4.5 and Figure 4.6 no clear conclusions can be drawn as to whether the ARIMA- is better than the AR- model. Normally one would expect that a pure autoregressive AR-model can be improved by adding a moving-average (MA) term in the long-term forecasting and also that a seasonal ARIMA-model mimics the seasonal fluctuations better than a pure AR-model.
However, because of the rather high complexity of the AR-models used here, with 12, respective, 24 coefficients, it may be that such an unusual high-degree model has the relatively large prediction power, as experienced in the present application.
The average performances – as measured by the NS- coefficients - of all optimal AR- and ARIMA-models in calibration and verification (one-year-ahead) of the monthly temperatures- and precipitation time-series at all climate stations for both the vrf1- and the vrf2- calibration/verification schemes are listed in Table 4.14. One can notice that the ARIMA-models have the best performance for the short-term (one-year-ahead) forecast in year 1986 (vrf1) with NS>0.5 which is considered a satisfactory level (Moriasi et al. 2007). However, for the vrf2- calibration/verification scheme, the AR-models work consistently better than ARIMA, although for the maximum temperature both model exhibit unacceptable performances (NS<0).
In any case, the predicting skills in the table exhibit that the forecasts for year 2000 are less reliable than those for year 1986, regardless of whether an AR- or an ARIMA- model is used.
One may speculate about the reason, but it is most likely due to some anomalous climate behavior that appears to have occurred between 1996 and 2003 in the study region, as may also be recognized from the climate time series fluctuations of Figure 2.10.
Figure 4.5. AR- model forecasts of monthly a) maximum and b) minimum temperature and c) rainfall in years 1986 to 1999 at station 48459, following the vrf1- calibration/verification scheme. Prediction intervals at the 80% and 95% - confidence levels are exhibited by orange and yellow shades. NS- coefficients at the top of the charts indicate the predicting performance of the short-term 12-months-ahead forecast and of the longer-term 1986-1999 (14-years-ahead) forecast.
a) AR (24) in predicting Tmax at station 48459 NS=0.552 (12 months), 0.185 (14 years)
b) AR (12) in predicting Tmin at station 48459 NS=0.745 (12 months), 0.317 (14 years)
c) AR (24) in predicting PCP at station 459201 NS=0.453 (12 months), 0.414 (14 years) Tmax (˚C) Tmin (˚C) PCP (mm/day)
Figure 4.6. ARIMA-model forecasts of monthly a) maximum and b) minimum temperature and c) rainfall in years 1986 to 1999 at station 48459, following the vrf1- calibration/verification scheme. Other notations are as in Figure 4.5.
a) ARIMA (1,0,1)(1,0,1) in predicting Tmax at station 48459 NS=0.597 (12 months), 0.495 (14 years)
b) ARIMA (1,0,0)(2,0,1) in predicting Tmin at station 48459 NS=0.714 (12 months), -0.378 (14 years)
c) ARIMA (0,0,2)(1,0,2) in predicting PCP at station 459201 NS=0.487 (12 months), 0.552 (14 years)
Tmax (˚C) Tmin (˚C) PCP (mm/day)
Table 4.14. Average performances of the optimal AR- and ARIMA-models in calibration and verification (one-year-ahead) of monthly temperatures- and precipitation time-series at all climate stations for the vrf1- and the vrf2- calibration/verification scheme.
calibration/
verification period predictand
NS-model efficiency coefficient
calibration verification (one year)
AR ARIMA AR ARIMA
cal: 1971-1985 vrf1: 1986
Tmax 0.61 0.51 0.41 0.57
Tmin 0.87 0.84 0.82 0.85
PCP 0.49 0.45 0.51 0.55
cal: 1971-1999 vrf2: 2000
Tmax 0.67 0.65 -1.01 -1.81
Tmin 0.86 0.85 0.69 0.61
PCP 0.49 0.46 0.32 0.27
*best models in each group are highlighted in bold italics
Validation of the ARIMAex- models 4.5.1.2
The optimal ARIMAex models with external regressors, i.e. GCM- predictors (GCMs) and/or ocean state indices (SSTs), set up in a previous section, are validated here by one-year-ahead forecasts of the climate series for the two calibration/verification schemes vrf1 and vrf2.
Although the total number of predictors from the GCMs (ECHO-G and Hi-Res) and the set of lagged ocean indices may exceed 100, only the best predictor out of each regressor- group, as listed in Table 4.9 for the STTs and in Table 4.10 for the GCMs are employed in the following ARIMAex- model validation.
Figure 4.7 and Figure 4.8 show the results of the vrf1- ARIMAex- validation, i.e. forecasts, for the climate station 48459, with ocean indices (SSTs) and GCM- predictors (GCMs) as external regressors, respectively. Similar to the previous AR- and ARIMA- validations (see Figure 4.5 and Figure 4.6), NS- coefficients of the prediction performance are computed and indicated for both the short-term- (12-months) (1986) and the longer term (14-years) (1986-1999) forecasts.
The NS- values listed atop of the panels of Figure 4.7 and Figure 4.8 prove that all ARIMAex-models provide satisfactory predicting skills (NS>0.5) for the short-term- forecast in year 1986.
Even though the performance of the ARIMAex predictions is again getting worse when forecasting over a longer term ahead (14 years), this is less the case than for the previous AR- and ARIMA- validations (see Figure 4.5 and Figure 4.6). This shows the advantage of the permanent updating (nudging) of the forecast-value by the most recent external regressor that can be either a SST-index or a GCM- predictor. For the rainfall forecasts one may also notice in the corresponding panels of Figure 4.7 and Figure 4.8 that the residual errors, i.e. the gaps between the observed and predicted values are usually high at the rainfall peaks.
The best regressors used in the ARIMAex-models in the vrf1- and vrf2- validation-schemes for predicting the 12-month-ahead climate at the various stations in the study region are listed in Table 4.15. The table indicates that for predicting the two temperatures and the precipitation, the air temperature (HiRes.tmp) and precipitation (HiRes.pre), respectively, both from the high-resolution GCM, are the most suitable. Moreover, whereas the temperature predictions are mostly optimal with Hi-Res- regressors, for the precipitation forecasts, ocean indices with lags of -3 to - 6 months are mostly applied.
Figure 4.7. ARIMAex-model forecasts with ocean indices as external regressors to predict monthly a) max and b) min temperatures and c) rainfall at station 48459 in years 1986 to 1999 following the vrf1- calibration/verification scheme. Other notations are as in Figure 4.5.
a) ARIMA(1,0,1)(1,0,1) + Nino3(lag:-6) in predicting Tmax at station 48459 NS=0.659 (12 months), 0.567 (15 years)
b) ARIMA(1,0,1)(1,0,1) + EPO(lag:-1) in predicting Tmin at station 48459 NS=0.876 (12 months), 0.289 (15 years)
c) ARIMA(0,0,0)(1,0,2) + Nino3(lag:-6) in predicting PCP at station 459201 NS=0.502 (12 months), 0.550 (15 years)
Tmax (˚C) Tmin (˚C) PCP (mm/day)
Figure 4.8. ARIMAex models with GCM predictors as external regressors to predict monthly a) max and b) min temperatures and c) rainfall at station 48459 in years 1986 to 1999, following the vrf1 calibration scheme. Other notations are as in Figure 4.5.
a) ARIMA(0,0,1)(1,0,1) + HiRes.tmp in predicting Tmax at station 48459 NS=0.865 (12 months), 0.793 (15 years)
b) ARIMA(1,0,0)(1,0,1) + ECHO-G.rldscs in predicting Tmin at station 48459 NS=0.763 (12 months), 0.335 (15 years)
c) ARIMA(0,0,1)(0,0,0) + Hi-Res.pre in predicting PCP at station 459201 NS=0.658 (12 months), 0.654 (15 years)
Tmax (˚C) Tmin (˚C) PCP (mm/day)
Table 4.15. Best regressors (GCM- predictors and SSTs) used in ARIMAex for predicting 12-month-ahead climate for the vrf1- and vrf2- calibration/verification schemes, with corresponding number of stations.
predictand vrf1: cal 1971-1985/vrf 1986 vrf2: cal 1971-1999/vrf 2000
optimal predictor1 number of
stations2 optimal predictor1 number of stations2
Tmax HiRes.tmp 4 HiRes.tmp 3
HiRes.vap 1
Tmin HiRes.tmp 3 HiRes.tmp 4
ep /lag -1 1
PCP HiRes.cld 1 HiRes.dtr 3
HiRes.dtr 1 HiRes.pre 10
HiRes.pre 7 ECHO-G.rsdt 1
HiRes.tmp 1 ECHO-G.tas 1
HiRes.vap 3 ep /lag -3 1
ECHO-G.rsut 2 setio/lag -4 1
ECHO-G.rtmt 1 swio/lag -3 1
ep /lag -2 2 swio/lag -5 1
ep /lag -6 1 swio/lag -6 1
nino4 /lag -5 1 wp/lag -5 1
nino4 /lag -6 1 wtio/lag -3 1
noi /lag -3 1 wtio/lag -5 1
noi /lag -4 1 wtio/lag -6 1
setio /lag -3 1
1stations associated with an ocean index, with lags as indicated, are highlighted in bold italics
2 number of stations add up to 4 for Tmin and Tmax and to 24 for PCP
Table 4.16. Average performance of ARIMAex-models for predicting the 12-month-ahead monthly climate for the vrf1- and vrf2- calibration/verification schemes, as measured by the mean error (ME), root mean square error (RMSE) and Nash–Sutcliffe model efficiency (NS).
calibration/
verification period predictand calibration verification (one year)
ME RMSE NS ME RMSE NS
cal: 1971-1985 vrf: 1986
Tmax -0.03 0.55 0.80 0.01 0.31 0.80
Tmin -0.04 0.54 0.94 -0.08 0.49 0.95
PCP -0.43 1.93 0.69 -0.27 1.66 0.73
cal: 1971-1999 vrf: 2000
Tmax -0.01 0.51 0.84 -2.94 0.55 0.20
Tmin 0.00 0.50 0.94 0.12 0.49 0.87
PCP -0.43 2.16 0.71 2.92 2.12 0.52
The average performance of all these optimal ARIMAex- models, as measured by the mean error (ME), root mean square error (RMSE) and Nash–Sutcliffe model efficiency (NS), are summarized in Table 4.16. This table indicates that for the three climate variables maximum, minimum temperature and precipitation, the ARIMAex-validations result in NS- coefficients ranging between 0.20 to 0.80, 0.87 to 0.90 and 0.52 to 0.71, respectively. Moreover, the 1986 short-term forecasts are generally better than the year-2000 ones, which hold particularly for the minimum temperature.
Finally, in Table 4.17 the predicting skills of the AR-, ARIMA- and ARIMAex- models for the vrf1- and vrf2- validation schemes are summarized. One may clearly notice that the ARIMAex-models, using either GCMs or SSTs, exhibit higher skills than the AR- and ARIMA- models. In fact, the ARIMAex- models using HiRes- predictors provide the best performances for minimum temperature and rainfall, with NS- values close to 0.5. However, for all three model-variants, the forecast performances are all weak for the maximum temperature for year 2000
(vrf2-scheme). Although the predictions of maximum and minimum temperatures and precipitation are mostly optimal for the ARIMAex+Hi-Res- GCM combination model, second-to-best results are obtained with the ARIMAex+SST- model. For precipitation, the latter model is even better than the former which, in turn, shows the high degree of teleconnections between ocean state indices and the rainfall pattern in the study area.
Table 4.17. Average performance, as measured by the RMSE and the NS, of the various autoregressive models (AR, ARIMA, ARIMAex) in predicting 12-month monthly temperature and precipitation time-series at all climate sites for the vrf1- and vrf2- validation schemes.
predictand model
vrf1
cal 1971-1985/vrf 1986 vrf2
cal 1971-1999/vrf 2000 calibration verification calibration verification RMSE NS RMSE NS RMSE NS RMSE NS Tmax
(°C)
AR 0.67 0.61 0.54 0.41 0.68 0.67 0.89 -1.01
ARIMA 0.71 0.5
1 0.47 0.57 0.68 0.6
5 1.00 -1.81 ARIMAex+SSTs 0.62 0.80 0.44 0.62 0.58 0.84 0.76 -0.47 ARIMAex+ECHO
-G
0.68 0.5
5 0.42 0.66 0.67 0.6
6 0.76 -0.44 ARIMAex+HiRes 0.55 0.70 0.31 0.80 0.52 0.79 0.58 0.06 Tmin
(°C)
AR 0.79 0.87 0.92 0.82 0.78 0.86 0.76 0.69
ARIMA 0.81 0.8
4 0.83 0.85 0.78 0.8
5 0.86 0.61
ARIMAex+SSTs 0.76 0.92 0.64 0.91 0.65 0.93 0.70 0.74 ARIMAex+ECHO
-G
0.78 0.8
5 0.79 0.87 0.73 0.8
7 0.76 0.70
ARIMAex+HiRes 0.54 0.93 0.58 0.92 0.50 0.94 0.49 0.87 PCP
(mm/day)
AR 2.42 0.49 2.17 0.51 2.56 0.49 2.54 0.32
ARIMA 2.37 0.4
5 2.07 0.55 2.57 0.4
6 2.58 0.27
ARIMAex+SSTs 2.31 0.68 1.79 0.68 2.35 0.71 2.31 0.44 ARIMAex+ECHO
-G
2.32 0.4
8 1.92 0.61 2.47 0.5
0 2.40 0.36
ARIMAex+HiRes 1.93 0.64 1.78 0.65 2.17 0.61 2.23 0.45
*the best models with the highest NS for Tmax, Tmin, and PCP are highlighted in bold italics.