Table 4.20. Average performance, as measured by the ME, RMSE and NS, of the optimal MLR-models in predicting the one-year-ahead monthly temperature and precipitation for the vrf1- and vrf2- calibration/verification schemes.
calibration/
verification period predictand calibration verification (one year)
ME RMSE NS ME RMSE NS
cal: 1971-1985 vrf: 1986
Tmax 0.09 0.43 0.87 0.09 0.30 0.84
Tmin 0.04 0.49 0.95 0.01 0.39 0.96
PCP 0.20 1.62 0.82 0.44 1.28 0.84
cal: 1971-1999 vrf: 2000
Tmax 0.02 0.53 0.79 -0.56 0.49 0.36
Tmin 0.01 0.49 0.94 -0.10 0.33 0.94
PCP 0.22 1.88 0.73 0.66 1.78 0.67
Performance comparison of short-term climate predictions
Figure 4.10. Average model performances, as measured by the Nash-Sutcliffe model coefficient, in predicting 12-month monthly maximum (Tmax) and minimum (Tmin) temperatures and precipitation (PCP) for the vrf1- (1986) (a) and vrf2- (2000) (b) calibration/verification schemes.
The MLR- GCM-predictor models in Figure 4.10 were all calibrated based on GCM- predictors of the 20th-century baseline period (20c3m- baseline GCM- climate simulation of Chapter 3), i.e.
these predictors have already been optimized for this reference period (1971-1999). However, starting with year 2000, climate models deliver “future” climate projections, based on some SRES. As the observed climate data in the study region is still available up to year 2006, it is interesting to see, how these “future” predictors are also useful for the short-term climate prediction in years after the 20th–century baseline calibration period, i.e. after 1999. Hence, in the
a) model performance in predicting climate for year 1986 (vrf1)
b) model performance in predicting climate for year 2000 (vrf2)
-0.67 0.17
0.64
0.34
-0.04 0.42
0.34
-0.04 0.50
0.56 0.62
0.50 0.80
0.62
0.38 0.83
0.94
0.78 0.82
0.92
0.83 0.83
0.93 0.95
0.85 0.85
0.92 0.91
0.41
0.48 0.50
0.44
0.56 0.53
0.50
0.57 0.54
0.65
0.52 0.49
0.67 0.68
-1.0 -0.8 -0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 0.8 1.0
LARS-WG SDSM ECHO-G HiRes SSTs GCMs GCMs+HiRes ECHO-G + SST GCMs+SSTs GCMs+HiRes+SSTs HiRes+SSTs AR ARIMA ARIMAex-GCM ARIMAex-SST
conventional MLR multi-domain
MLR multi-domain
MLR+teleconnection Autoregressive
Nash–Sutcliffe model efficiency coefficient
Tmax Tmin PCP
0.20
-0.47
0.53 0.52
0.32
-0.06 0.17
0.42
0.87 0.86
-0.28 0.47
0.66
0.53 0.87
0.74
0.17 0.14
0.42 0.47
0.29
0.22 0.15
0.21
0.46 0.44
-1.0 -0.8 -0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 0.8 1.0
LARS-WG SDSM ECHO-G HiRes SSTs GCMs GCMs+HiRes ECHO-G + SST GCMs+SSTs GCMs+HiRes+SSTs HiRes+SSTs AR ARIMA ARIMAex-GCM ARIMAex-SST
conventional MLR multi-domain
MLR multi-domain
MLR+teleconnection autoregressive
Nash–Sutcliffe model efficiency coefficient
Tmax Tmin PCP
following paragraph, the various short-term forecast methods have been applied to predict the climate one-year ahead for the two “future” years 2001 and 2002, wherefore the calibration of the models is then extended to 1971-2000 and 1971-2001, respectively, and the SRES A2, which turned out to be the most appropriate scenario to describe the recent (2000-2006) climate in the study area (see Section 3.7.1), is used.
Table 4.21 shows the average models prediction performances for the three climate variable for these two years 2001 and 2002. For comparison, the results of the one-year-ahead climate prediction for year 2000, i.e. those corresponding to the calibration/verification scheme vrf2 are relisted again. One can notice from the table that the MRL- or ARIMAex- models which use GCMs- predictors alone have less prediction performances than those with teleconnective ocean (+SSTs) regressors and, moreover, these performances become lower when the prediction time moves further into the “future”, i.e. from years 2000 to 2002.
Meanwhile, the ARIMAex- and the MLR- model associated with SSTs offer stronger performance than the ones with only GCMs- predictors for years 2001 and 2002, i.e. further away from the end of the calibration period in 1999, wherefore ARIMAex works generally better than MLR for the forecasts of the two temperatures, namely, Tmin, while that of Tmax appears to be plagued again with some inconsistencies, likewise to what has already been found earlier for this climate variable.
Table 4.21. Average model performances, as measured by the NS- coefficients of the set of MLR- and autoregressive models with different predictor sets in predicting the one-year-ahead climate in years 2000, 2001 and 2002 by calibration from year 1971 until one year before the beginning of the corresponding prediction period.
NS- coefficient*
predictand
year
MLR- models ARIMAex- models
GCMs SSTs GCMs+SSTs GCMs SSTs
ECHO-G HiRes GCMs SSTs GCMs+HiRes ECHO-G + SST GCMs+SSTs GCMs+HiRes+SS Ts HiRes+SSTs ARIMAex-HiRes ARIMAex-GCMs ARIMAex-SST
Tmax 2000
-0.93 -0.66 -0.53 -0.51 0.03 -0.26 0.01 -4.08 -0.13 0.31 0.15 0.16 2001 -1.89 -1.81 -1.21 -1.29 0.13 -0.85 0.00 -4.08 -1.02 0.06 -0.44 -0.47 2002 -0.38 -0.18 -0.06 -0.06 0.29 -0.04 0.30 -1.56 -0.08 0.39 0.43 0.42
Tmin 2000
0.48 0.48 0.67 0.89 0.93 0.68 0.92 0.74 0.84 0.87 0.70 0.74 2001 0.62 0.63 0.68 0.62 0.64 0.68 0.64 0.50 0.75 0.76 0.77 0.76 2002 0.28 0.28 0.64 0.37 0.48 0.65 0.48 -0.13 0.69 0.67 0.68 0.80
PCP 2000
0.11 0.17 0.45 0.51 0.55 0.46 0.54 0.01 0.56 0.45 0.36 0.44 2001 -0.39 -0.35 0.23 0.09 0.17 0.19 0.17 -4.82 0.42 0.25 0.23 0.36 2002 -1.33 -1.15 0.04 -0.70 -0.67 -0.15 -0.65 -6.78 0.44 0.18 0.20 0.40
*best performance in each year is highlighted in bold italics
In conclusion of this paragraph, the use of GCM-predictors alone in longer-term (several years) climate forecasts is not sufficient, when the target forecast time is too far ahead of the end of the GCM- calibration period, which corresponds to the baseline 20th- century reference period, denoted 20c3m in Chapter 3. Using precursory teleconnective, oceanic SSTs- data in the
-7.00 0.00 1.00
Scale legend
prediction models can then serve as a partial remedy. However, it must be noted here again, that the leading time of the SSTs with regard to the local climate in the study region is only a few months, which means that this prediction approach is only feasible, if ocean-state data arrive in in due time on a continuous basis. In fact, as the monthly ocean indices are archived generally with a delay of one month delay, e.g. January indices are available only in the beginning of February (NOAA Climate Prediction Center (CPC) 2009), the actually forecasted lead (pre-warning) time is also one-month shorter.
4.6.2 Annual and seasonal prediction performances of the optimal models
In this section, the short-term climate prediction skills of the various prediction models are examined in more detail on both the annual and seasonal scales.
Table 4.22 lists the optimal model- (MLR, ARIMAex)/predictor set (GCMs, SSTs) combinations that have been used for the prediction of the individual climate series at the various climate stations in the study region for both the vrf1- and vrf2- calibration/verification schemes. One may notice from the table that for all three climate variables the MLR-models using both GCM- and ocean SSTs- regressors work best for most of the climate stations. And among all available predictor-sets, the most often applied are predictors from the HiRes- GCM model, and Niño4- and WTIO- indices from the SSTs- set .
Table 4.22. Best model/predictor- combinations with corresponding number of climate stations for 12-month-ahead climate forecasting for the vrf1- and vrf2- calibration/verification schemes.
predictand
vrf1
cal 1971-1985 / vfr 1986 vrf2
cal 1971-1999 /vrf 2000 optimal model/ predictor set number of
stations1 optimal model/ predictor set number of stations1
Tmax ARIMAex:HiRes.tmp 2 ARIMAex-HiRes.tmp 1
MLR:GCMs+HiRes+SSTs 1 MLR:GCMs+HiRes 1
MLR:HiRes+SSTs 1 MLR:GCMs+HiRes+SSTs 1
MLR:HiRes 1
Tmin ARIMAex-HiRes.tmp 1 MLR:GCMs+HiRes 1
MLR:GCMs+HiRes 1 MLR:HiRes 1
MLR:HiRes+SSTs 1 MLR:HiRes+SSTs 2
MLR:SSTs 1
PCP ARIMAex:ECHO-G.rtmt 1 ARIMAex-HiRes.dtr 1
ARIMAex-nina4/lag -6 1 ARIMAex:ECHO-G.rsdt 1
MLR:GCMs 4 ARIMAex-wtio/lag -6 1
MLR:GCMs+HiRes 3 MLR:GCMs 5
MLR:GCMs+HiRes+SSTs 1 MLR:GCMs+HiRes 6
MLR:GCMs+SSTs 1 MLR:HiRes 2
MLR:HiRes 2 MLR:HiRes+SSTs 4
MLR:HiRes+SSTs 5 MLR:SSTs 4
MLR:SSTs 6
1number of stations add up to 4 for Tmin and Tmax and to 24 for PCP
Using the corresponding optimal predictor-sets for the ARIMAex- and MLR- models, their average performances in the 12-month-ahead climate forecasts for the vrf1- (1986) and vrf2 (2000) - calibration/verification schemes are computed. The results are listed in Table 4.23, which indicates that for all three climate variables, at least for the averages, the MLR- prediction models work better than the ARIMAex- models.
Table 4.23. Average performances, estimated by RMSE and NS, of the optimal ARIMAex- and MLR- models in predicting the 12-month-ahead climate for the vrf1- (1986) and vrf2 (2000) - calibration/verification schemes.
calibration/
verification
period predictand model performance*
RMSE NS
ARIMAex MLR ARIMAex MLR
cal 1971-1985 vrf 1986
Tmax (°C) 0.31 0.30 0.80 0.84
Tmin (°C) 0.49 0.39 0.95 0.96
PCP (mm/day) 1.66 1.28 0.73 0.84
cal 1971-1999 vrf 2000
Tmax (°C) 0.55 0.49 0.20 0.36
Tmin (°C) 0.49 0.33 0.87 0.94
PCP (mm/day) 2.12 1.78 0.52 0.67
*best model performance for each climate variable is highlighted in bold italics
Table 4.24 shows the average seasonal prediction performances - based on the 4-season scheme of the best ARIMAex- and MLR- models for the vrf1- (1986) and vrf2- (2000) calibration /verification schemes. For comparison, the model results of the best annual forecasts are also listed. Firstly, one may notice from the table that the climate prediction performances of most of the models are well acceptable (NS>0.5) for most of the three climate variables - with the exception of the maximum temperature for prediction year 2000 (vrf2 scheme) - the two verification/forecast periods, particularly, for the annual, the dry- and the pre-monsoon season- forecasts, but, somewhat less, for the two monsoon seasons. In fact, for year 2000 (vrf2) the precipitation forecasts during these two seasons are also not satisfactory.
Secondly, likewise to the optimal annual prediction model, for most of the climate variables, the best seasonal performances are achieved with the MLR- model. However, for the minimum temperature of the first monsoon of year 1986 (vrf1), and the maximum temperature of the pre-monsoon- and the monsoon2 season of year 2000 (vrf2), the ARIMAex- model exhibits the best prediction skills.
Table 4.24. Average annual and seasonal performances, estimated by the NS, of the optimal ARIMAex (ARex) and multi-linear regression (MLR) techniques in predicting the 12-month-ahead (annual) and the separate 3-month-(seasonal)-12-month-ahead climate variables for the vrf1- (1986) and vrf2 (2000) - calibration/verification schemes.
validation
scheme predictand
Nash-Sutcliffe (NS) model coefficient* annual
optimal model
seasonal optimal model
dry pre-monsoon monsoon1 monsoon2 MLR ARex MLR ARex MLR ARex MLR ARex MLR ARex cal: 1971-1985
vrf1: 1986
Tmax 0.84 0.80 0.81 0.51 0.92 0.84 0.69 0.01 0.59 -0.31 Tmin 0.96 0.95 0.99 0.93 0.96 0.93 0.33 0.34 0.84 0.01
PCP 0.84 0.73 0.95 0.87 0.44 -1.61 0.76 0.58 0.74 0.52 cal: 1971-1999
vrf2: 2000
Tmax 0.36 0.20 0.91 -0.60 0.68 0.87 0.65 0.28 0.35 0.46 Tmin 0.94 0.87 0.88 0.72 0.97 0.80 0.71 0.25 0.86 0.79 PCP 0.67 0.52 0.91 0.85 0.29 -2.00 0.27 0.06 -0.03 -1.28
*best models for each climate variable and each season are highlighted in bold italics
Table 4.25 lists the actual number of climate stations with the optimal models, either from the group of MLR- or ARIMAex- models, for both the annual and seasonal predictions for the vrf1- and vrf2- calibration/verifications schemes. Although the climate variables at the majority of
stations are best fitted or forecasted by the MLR- method, for a good number of stations the climate, namely, the precipitation, is better predicted by the ARIMAex- model. However, this observation does not disprove the earlier statements that forecasts of the precipitation in the study region are less reliable than those of the temperatures (see also Table 4.24).
Table 4.25. Number of climate stations with the best models from one of the groups of MLR- and ARIMAex (ARex)- models for the annual and seasonal predictions for the vrf1- and vrf2- calibration/verifications schemes.
calibration/
verification
period predictand
number of stations1,2
annual optimal model
seasonal optimal model
dry pre-monsoon monsoon1 monsoon2 MLR ARex MLR ARex MLR ARex MLR ARex MLR ARex cal: 1971-1985
vrf: 1986
Tmax 2 2 4 4 4 2 2
Tmin 2 2 3 1 4 3 1 2 2
PCP 18 6 18 6 17 7 19 5 18 6
cal: 1971-1999 vrf: 2000
Tmax 3 1 4 3 1 4 1 3
Tmin 4 4 4 4 3 1
PCP 21 3 19 5 15 9 21 3 18 6
1number of stations add up to 4 for Tmin and Tmax and to 24 for PCP for each season
2mostly selected models for each climate variable and season are highlighted in bold italics
To depict the inter-season forecasts in more detail, the 1986- one-year-ahead MLR- and ARIMAex- predicted climate series (vrf1 scheme) at temperature stations 48459 and precipitation station 459201 are exhibited, together with the observed series, in the three panels of Figure 4.11. One can recognize from Figure 4.11a that for the maximum temperature the ARIMAex- method works better than the MLR- model, as the latter underestimates the observed temperatures significantly from the beginning to the end of the monsoon seasons (April-October). The predicted and observed minimum temperature series of Figure 4.11b exhibit that the ARIMAex- forecast works better at the beginning and the MLR- one better for the second half of the year.
Finally, for the precipitation series in Figure 4.11c, the MLR- method provides overall a better forecasting skill than ARIMAex. Nonetheless, MLR can better simulate the first peak of the observed rainfall at the beginning of the first monsoon season, whereas ARIMAex works slightly better for representing the observed rainfall peak of the second monsoon season. These results show that for good seasonal climate forecasting, the selection of an optimal seasonal forecasting model should not only be based on its average annual prediction skill, but should also consider its seasonal performances.