Validation of the downscaling models 3.6
3.6.3 Performance comparison of the various downscaling models
An inter-comparison of downscaling results is encouraged (Mearns et al. 2003) when applying different downscaling approaches. Therefore, in this section, the performance of all downscaling models, i.e., SDSM, LARS-WG, single-domain- and multi-domain MLR models are evaluated and compared.
Performance of all single-domain downscaling models 3.6.3.1
The performance of SDSM, LARS-WG and the single-domain MLR-models which employ daily predictors from a single-domain GCM (ECHO-G) to project the maximum and minimum temperature and rainfall in the study area are comparatively evaluated here. One way to evaluate the skill of daily climate prediction is to see how well the predicted climate series mimics the fluctuations of the observed series. This can be quantified by the computation of the cross-correlation between the two series, as defined in Eq. (2.1). For SDSM and LARS-WG, the average cross-correlation of all realizations is used in the assessment of the model performance.
Figure 3.19 shows these cross-correlations between observed and predicted climate series obtained with the SDSM, LARS-WG and the MLR-daily model for the 1986-1999 verification period. One may notice that the MLR- daily model has overall the best prediction skill, except for the minimum temperature, where LARS-WG works slightly better.
Figure 3.17. Calibration and verification of multi-domain MLR- downscaling model employing multi-domain GCMs (GCMs+HiRes) and single-domain GCM (HiRes) to predict monthly maximum and minimum temperature at station 48478 for years 1971-1999, and using 1971-1985 as calibration time period.
Figure 3.18. Calibration and verification of multi-domain MLR-model employing multi-domain GCMs (GCMs+HiRes) to predict monthly rainfall at station 48092 for years 1971-1999, and using 1971-1985 as calibration time period.
Figure 3.19. Similarity of observed and downscaled daily temperatures (at 4 stations) and precipitation (at 24 stations), as measured by the cross-correlation coefficients for the verification time period 1986-1999, following the calibration time period 1971-1985.
As discussed, a model’s prediction skill can also be evaluated by the Nash-Sutcliffe (NS) coefficient. In Table 3.23 the NS- coefficients obtained with the three downscaling methods are listed, wherefore the estimation has been made separately for the calibration- (1971-1985) and the verification (1986-1999) period. Also, the models are evaluated for the prediction on the daily and on the monthly time scale, whereby in SDSM, LARS-WG and MLR-daily daily ECHO-G predictors (see Table 3.1) are used and, in the case of monthly prediction, their downscaled daily climate series are aggregation to monthly values. Only for MLR-monthly (MLRm), monthly ECHO-G predictors are used directly.
Table 3.23 indicates that for daily downscaling MLR-daily (MLRd) provides the best predicting performance for daily maximum temperature and precipitation, whereas LARS-WG is best for daily minimum temperature. For monthly predictions, MLR-monthly (MLRm) is best in predicting temperatures (Tmin and Tmax) and LARS-WG is best in predicting precipitation.
When considering these performances, despite that the prediction of Tmin by MLR-monthly is satisfied (NS>0.5), the other variables are just only acceptable (NS>0). These outcomes show that SDSM performs rather poorly, whereas the LARS-WG- and MLR-daily/monthly- models deliver an acceptable performance in downscaling climate predictors from a single-domain parent GCM (ECHO-G).
Table 3.23. Performance of daily and monthly climate predictions as measured by the Nash-Sutcliffe (NS) efficiency coefficient in calibration- (1971-1985) and verification mode (1986-1999), using SDSM, LARS-WG and MLR-downscaling models, based on daily and monthly ECHO-G GCM- large scale predictors (see text for further explanations).
type predictand calibration verification
SDSM LARS-WG MLRd MLRm SDSM LARS-WG MLRd MLRm daily
prediction
Tmax -0.20 0.07 0.11 - -0.36 -0.2 -0.09 -
Tmin -0.03 0.53 0.43 - -0.33 0.40 0.30 -
PCP -0.30 -0.12 0.04 - -0.29 -0.09 0.03 -
monthly prediction
Tmax -0.17 0.39 0.31 0.35 -0.69 -0.27 -0.15 0.00 Tmin 0.04 0.81 0.74 0.80 -0.4 0.71 0.59 0.73
PCP -2.27 0.41 0.33 0.40 -1.92 0.41 0.3 0.37
Note: The optimal models are highlighted in bold italics
The direct comparison of the daily- and monthly prediction rows in Table 3.23 clearly indicates the superiority of monthly- over daily predictions, as the NS-values of the former are consistently higher - with a few exceptions - than those of the latter. For this reason, climate prediction on the monthly time scale is advocated in long-term climate assessment.
Performance comparison of single- and multi-domain downscaling 3.6.3.2
The prediction performance of the various combinations of GCM/downscaling methods as outlined in Table 3.1, for climate prediction on the monthly scale is investigated in this section.
This means that the three daily downscaling-methods, i.e. LARSW-WG, SDSM and MLR-daily, are applied to daily GCM-predictors and the ensuing daily predictand output is then summed-up (aggregated) to monthly series. Besides, the pure-monthly downscaling method, using monthly GCM-predictors, i.e. the multi-domain MLR (see Table 3.1), is also employed in this comparison. As mentioned in the previous section, LARS-WG and MLR-monthly are the best models among the single-domain downscaling techniques.
Table 3.24 shows the performance of the single- and multi-domain GCM/downscaling options.
One can notice that, although LARS-WG and single-domain-MLR are best in single-domain downscaling (see previous section), when including the three multi-domain MLR, HiRes and, in particular, GCMs+HiRes are best then. Based on these results, HiRes and GCMs+HiRes are considered as the optimal input-domain for multi-domain MLR downscaling in the eastern seaboard study region.
The quality of the two downscaling methods HiRes and GCMs+HiRes is also demonstrated by the scatterplots of Figure 3.20, where the various downscaled monthly climate predictands are plotted over the observed ones for the 1986-1999 verification period. One can notice that for both models the predicted monthly maximum temperatures are generally underestimating the
Table 3.24. Nash-Sutcliffe (NS)- measured performances of monthly climate predictions in verification years 1986-1999, using LARS-WG-single domain- and MLR- single- and multi-domain GCM-/downscaling combinations.
predictand
single-domain multi-domain
ECHO-G HiRes GCMs GCMs+HiRes
LARS-WG single-domain MLR multi-domain
MLR* multi-domain
MLR multi-domain MLR
Tmax -0.27 0 0.44 0.13 0.27
Tmin 0.71 0.73 0.87 0.79 0.88
PCP 0.41 0.37 0.59 0.57 0.62
Note: The best GCM/MLR- model for each predictand is highlighted in bold italics
*Although the MLR-downscaling model is used here with the single-domain HiRes-GCM, it is designated here as a multi-domain MLR-model, which unlike the single-domain MLR-variant, allows for seasonal prediction and the incorporation of the large predictor set from the multi-domain GCM group.
observed ones, namely for the extreme values. For the minimum temperatures, on the other hand, the predictions fit the observations very well, as the scatter points lie closer on the 45-degree line. As for the precipitation, more scatter is observed, as is to be expected for this rather deviant climate variable, and its extreme values are usually also underestimated. Also, the GCMs+HiRes domain appears to provide better results for the extreme-value rainfall.
The extreme value distributions of the downscaled predictions of the various climate variables will be examined further in Section 3.6.4.2.
Inter-comparison of all downscaling models 3.6.3.3
A summary of the prediction performance of all combinations of GCM/downscaling methods, as outlined in Table 3.1, for climate prediction on the monthly scale is provided in Figure 3.21.
Shown are the NS- coefficients for the verification period 1986-1999. As discussed, while the SDSM and LARS-WG models use daily data, the newly proposed MLR- technique uses both daily and monthly data in downscaling. Consequently, in predicting monthly series, the MLR is then separated into a daily- and monthly downscaling option. Moreover, the autoregressive approach that consists of AR, ARIMA and ARIMAex models, which are mainly developed for short-term prediction (Chapter 4), are applied here also for long-term prediction.
The NS- barplots of Figure 3.21 indicate that even though the performance of LARS-WG is much better than SDSM and also slightly better than MLR-daily in downscaling on the daily time scale, in the prediction of monthly series LARS-WG’s performance is still lower than that of the MLR-monthly method, which employs monthly GCM-predictors directly.
Figure 3.21 shows also that although LARS-WG and MLR-daily perform the best among the single-domain downscaling group (see Table 3.1), their NS-coefficients are consistently lower than that of the monthly-basis multi-domain MLR- method which uses either the Hi-Res- GCM, multi-domain GCMs, or a combination of the two (GCMs+HiRes). In fact, for the monthly maximum temperatures, the HiRes/MLR pair shows the best performance, whereas for the minimum temperature and rainfall, GCMs+HiRes works best. Nonetheless, although the autoregressive models in association with GCM variables (ARIMAex-GCM) (experiment 5 in Table 3.1) is not the best model, it shows some good performance in predicting long-term climate, as will be further discussed in Chapter 4.
Figure 3.20. Scatterplots of the observed and predicted values of a) minimum, b) maximum temperatures at station 48478 and c) precipitation at station 48092 in years 1986-1999, using the MLR- model and employing the single-domain HiRes GCM (left panels) and the multi-model GCMs+HiRes (right panels) large-scale predictor sets.
The top two climate prediction downscaling models, i.e. multi-domain MLR- models using HiRes and GCMs+HiRes have been further applied for climate prediction on the seasonal scale, namely, the classical 4-season scheme which describes best the seasonal climate variability in Thailand. The results are shown in for the three standard climate variables in the three panels of Figure 3.22. One may notice firstly that the predicting skills are generally higher for the dry than for the wet season, the latter including the two monsoon seasons. Thus for both GCM-domains satisfactory NS-coefficients for the temperatures and the rainfall are obtained for the dry season.
For the two monsoon seasons, although showing less prediction skills than for the non-monsoon season, the performance of the mixed multi-domain GCMs+HiRes is better than that of high-resolution GCM domain HiRes. The maximum temperatures are particularly poorly
HiRes GCMs+HiRes
a) maximum temperature
b) minimum temperature
c) precipitation
Figure 3.21. Model performances, as measured by the average Nash-Sutcliffe model efficiency coefficient (NS), of the various GCM/downscaling combinations (see Table 3.1), for predicting monthly maximum (Tmax) and minimum (Tmin) temperature and precipitation (PCP) in the study area during the 1986-1999 verification period, using 1971-1985 as calibration period.
predicted in the second monsoon season by both downscaling variant, whereas the precipitation predictions are still acceptable during this time period of extreme rainfall, wherefore GCMs+HiRes works better than HiRes alone.
The predicting skills of the multi-domain MLR- models on the seasonal and annual bases, using the optimal GCM sets according to the selection in Table 3.22 are exhibited in Table 3.25 for the model verification period. The results of the table show that the annual predictions of all climate variables, except Tmax, are at a satisfactory level (NS>0.5). The seasonal downscalings are all acceptable, wherefore the climate predictions for the dry season are the most reliable.
According to the comparative performance of daily and monthly basis, the forecasting of monthly climate based on monthly-basis assembles shows better predicting skill than of daily-basis. The multi-domain MLR model developed in this study can provide the finest tool to downscale monthly climate in study area by selecting suitable predictor-sets, which is optimal depending on site to site (see Table 3.22). The use of downscaled climate from optimal datasets in the study area is later applied to assess the change of climate and its impact in chapter 6.
Moreover, these downscaled series of monthly climate can be further refined into daily resolution through resampling process (Maurer and Hidalgo 2008) that the rescale of these monthly climate series into daily data is then developed in Chapter 5.
-0.69
-0.27
-0.15
0.00
0.44
0.13
0.27
-0.31
-0.13
0.37
-0.40
0.71
0.59
0.73
0.87
0.79
0.88
0.68
0.53
0.76
0.41
0.30 0.37
0.59 0.57 0.62
0.22
0.38
0.55
-1.0 -0.8 -0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 0.8 1.0
SDSM LARS-WG MLR MLR MLR MLR MLR AR ARIMA ARIMAex-GCM
ECHO-G ECHO-G ECHO-G HiRes CMIP3 CMIP3
+HiRes - - CMIP3
+HiRes daily ensemble daily
ensemble monthly
ensemble monthly ensemble monthly ensemble conventional single-domain MLR multi-domain MLR autoregressive
Nash–Sutcliffe model efficiency coefficient
Tmax Tmin PCP
GCMs GCMs GCMs
Figure 3.22. Seasonal performances of the MLR model, as measured by the NS- coefficient, using the single-domain high-resolution GCM (HiRes) and the multi-domain combination of GCMs with HiRes (GCMs+HiRes) to downscale monthly a) minimum and b) maximum temperature at 4 stations and c) precipitation at 24 stations, for verification period 1986-1999.
Table 3.25. Performances, as measured by the NS-coefficients, of the MLR-model employing optimal predictor-sets for the verification period 1986-1999 on the seasonal and annual bases.
predictand seasonal
annual
1) dry 2) pre-monsoon 3) monsoon_1 4) monsoon_2
Tmax 0.53 0.30 0.30 -0.04 0.45
Tmin 0.83 0.84 0.16 0.17 0.88
HMD 0.81 0.29 0.27 0.46 0.66
SLR 0.12 0.40 0.72 0.45 0.63
PCP 0.75 0.29 0.34 0.37 0.63
%Wet 0.78 0.01 0.37 0.26 0.71
Note: Performances with NS>0.5 are highlighted in bold italics