6.3 Analysis of empirical results
6.3.2 In-sample forecasting accuracy
6.2.3.4, the GRW beta can be thought of as a smoothed RW beta. For Kalman filter betas, the characteristics of the stochastic process of systematic risk depend strongly on the estimated values for the transition parameter: while φi is highly significant and close to 0.5 for the MR model, it is insignificant and close to zero for the MMR model. At the same time, the variance term σζ is significantly different from zero. Hence, the MMR model turns out to behave like a random coefficient model that fluctuates randomly around a moving mean.
6.3.2 In-sample forecasting accuracy
The results above strongly indicate that systematic risk is not stationary and that the nature of the time-varying behavior of beta depends on the chosen modeling
technique. In order to determine the relatively best measure of time-varying sys-tematic risk, the quality of estimated conditional betas could be evaluated based on the goodness of fit and diagnostic statistics reported in 6.2. However, in the con-text of this thesis at least two problems arise in connection with such an approach.
The first issue is related to the residuals that are employed to calculate fit statistics.
With the exception of the GARCH model, the fit statistics to be considered could either be calculated based on recursive residuals or generalized residuals (cf. 3.6.1).
For a comparison with OLS, the generalized residuals should be used as these are also based on the full sample. However, for GARCH based models only recursive residuals are available. As a consequence, all derived fit statistics would employ less information than in case of the alternative modeling techniques. The GARCH based approach to conditional betas would be at a structural disadvantage. The second problem refers to the way time-varying betas are derived. While GARCH based be-tas are constructed indirectly, conditional bebe-tas derived by a state space model are calculated directly. As the resulting test statistics refer to different aspects of the respective models, they should only be employed for comparative purposes within the same modeling class.
To avoid these problems, in the following the different techniques are formally ranked based on their in-sample forecast performance. Following previous studies, the two main criteria used to evaluate and compare the respective in-sample forecasts are the mean absolute error (MAE) and the mean squared error (MSE):
M AEi = 1 T
XT
t=1
|Rˆi,t−Ri,t|, (6.41) M SEi = 1
T XT
t=1
( ˆRi,t−Ri,t)2, (6.42) where T is the number of forecast observations; ˆRi,t = ˆβi,tR0,t denotes the series of return forecasts for sector i, calculated as the product of the conditional beta series estimated over the entire sample and the series of market returns. The latter is assumed to be known in advance, which is a commonly made assumption in the context of forecast evaluation. The forecast quality is inversely related to the size of these two error measures.
Figure 6.10 displays the average MAE and MSE measures across all sectors for the different modeling techniques on the left hand side. Panel (b) shows the respective average ranks of mean absolute and mean squared errors for each approach under consideration. For a more detailed sectoral breakdown, see Tables C.2 and C.3 in the appendix. A comparison of the different modeling techniques confirms the conjecture that the forecast performance of standard OLS is worse than for any time-varying technique. However, compared to the GARCH based techniques and the two Markov switching approaches, the degree of OLS’ inferiority is remarkably
low.21 The MSE of OLS equals 2.89, which is only slightly higher than the MSE for the Markov switching model (2.78).
OLS tG MSM MS GRW SV RW MR MMR
Figure 6.10: In-sample forecasting evaluation: (a) average MAE and MSE across sectors and (b) average ranks across sectors.
For the investigated sample, the two mean reverting techniques clearly outperform their competitors. The average mean squared error of 1.56 is nearly 50% lower than in case of OLS. With respect to both error measures, the MMR model ranks first in ten occasions and second in eight cases. With the exception of the MAE for Personal & Household Goods, each time the MMR only ranks second, the MR takes the top spot. The average rank of the mean absolute and mean squared errors for the RW model is equal to 3.3 and 3.2, respectively. Whenever the RW model does not rank third, it is usually outperformed by the SV model. The GRW model ranks behind the SV model. As the proposed generalization was intended not to capture every spike and to yield smoother conditional betas than the standard RW model, the comparably weak in-sample performance is not surprising. Within the class of volatility models, the SV approach seems to be better qualified to model the time-varying behavior of systematic risk than the well established GARCH model.
On average, the MAE (MSE) for the SV model is 6% (13%) lower than the error measures for the GARCH based models, and 25% (59%) higher compared to the overall best model. Within the Markov switching framework, the MS betas lead to lower average errors than the MSM in case of fourteen sectors.
While the mean error criteria can be used to evaluate the average forecast perfor-mance over a specified period of time for each model and each sector individually, they do not allow for an analysis of forecast performances across sectors. From a practical perspective, it is interesting to see how close the rank order of fore-casted sector returns corresponds to the order of realized sector returns at any time.
21When interpreting the in-sample results, it should be remembered that, in contrast to all other techniques, the GARCH based betas are based on filtered instead of smoothed estimates.
Spearman’s rank correlation coefficient,ρSt , represents a non-parametric measure of correlation that can be used for ordinal variables in a cross-sectional context. It is introduced as the third evaluation criteria: after ranking the predicted and ob-served sector returns separately for each date, where the sector with the highest return ranks first, ρSt can be computed as
ρSt = 1− 6PNt
i=1D2i,t
Nt(Nt2−1), (6.43)
with Di,t being the difference between the corresponding ranks for each sector, and Nt being the number of pairs of sector ranks, each at time t. Figure 6.11 plots histograms of the in-sample rank correlations for the different modeling techniques together with their respective medians. The reported value of F(0) denotes the proportion of rank correlations that are smaller than zero.
probability density
Figure 6.11: Histograms of Spearman’s in-sample rank correlations.
The highest medians of in-sample rank correlations are observed for the MMR and the MR models, where 50% of the computed rank correlations exceed the value of 0.616 and 0.513, respectively. For both models the distribution of rank correlations is negatively skewed, with the share of negative rank correlations being smaller than 10% (11%) for the MMR (MR) model. This confirms the finding gained from the analysis of mean errors above that these two models provide the best in-sample measures of time-varying betas. The next best results are observed for the RW,
the SV and the GRW model. In contrast to the two mean reverting models, about a quarter of computed rank correlations is negative. The GARCH and Markov switching models do only slightly better than OLS.
Overall, the analysis of in-sample estimates suggests that time-varying European sector betas as modeled by one of the proposed Kalman filter approaches are superior to the considered alternatives. This is in line with previous findings presented by Brooks et al. (1998) and Faff et al. (2000) for industry portfolios in Australia and the UK.