Out-of-Sample Forecast Evaluation

To evaluate whether the focus of the search interest on Google really helps to predict returns or volatility, we perform a one day, one week and 2 weeks ahead out-of-sample forecast of returns and volatility by estimating model specifications 0 to 5 on a growing sample size. We start with 180 observations and add one observation at a time to predict the next day, next week (7 days) or the next two weeks (14 days). In order to arrive at a 2 week forecast for volatility, after transforming the forecasted, daily, log-volatility values to volatility, we sum them up. Log-returns are also summed up. During each estimation step the optimal lag-length is determined anew via the SIC based on the observations within the growing sample window. As the one-step-ahead forecasts are exemplary for the one-week (7-steps-ahead) and two-week (14-steps-ahead) forecasts, we only report these.

All other results are available from the authors upon request.

The one-day-ahead forecasts bear some remarkable results. In the case of the return models, for four coins (BitcoinCash, Ethereum, Monero, and zCash) the RM SE is significantly improved when Google search volumes are added to model 0 as can be seen from Table 2.8.

In all other cases, albeit we might observe a reduction of the RM SE, it is not statistically significant. The R²_{M Z} is very low: In general, the variation in the forecast can explain less than 1% of the variation in the observed returns. Hence, the prediction can be considered random which is illustrated in Figures 2.3a and 2.4a.

Figure 2.3a presents a scatter plot of the forecasted and the observed returns for Bitcoin.

The black line is the 45^○ line which marks the location of a perfect fit. As can be seen, the forecasted values do not vary a lot, the prediction is always close to zero. Hence, the location of the points is limited to a wide, ellipse-like area around the origin. For all other cryptocurrencies, the shape of such a scatter plot is similar.

Figure 2.4a provides further details. We zoom into the prediction based on the best fitting model for Bitcoin returns. The autoregressive process specified in Equation (2.1) cannot

Table2.7:In-SampleFitVARModelforVolatility Thetableliststherootmean-squarederror(RMSE)multipliedby100ofthein-samplepredictionsofthemodelsaswellastheR2ofaMincer-Zarnowitzregression ?inpercentages.Furthermore,theQuasi-maximumlikelihoodmeasure(QL)isreportedasanoutlierrobustmeasure(cp.Patton2011).Usingtheforecast evaluationtestof?fornestedmodels,theRMSEofModels1-5canbetestedwhethertheyresultinasmallerRMSEthanModel0,ourbenchmarkmodel.One starindicatesthatthenullhypothesis(thattheRMSEofthebenchmarkModelissmaller)canberejectedona10%significancelevel,twostarssignifyrejection onthe5%significancelevel.FortheRMSEandtheQL,thesmallestvalue,foreachcryptocoin,acrossthemodelsistypesetinbold.FortheR2 MZ,thehighest valueisreportedboldfaced. Model0Model1Model2Model3Model4Model5 QLRMSER2 MZQLRMSER2 MZQLRMSER2 MZQLRMSER2 MZQLRMSER2 MZQLRMSER2 MZ BitcoinCash0.926.0530.840.835.3735.460.815.3633.470.815.3932.620.795.22**37.550.825.3535.95 Bitcoin1.113.0133.751.052.97**35.521.113.01**34.01–––1.052.97**35.53––– Dashcoin0.974.7318.780.914.66**21.440.914.65**21.730.944.69**20.000.914.65**21.720.914.64**21.94 EOSToken0.826.6632.030.685.5737.120.614.1030.280.614.0930.700.685.5737.150.675.5238.93 EthereumClassic0.914.8530.700.764.28**35.790.764.29**35.500.744.26**35.770.804.34**33.520.804.34**33.40 Ethereum1.177.9522.070.914.65**34.380.924.62**33.980.904.63**33.700.904.64**34.540.894.64**34.48 Gnosis0.995.8719.441.015.8819.231.005.8819.381.005.8919.111.015.8819.331.035.9716.70 Litecoin1.395.8327.451.385.77**28.911.355.77**28.391.325.74**28.981.375.75**29.271.305.68**30.21 AugurCoin0.966.2021.870.855.65**25.200.895.7823.650.895.7923.480.845.69**26.620.845.69**26.51 Monero0.683.8530.130.683.8529.920.683.79**31.880.663.8629.720.663.83**30.930.663.83**30.58 Ripple0.925.0734.430.874.86**39.230.925.0534.480.915.1032.310.874.86**39.100.854.89**37.87 zCash1.2410.8242.000.966.6041.270.986.9150.571.007.0047.820.986.9450.150.986.9649.03

Table2.8:Out-of-SampleFitOne-Day-AheadForecast–Returns Thetableliststherootmean-squarederror(RMSE)multipliedby100fortheone-day-aheadout-of-samplepredictionsofreturnsforeachmodelaswellasthe fitofaMincer-ZarnowitzregressionR2 MZ?inpercentages.Furthermore,theQuasi-maximumlikelihoodmeasure(QL)isreportedasaoutlierrobustmeasure (cp.Patton2011).Usingtheforecastevaluationtestof?fornestedmodels,theRMSEofModels1-5canbetestedwhethertheyresultinasmallerRMSE thanModel0,ourbenchmarkmodel.Onestarindicatesthatthenullhypothesis(thattheRMSEofthebenchmarkModelissmaller)canberejectedona10% significancelevel,twostarssignifyrejectiononthe5%significancelevel.FortheRMSEandtheQL,thesmallestvalue,foreachcryptocoin,acrossthemodelsis typesetinbold.FortheR2 MZ,thehighestvalueisreportedboldfaced. Model0Model1Model2Model3Model4Model5 RMSER2 MZRMSER2 MZRMSER2 MZRMSER2 MZRMSER2 MZRMSER2 MZ BitcoinCash6.970.546.93*0.137.021.496.980.736.290.196.320.42 Bitcoin3.870.003.900.113.890.01––3.890.01–– Dashcoin6.640.116.690.166.690.166.670.396.720.086.710.22 EOSToken5.690.005.790.445.960.736.000.005.930.185.980.55 EthereumClassic7.670.007.750.247.740.087.750.027.810.177.810.07 Ethereum6.680.086.62**0.176.67**0.116.64**0.066.680.026.680.00 Gnosis5.290.005.350.495.372.025.310.245.433.045.442.81 Litecoin5.930.006.040.006.060.006.010.016.080.006.100.00 AugurCoin9.400.129.461.089.451.639.490.049.701.299.540.03 Monero7.140.797.190.037.140.267.14*0.607.180.067.190.23 Ripple8.080.008.190.258.270.068.110.297.490.107.960.31 zCash10.270.077.52**0.007.41**0.027.49**0.037.51**0.067.57**0.19

Figure 2.3: Fit of One-Day-Ahead Forecasts

The graphs depict the observed returns (horizontal axis) against the one-day-ahead forecasts based on Model 0 (orange ◇), Model 1 (blue○), Model 2 (red▽) and Model 5 (black ⊠) for Dashcoin (left) and Monero (right). A perfect fit would mean that the values are aligned on the 45^○-degree line (black; note the scaling of the axes).

−0.3 −0.2 −0.1 0.0 0.1 0.2 0.3

−0.10−0.050.000.050.10

Observed

Forecast

(a) Bitcoin Returns

0.0 0.1 0.2 0.3 0.4 0.5

0.000.020.040.060.080.100.120.14

Observed

Forecast

(b) Bitcoin Volatility

mimic the high volatility of the return series and therefore results in smooth forecasts centered around zero. Furthermore, individual large spikes result in a forecast that is further away from zero, albeit one period too late. In general, we conclude that for the out-of-sample prediction of returns on a daily basis, Google searches are not helpful.

Table 2.9 holds the forecast evaluation results for the volatility models. In general, the one-day-ahead forecast of volatility is improved when one or more of Google’s SVIs are added to Model 0. Only for Ethereum Classic and AugurCoin none of the SVI models reduces the RM SE significantly. For Dashcoin theQL selects the benchmark Model 0, whileRM SE andR²_{M Z} favor Model 4. The R²_{M Z} of the BitcoinCash-models favors the benchmark Model 0, while QLand RM SE choose Model 4. With the exception of these four coins, the evaluation criteria of all other coins select a model in which Google’s SVI is added.

Figure 2.3b shows the scatter plot of the actual and forecasted volatility for Bitcoin. There are two important takeaways. First, the model fit is much better than for returns (depicted in Figure 2.3a) as the values are much more clustered around the 45^○ line. Second, the points that belong to Model 0 (orange) are further away from the 45^○ line than the points associated with any other model. In particular, the points which belong to Model 5 (black) are the ones that are the closest to the line of the perfect fit, especially for high values of volatility. Hence, we conclude that in general, the addition of Google-search volume helps

Figure 2.4: Time Series of One-Day-Ahead Forecasts

The graph presents the observed time series of Bitcoin returns (a) and volatility (b) as a black solid line and the one day ahead forecast (blue dots) based on Model 1, only including the SVI of the search-term Bitcoin. Confidence intervals on the 0.95% level are shaded in gray.

Sep 03 Sep 10 Sep 17 Sep 24 Okt 01

−0.050.000.050.10

Time

Returns

(a)Bitcoin Returns

Sep 03 Sep 10 Sep 17 Sep 24 Okt 01

0.000.020.040.060.080.100.12

Time

Volatilty

(b)Bitcoin Volatility

to predict volatility, which is further illustrated in Figure 2.4b. As in the case of returns, when large changes occur in the volatility series, the forecast does not react as quickly and picks up the movement with one period lag. It turns out that, in periods of extreme volatility, the inclusion of SVIs is more helpful than in periods with low volatility which is in line with the results of Dimpfl and Jank (2016).