other common models will be discussed in the following Chapter 4.
Chapter 4
Bayesian-based Boosting for
Quantifying Uncertainty in Structured Additive Regression
The boosting technique has been widely used in making inference for statistical models due to its stable variable selection feature and flexibility regarding the type of predictors.
From the original boosting algorithm (Schapire, 1990; Freund, 1995), which aims to obtain a strong predictor by combining the solutions produced by iteratively applying simple weak classifiers, to Adaboost (Freund and Schapire, 1996, 1997), which has been hailed as the “best off-the-shelf classifier in the world” (Hastie et al., 2009), and later to gradient boosting (Friedman et al., 2000; Friedman, 2001), which adapts the concept of boosting to the field of statistical modeling, boosting has been implemented into almost all statistical topics over the last two decades.
One of the most successful variant of boosting in statistical learning is the compo-nentwise gradient boosting (Bühlmann and Yu, 2003), which updates only one additive base-learner in each iteration. This simple but effective idea not only reduces the high-dimensional analysis to a simple regression problem (Bühlmann, 2006), but also provides the flexibility to estimate various types of base-learners in one additive regression model.
The general estimation method used in gradient boosting is the least squares method. In low-dimensional settings, another typical inference method is the maximum likelihood.
Based on this idea, Tutz and Binder (2006) propose likelihood-based boosting, in which the base-learners are directly estimated via optimizing the overall likelihood by using
69
the additive predictor from the previous iteration as offset (Tutz and Binder, 2007; Groll and Tutz, 2012). Generally, likelihood-based boosting (including the componentwise likelihood-based boosting, which implements the componentwise concept) generates similar results to gradient boosting, and especially in the case ofL2 loss, likelihood-based boosting coincides with gradient boosting. However, in contrast to gradient boosting, approximate confidence intervals can be obtained by likelihood-based boosting (Tutz and Binder, 2006).
Other researches pay more attention to model generalization and regularization techniques. For example, boosting has already been implemented to generalized additive models (GAM) (Tutz and Binder, 2006; Schmid and Hothorn, 2008; Hofner et al., 2014;
Hothorn et al., 2022), the more complex generalized additive models for location, scale and shape (GAMLSS) (Mayr et al., 2012; Thomas et al., 2018; Zhang et al., 2022b), the generalized additive mixed models (Groll and Tutz, 2012), and Cox models (Binder and Schumacher, 2008; Binder, 2013; De Bin, 2016). In addition to the inherent property of the componentwise concept in variable selection, the combination between boosting and ridge (Tutz and Binder, 2007) or lasso (Zhao and Yu, 2004) has also been investigated.
Moreover, the selection performance has been enhanced and improved by using the stability selection approach (Meinshausen and Bühlmann, 2010; Thomas et al., 2018).
For more details on the evolution of boosting, please review Mayr et al. (2014, 2017a).
Although more and more studies regarding statistical boosting have been published in recent decades, The majority of the published papers regarding boosting techniques focus on improving the estimation accuracy or in combination with various statistical models. Nevertheless, to the best of our knowledge, little literature has studied the fusion of boosting and Bayesian statistics, the latter of which, however, occupies half of modern statistics due to its unique philosophical perspective and also the computational advantages. In the few papers we found that contain both keywords (Bayesian and boosting), Elkan et al. (1997) propose a boosted naive Bayesian learner, which is equivalent to standard feedforward multilayer perceptrons. Similar models as well as their improvements can also be found in Bauer and Kohavi (1999); Ting and Zheng (1999), but due to their early publication, they are limited to the AdaBoost framework. Another paper (Nock and Sebban, 2001) proposes a so-called “Bayesian boosting theorem”, which concernes the AdaBoost as well, aiming, however, at bounding
71 the error of the boosting algorithm and increasing the convergence speed instead of combining the two (Bayesian and boosting) philosophies. The most relevant yet unpublished work by Lorbert et al. (2012), while still built on AdaBoost, performs approximate inference about the posterior distribution associated with latent variables or weights placed on the base classifiers. In addition, the quality of the learned classifier can be measured by the noise statistics of the classifier produced by the algorithm.
Even though Tutz and Binder (2006) point out the possibility of constructing approximate confidence intervals in likelihood-based boosting, this concept did not receive much attention until the publication of Rügamer and Greven (2020), which may be the first systematic study of uncertainty quantification in boosting. They propose inference for L2-boosting in the special case of linear, grouped, and penalized additive models selected by L2-boosting using the selective inference framework (Fithian et al., 2014; Tibshirani et al., 2016; Yang et al., 2016), a method that transfers classical statistical inference to algorithms with preceding selection of model terms. Compared to the previous ad-hoc solutions such as the permutation test (Mayr et al., 2017b), which is restricted to certain special cases, or the bootstrap (Brockhaus et al., 2015;
Rügamer et al., 2018; Hepp et al., 2019), which does not lead to confidence intervals with proper coverage due to the bias induced by the shrinkage effect, the advantage of using a classical statistical method to quantify the uncertainty of boosting estimates is obvious. Yet, as possibly the first paper studying uncertainty in boosting, it focuses only on special additive models. Therefore, further work is still needed to cover a more general model family.
Instead of using this frequentist statistical approach to quantify uncertainty in boosting, we provide a Bayesian solution. In our previous studies (Zhang et al., 2022a), we introduced a method that integrates a Bayesian learner into the boosting framework for linear mixed models. However, the proposed algorithm was still preliminary, only providing uncertainty information for random effects. In this chapter, we extend this method by proposing a more general boosting framework based on Bayesian methodology, that integrates Bayesian penalized regression in the componentwise boosting framework.
Compared to the previous work, the more general approach not only makes it possible to extract uncertainty information from the fixed effects, but also extends the model family to the generally structured additive regression (STAR) models, which is friendly
to nonlinear and spatial base-learners.
In addition to richer technical features, this novel approach fills the gap in the field of applying Bayesian inference to boosting. In contrast to the dogmatic estimate of boosting, which delivers only an unquestionable point estimation, the proposed method not only benefits from the uncertainty and prior knowledge of Bayesian methods, but also maintains the useful features of boosting, for example, the intuitive variable selection procedure and the flexibility of various types of base-learners. Furthermore, this combination also provides a new way of thinking about the regularization research of classical Bayesian methods. We denote our proposed novel method asBayesian-based boosting or bboost throughout this context.
This chapter is structured as follows: Section 4.1 describes briefly the basics of Bayesian penalized regression and componentwise gradient boosting and then proposes the details of the Bayesian-based boosting algorithm. In section 4.2, we compare our method with other commonly used methods through simulations of linear and non-linear scenarios, especially the performance in estimation accuracy, uncertainty, and variable selection. An empirical study, which analyzes the Munich rent index data including additional spatial variables, is presented in section 4.3. The study helps to demonstrate the effectiveness of the proposed method in STAR models. The final section 4.4 summarizes the chapter and discusses potential improvements of the proposed method and possibilities of other relevant further works.