In this section, we want to show that the same results, which held for the probabilistic loss for the known variance case, continue to hold up to small correction terms in the bootstrap version of the method. In the following, we define the quantities that will govern how well the bootstrap method will perform. We write Ψm,i for the ith column of Ψm.
We measure thedesign regularity by the value δΨ
δΨ def= max
m∈M max
1≤i≤nσi∥Sm−1/2Ψm,i∥. (4.3.1) where Sm=ΨmΣΨm⊤.
Thepresmoothing biasfor the presmoothing operator Π is measured by the vector
B = Σ−1/2(f∗−Πf∗).
It is the approximation bias of f∗ by Πf∗ weighted by the standard devi-ations of the noise terms.
We also measure the presmoothing stochastic noisein terms of the covariance matrix Var(˘ε) of the smoothed noise ˘ε=ε−ΠΣε, where ΠΣdef
= Σ−1/2ΠΣ1/2. Namely, this matrix is assumed to be sufficiently close to the unit matrix 1n, in particular, its diagonal elements should be close to one.
This is measured by the operator norm of Var(˘ε)−1n and by the deviation of the individual variances IEε˘2i from one. We also need a control on the
maximal variance of the individual variances of ˘εi−εi. We write this as:
c1 def
= ∥Var(˘ε)−1n∥op, δ1
def= max
1≤i≤n
Var (˘εi−εi), δ2 def= max
1≤i≤n|IEε˘2i −1|1/2.
In particular, in the case of homogeneous errors Σ =σ21n and assuming the presmoothing operator Π to be a p-dimensional projector of the form Π=Ψm⊤†
Ψm†Ψm⊤†
−1
Ψm† for some model m†∈M. We have Var(˘ε) = (1n−Π)2=1n−Π≤1n, and
c1 = ∥Var(˘ε)−1n∥op = ∥Π∥op = 1, δ21 = max
1≤i≤nVar (˘εi−εi) = max
1≤i≤nΨm⊤†,i
Ψm†Ψm⊤†
−1
Ψm†,i δ22 = max
1≤i≤n|IEε˘2i −1| = max
1≤i≤nΨm⊤†,i
Ψm†Ψm⊤†
−1
Ψm†,i, and δ1, δ2 will usually be of order
p
n, if Ψm† satisfies some Lindeberg-type condition similar to (4.3.1). In the following theorem, we will show that when using the bootstrap calibrated bounds zm,m♭ ◦(x+qm♭◦) instead of zm,m◦(x+qm◦) , we will get almost the same probability statements for the stochastic noise terms ξm,m◦. For the statement of the theorem, we assume that we can write our models as the projection of a larger model onto a smaller feature set. We write Ψ =Ψmmax ∈ IRp×n for the design matrix of the largest model mmax ∈ M with feature dimension p. We assume that the Ψm from (4.1.2) can be written as projections of the largest model onto a smaller feature set:
Ψm =ΠmΨ for a projector Πm.
Theorem 4.3.1. Let Y = f∗ +ε be a Gaussian vector in IRn with in-dependent components, Y ∼ N(f∗, Σ) for Σ = diag(σ21, . . . , σn2). Let
61 the design matrix Ψ ∈ IRp×n of the largest model in M, be such that S = Ψ ΣΨ⊤ ∈ IRp×p is invertible. For a given presmoothing operator Π: IRn → IRn let the values zm,m♭ ◦(x) and qm♭◦ for all m > m◦ be de-fined by (4.2.2) and (4.2.1). Then it holds
IP
m>mmax◦
∥ξm,m◦∥ −zm,m♭ ◦(x+qm♭◦)
≥0
≤ 8 exp(−x) +∆ξ(x).
with
∆ξ(x) def= p1/2(4δ12xn+ 4
√
2δ1xn+ 4
√
2∥B∥∞δ1
√xn+ 2δΨ
x+ log(2p) +2δ2Ψ(x+ log(2p)) +∥B∥2∞+δΨ2∥B∥√
2x, if ∆ξ/p1/2≤1/2.
To give a more readable version of the bound, we make the assumption, that we can bound δΨ, δ1 by some common δ and 2p≤n, then we can get a simpler bound:
∆ξ(x) ≤ C xnp1/2
δ+∥B∥∞δ+∥B∥2∞+∥B∥δ2 for some numerical constant C>0 .
The SmA procedure also involves the values pm,m◦ which are unknown and depend on the noise structure Σ. The next result shows that the bootstrap counterparts pm,m♭ ◦ can be well used in place of pm,m◦.
Theorem 4.3.2. Assume the conditions of Theorem 4.3.1. Then it holds for pm,m♭ ◦ =IE♭∥ξm,m♭ ◦∥2:
IP
m>mmax◦,m,m◦∈M
pm,m♭ ◦ pm,m◦
−1
> ∆p(x)
≤ 3 exp(−x), where
∆p(x) def= 4x1/2M δΨ + 2xMδΨ2 +∥B∥2∞+ 4x1/2M δΨ2 ∥B∥+δ2
with xM=x+ 2 log(|M|).
Again we are giving a simplified bound under the assumption that some δ bounds δ2 and δΨ from above and that x≥1 .
∆p ≤C xM
δ+δ2∥B∥+∥B∥2∞
for some numerical constant C >0 . Finally, we show that the bounds we used in Theorem 3.4.2 in the section to bound the payment for adaptation are the same as for the bootstrap-case up to some correction term:
Proposition 4.3.3. Under the conditions of Theorem 4.3.1 IP
m>mmax◦
zm,m♭ ◦−Kz(x)
(1 +β)√
pm,m◦−
2λm,m◦xM
≥0
≤3 exp(−x) +∆ξ(x), where
Kz(x) = max{
1 +∆ξ(x),
1 +∆p(x)}
with ∆ξ, ∆p from Theorems 4.3.1 and 4.3.2 and again xM=x+ 2 log(|M|). The above results allow to extend all the oracle bounds for probabilistic loss of Chapter 3 with the obvious corrections of the error probability. We will give one example of such an extension in Theorem 4.3.5 further below.
But first we discuss the sense of the required conditions for bootstrap validity. In typical situations, we have δ = max{δΨ, δ2, δ1} ≤ C
p n. One can see that the bootstrap approximation is accurate if the values ∆ξ and
∆p are small. This requires that the values δ2p, ∥B∥4∞p, and δ2∥B∥ are sufficiently small. It is easy to see that the last term is smaller in order than the others. We have
δ2p=O(p2/n).
Further, the bias component does not damage the bootstrap validity result if ∥B∥4∞p is a small value. If f∗ is H¨older-smooth with the parameter s, that is, if
∥B∥∞ ≤Cp−s, (4.3.2)
63 then the bootstrap procedure is justified for s > 1/4 if p = pn → ∞ as n → ∞ and p2log(n)/n goes to zero. We state one asymptotic result of this sort.
Corollary 4.3.4. Assume that δ = max{δΨ, δ2, δ1} ≤ C
p
n. Let also p=pn satisfy p2nlog(n)/n→0 as n→ ∞, and (4.3.2) hold for s > 1/4. Then the results of Theorem 4.3.1, 4.3.2 and Proposition 4.3.3 apply with a small value ∆ξ=∆ξ,n→0 as n→ ∞.
We now give a bootstrap version of Theorem 3.4.1. We have to change the definition of m∗ slightly for this. We define:
m∗ def= min
m◦∈M: max
m>m◦
∥bm,m◦∥2−β2(1 +∆p)pm,m◦
≤0
. (4.3.3) If we assume to be in a case where ∆p is small, this means that we slightly change the value of β. For ∆p going to zero for n going to infinity this definition coincides asymptotically with our original definition . We are now ready to give the following probabilistic oracle result.
Theorem 4.3.5. Assume the conditions of Theorem 4.3.1. Given x and β, let the critical values for the SmA-method be given by zm,m♭ ◦ from(4.2.3) and let m∗ ∈M satisfy(4.3.3). Then the SmA estimator ϕ =ϕ
m satisfies the following bound:
IP
ϕ−ϕm∗
>zm♭∗
≤ 11 exp(−x) +∆ξ(x), (4.3.4) where zm♭∗ is defined as
zm♭∗ def
= max
m∈M+(m∗)
zm♭∗,m.
This implies the probabilistic oracle bound: with probability at least 1− 11 exp(−x)−∆ξ(x)
ϕ−ϕ∗
≤
ϕm∗−ϕ∗
+zm♭∗. (4.3.5) In the next section, we are going to present some simulations for the method.