D Proofs for Sections 6 and 7
D.2 Proof of Theorem 7.1
Lemma D.3. Define ρ(βS) =E(Y −h(XTSβS))h′(XTSβ0S)XSσ(XS)−2. Under the theorem assumptions,
sup
βS∈Θkρ(βS)−ρn(βS)k=op(1).
Proof. Given E(supβ∈Θh(XTSβ)4) < ∞ and supt|h′′(t)| < ∞, we have the uniform law of large number (Newey and McFadden 1994, Lemma 2.4)
sup
sup
Using these, we show three convergence results:
1 For (D.1), the left hand side is upper bounded by (for some ˜βin the segment joiningβ0S and βbS, and apply Cauchy-Schwarz inequality)
1
where in the second inequality, we used the uniform weak law of large number. Similarly, the left hand side of (D.2) is upper bounded by
sup
where both the first and second inequalities follow from the Cauchy-Schwarz inequality, and the third inequality follows from the uniform law of large number. (D.3) can be established
in a similar way sinceσ(Xb S)2 uniformly converges to σ(XS)2.
Due to the previous convergences and that the event XS =XbS occurs with probability approachong one, it remains to show that supβ
S∈Θkρ(βS)k<∞and sup
βS∈Θk1 n
Xn i=1
XiSh′(XTiSβ0S)(Yi−h(XTiSβS))σ(XiS)−2
−EXSh′(XTSβ0S)(Y −h(XTSβS))σ(XS)−2k=op(1).
The above result follows from the uniform law of large number to n1 Pn
i=1h(XTiSβS)2 − Eh(XTSβS)2, given that EsupβS∈Θh(XTSβS)4 < ∞. The fact that supβS∈Θkρ(βS)k < ∞ follows from repeatedly using Cauchy-Schwarz inequality.
Q.E.D.
Given the foregoing Lemma D.3, Theorem 7.1 follows from a standard argument for the asymptotic normality of GMM estimators as in Hansen (1982) and Newey and McFad-den (1994, Theorem 3.4). The asysmptotic variance achieves the semiparametric efficiency bound derived by Chamberlain (1987) and Severini and Tripathi (2001). Therefore, βb∗ is semiparametric efficient.
Q.E.D.
References
Andrews, D. (1999). Consistent moment selection procedures for generalized method of moments estimation. Econometrica,67 543-564
Andrews, D. and Lu, B. (2001). Consistent model and moment selection procedures for GMM estimation with application to dynamic panel data models. J. Econometrics, 101 123-164
Antoniadis, A. (1996). Smoothing noisy data with tapered coiflets series. Scand. J. Stat., 23, 313-330
Belloni, A. and Chernozhukov, V. (2011a). Least squares after model selection in high-dimensional sparse models. Forthcoming in Bernoulli. Manuscript. MIT.
Belloni, A. and Chernozhukov, V. (2011b). l1-penalized quantile regression in high-dimensional sparse models. Ann. Statist.,39, 82-130.
Bickel, P., Klaassen, C., Ritov, Y. and Wellner, J. (1998). Efficient and adaptive estimation for semiparametric models. Springer, New York.
Bradic, J., Fan, J. and Wang, W. (2011). Penalized composite quasi-likelihood for ultrahigh-dimensional variable selection. J. R. Stat. Soc. Ser. B, 73, 325-349.
B¨uhlmann, P., Kalisch, M. and Maathuis, M. (2010). Variable selection in high-dimensional models: partially faithful distributions and the PC-simple algorithm.
Biometrika,97, 261-278
B¨uhlmann, P.andvan de Geer, S.(2011).Statistics for High-Dimensional Data: Meth-ods, Theory and Applications. Springer, New York.
Caner, M. (2009). Lasso-type GMM estimator. Econometric Theory, 25 270-290
Caner, M. andZhang,H.(2009). General estimating equations: model selection and esti-mation with diverging number of parameters. Manuscript, North Carolina State University Candes, E. and Tao, T. (2007). The Dantzig selector: statistical estimation when p is much larger than n. Ann. Statist., 35 2313-2404
Chamberlain, G. (1987). Asymptotic efficiency in estimation with conditional moment restrictions. J. Econometrics, 34 305-334
Daubechies, I., Defrise, M. and De Mol, C. (2004). An iterative thresholding algo-rithm for linear inverse problems with a sparsity constraint. Comm. Pure Appl. Math., 57, 1413-1457.
Donald, S., Imbens, G. and Newey, W. (2003). Empirical likelihood estimation and consistent tests with conditional moment restrictions. J. Econometrics,117 55-93
Donoho, D. L. (2006). Compressed sensing. IEEE Trans. Inform. Theory 52, 1289–1306.
Donoho, D. L. and Elad, E. (2003). Maximal sparsity representation via l1 Minimization, Proc. Nat. Aca. Sci., 100, 2197-2202.
Engle, R., Hendry, D.andRichard, J.(1983). Exogeneity.Econometrica.51, 277-304.
Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. J. Amer. Statist. Assoc., 96 1348-1360
Fan, J. and Lv, J. (2008). Sure independence screening for ultrahigh dimensional feature space. J. R. Stat. Soc. Ser. B, 70, 849-911.
Fan, J. andLv, J.(2011). Non-concave penalized likelihood with NP-dimensionality. IEEE Trans. Inform. Theory, 57,5467-5484.
Fan, J. and Yao, Q. (1998). Efficient estimation of conditional variance functions in stochastic regression. Biometrika, 85, 645-660.
Fu, W. (1998). Penalized regression: The bridge versus the LASSO. J. Comput. Graph.
Statist., 7, 397-416.
Gautier, E. and Tsybakov, A. (2011). High dimensional instrumental variables regres-sion and confidence sets.Manuscript.
Hansen, B. (2010).Econometrics, Unpublished manuscript. University of Wisconsin.
Hansen, L. (1982). Large sample properties of generalized method of moments estimators.
Econometrica, 50 1029-1054
Horowitz, J.(1992). A smoothed maximum score estimator for the binary response model.
Econometrica 60 505-531
Huang, J., Horowitz, J.and Ma, S. (2008). Asymptotic properties of bridge estimators in sparse high-dimensional regression models. Ann. Statist. 36 587-613
Kitamura, Y., Tripathi, G. and Ahn, H. (2004). Empirical likelihood-based inference in conditional moment restriction models. Econometrica, 72 1667-1714
Liao, Z. (2010). Adaptive GMM shrinkage estimation with consistent moment selection.
Manuscript. Yale University.
Lounici, K. (2008). Sup-norm convergence rate and sign concentration property of Lasso and Dantzig estimators. Electron. J. Stat., 2, 90-102.
Lv. J.andFan, Y.(2009). A unified approach to model selection and sparse recovery using regularized least squares. Ann. Statist. 37 3498-3528
Newey, W.(1990). Semiparametric efficiency bound J. Appl. Econometrics,5 99-125 Newey, W. (1993). Efficient estimation of models with conditional moment restrictions, in Handbook of Statistics, Volume 11: Econometrics, ed. by G. S. Maddala, C. R. Rao, and H. D. Vinod. Amsterdam: North-Holland.
Newey, W. and McFadden, D.(1994). Large sample estimation and hypothesis testing, inHandbook of Econometrics, Chapter 36, ed. by R. Engle and D. McFadden.
Owen, A. (1988). Empirical likelihood ratio confidence intervals for a single functional.
Biometrika,75, 237-249.
Raskutti, G., Wainwright, M. and Yu, B. (2011). Minimax rates of estimation for high-dimensional linear regression overlq-balls. IEEE Trans. Inform. Theory,57,6976-6994.
St¨adler, N., B¨uhlmann, P. and van de Geer, S. (2010). l1-penalization for mixture regression models (with discussion). Test, 19, 209-256
Severini, T. and Tripathi, G. (2001). A simplified approach to computing efficiency bounds in semiparametric models. J. Econometrics,102, 23-66.
Tibshirani, R. (1996). Regression shrinkage and selection via the Lasso. J. R. Stat. Soc.
Ser. B, 58 267-288
Verbeek, M. (2008). A guide to modern econometrics. 3rd edition. John Wiley and Sons, England.
Wasserman L.andRoeder, K.(2009). High-dimensional variable selection.Ann. Statist., 37 2178-2201.
Zhang, C.(2010). Nearly unbiased variable selection under minimax concave penalty,Ann.
Statist., 38 894-942
Zhang, C. and Huang, J. (2008). The sparsity and bias of the Lasso selection in high-dimensional linear models.Ann. Statist.,36 1567-1594.
Zhang, C. and Zhang, T. (2012). A general theory of concave regularization for high dimensional sparse estimation problems/Manuscript, Rutgers University.
Zhang, T. (2010). Analysis of multi-stage convex relaxation for sparse regularization. J.
Mach. Learn. Res., 11 1087-1107.
Zhao, P. and Yu, B. (2006). On model selection consistency of Lasso. J. Mach. Learn.
Res.,7 2541-2563
Zou, H.(2006). The adaptive Lasso and its oracle properties.J. Amer. Statist. Assoc., 101 1418-1429
Zou, H. and Hastie, t. (2005). Regularization and variable selection via the elastic net.
J. R. Stat. Soc. Ser. B 67 301-320
Zou, H. and Li, R. (2008). One-step sparse estimates in nonconcave penalized likelihood models. Ann. Statist., 36 1509-1533