Σ, where Σ is a regular matrix and the matrix X has rank k

Mathematical Statistics, Winter semester 2020/21 Problem sheet 1

1) Suppose that

Y = Xβ + ε

holds for someβ ∈R^kand thatEε = 0_n, Cov(ε) = Σ, where Σ is a regular matrix and the matrix X has rank k.

(i) Show that X^TΣ⁻¹X is a regular matrix and that

βb = (X^TΣ⁻¹X)⁻¹X^TΣ⁻¹Y is an unbiased estimator of β and compute E_βh

(βb−β)(βb−β)^Ti . (ii) Letβe=LY be any arbitrary unbiased estimator ofβ.

Show that

Eβ

h

(βe−β)(βe−β)^T i

− Eβ

h

(βb−β)(βb−β)^T i

is non-negative definite.

Hint: A symmetric and positive definite (n × n)-matrix M can be represen- ted as M = Pn

i=1λ_ie_ie^T_i , where λ₁, . . . , λ_n are the (positive) eigenvalues and e1, . . . , en are corresponding eigenvectors with e^T_i ej = 0 for i 6= j. Then M^1/2 :=

Pn i=1

√λ_ie_ie^T_i and M^−1/2 :=Pn

i=1(1/√

λ_i)e_ie^T_i. To prove (ii), use the fact that

LΣ^1/2 − (X^TΣ⁻¹X)⁻¹X^TΣ^−1/2

LΣ^1/2 − (X^TΣ⁻¹X)⁻¹X^TΣ^−1/2 T

is non-negative definite.

2) (i) Let

X =











1 v1 v²₁ · · · v₁^k 1 v₂ v²₂ · · · v₂^k ... ... ... . .. ...

1 v_n v_n² · · · v_n^k









 .

Prove thatX^TX is regular if the set {v₁, . . . , v_n} contains at least k+ 1 different values.

Hint: Choose c= (c₁, . . . , c_k+1)^T 6= 0_k+1 := (0, . . . ,0)^T and compute c^TX^TXc.

(ii) Let

X =











v₁ v²₁ · · · v₁^k v2 v²₂ · · · v₂^k ... ... . .. ...

v_n v_n² · · · v_n^k









 .

Prove that X^TX is regular if the set {v₁, . . . , v_n} contains at least k different non-zero values.

Hint: Consider the matrix

Xe =















1 0 0 . . . 0 1 v₁ v²₁ · · · v₁^k 1 v2 v²₂ · · · v₂^k ... ... ... . .. ...

1 v_n v_n² · · · v_n^k













 .

3) Consider the linear regression model Yi =β1+xiβ2+εi, i= 1, . . . , n, where ε1, . . . , εn

are i.i.d. with ε_i ∼ N(0, σ²). Let βbbe the least squares estimator ofβ.

(i) Suppose that x_i 6=x_j, for some (i, j).

Compute E[(βb_i −β_i)²], for i= 1,2.

Hint: The inverse of a regular matrix

a b c d

is given by _ad−bc¹

d −b

−c a

. (ii) Suppose thatx₁, . . . , x_ncan be chosen by an experimenter, wherex_i ∈[−1,1] and

n≥2 is even.

Which choice of x₁, . . . , x_n minimizes E[(βb_i − β_i)²]? (Take into account that x₁, . . . , x_n have to be chosen such that x_i 6= x_j, for some (i, j); otherwise the least squares estimator is not uniquely defined.)