Prob., distributions and identitites

AML Cheat Sheet

Prob., distributions and identitites

SVD X=UDV^>,U∈R^n×d,V∈R^d×d Cauchy-Schwarz

P

iuivi

2 ≤P

j|uj|²P

k|vk|² Cov. (univariate) Cov[x, y] =E[(x−E[x])(y−E[y]) Cov. (mult’vari.) Cov[x, y] =Ex,y[xy^>]−Ex[x]Ey[y]^>

V[x±y] =V[x] +V[y]±2Cov[x, y]

Vx[Ax+b] =Vx[Ax] =AVx[x]A^>

Sum Rule P(X=x) =P

yp(X=x, Y =y) Conditional P(X|Y) =P(X, Y)/P(Y) Bayes’ Rule P(Y|X) = ^P(X|Y_P(X)^)P(Y) Multi Gaussian

p(x|µ,Σ) = ((2π)^d· |Σ|)^−1/2exp(−¹₂(x−µ)^>Σ⁻¹(x−µ)) P(xi|c, µc, σ²c) = √¹

2πσ²_cexp

−^(xi_2σ^−µ2^c)2 c

Markov P[X≥]≤E[X]/, X≥0, >0 Hoeffding L. E[e^sX]≤exp(s²(b−a)²/8) Hoeffding Thm

P[Sn−ESn≥t]≤exp(−2t²/P

(bi−ai)²) P[Sn−ESn≤ −t]≤exp(−2t²/P

(bi−ai)²) ifSn→X¯n t→n

Kernels

•Gaussian (RBF) kernel:k(x, x⁰) = exp(−kx−x⁰k²2/h²)

•Dimensionofk(x,x⁰) = (x^>x⁰+c)^dis ^N+d_d

Properties

Symmetry k(x, x⁰) =k(x⁰, x) Pos semi-def R

Ωk(x⁰,x)f(x)f(x⁰)dxdx⁰, f∈L2,Ω⊆R^d construct θ:xi7→(√

λtvit)ⁿ_t=1

Identities

Addition k(x, x⁰) =k1(x, x⁰) +k2(x, x⁰) Multiply k(x, x⁰) =k1(x, x⁰)k2(x, x⁰) Scalar k(x, x⁰) =ck1(x, x⁰) forc >0 Transform k(x, x⁰) =f(k1(x, x⁰))

f polynom with positive coeff. or exp.

func. multiply f(x)k1(x, x⁰)f(x⁰) for anyf

Risks

Q(y, f(x)) =







(y−f(x))² quadratic loss (regr.) I^{y6=f(x)} 0-1 loss (class.)

exp(−βyf(x)) exponential loss (class.) Cond. Exp. Risk R(f, X) =R

YQ(Y, f(X))P(Y|X)dY (Total) Exp. Risk R(f) =E^X[R(f, X)]

Emp. Error R(f, X) =ˆ ¹_nPn

i=1Q(yi, f(Xi))

Maximum Likelihood Estimators

θˆ^{M L} ∈arg maxP(X |θ)^i.i.d.= Qn

i=1P(xi|θ) Definitions:

Bias bias(ˆθn) =E[ˆθn]−θ

bias[ ˆf(x)] =E^Df(x)ˆ −E[Y|x]

Consistency ∀, P[|θˆn−θ|> ]^n→∞→ 0 Score Λ(θ) := ^∂log_∂θ^P(x|θ)

• EX|θ[Λ] = 0; • EX|θ[Λ] =_∂θ^∂bθˆ+ 1 Fisher Information I(θ) =V

h_{∂logP(x|θ)}

∂θ

i

Asymptotical efficiency limn→∞(V[ˆθn(x1, ..., xn)]I(θ))⁻¹= 1

Results

Rao-Cramer E^X|θ[(ˆθ−θ)²]≥(1 +_∂θ^∂bθˆ)²/I⁽ⁿ⁾(θ) +b²_ˆ

θ

ML converg. √

n(ˆθn^{M L}−θ0)→ N^D (0, J⁻¹(θ0)I(θ0)J⁻¹(θ0)) J(θ) =−E[^∂2log_∂θ∂θ^P(x|θ)_T ]

ML consist. θˆ^{M L}n

→p θ0

If ˆθ^{M L}forθ g(ˆθ^{M L}) forg(θ)

Bayesian Learning

Maximum a Posteriori θˆ∈arg maxθp(x|θ)p(θ) Prediction:

p(X=x|X) = R

p(x|θ)p(θ|X)dθ Rec. Bayesian Est.

p(θ|Xⁿ) = R^{p(xn|θ)p(θ|Xn−1)} p(xn|θ)p(θ|Xⁿ⁻¹)dθ

Regression

ε∼ N(0, σ) y=Xβ+ε

Least-square fit βˆ= (X^TX)⁻¹X^>y f^∗= opt. estimator βˆ∼ N(β,(X^TX)⁻¹σ²) For ˆθ=c^Tyunbiased: V[a^Tβ]ˆ ≤V[c^Ty]

MSE EDEX,Y( ˆf(X)−Y)²= variance +EXED( ˆf(X)−EDf(X))ˆ ² bias² +E^X(E^Df(X)ˆ −E^Y[Y|X])² noise +EX,Y(Y −EY[Y|X])² Ridge βˆ= (X^TX+λI)⁻¹X^>y Gen. Reg. βˆ= arg minβRSS(β) +λPd

j=1|βj|^q bias( ˆf) =E[ ˆf]−f^∗ V ar[ ˆf] =E[( ˆf−E[ ˆf])²

Bayesian Linear Regression

M odel: Y =Xβ+, ∼ N(0, σ²) Likelihood: p(Y|X, β, σ²) =N(Xβ, σ²)

P rior: p(β|Λ) =N(0,Λ⁻¹) P osterior: p(β|X,y,Λ) =N(µβ,Σβ)

µβ= (X^TX+σ²Λ)⁻¹X^TyΣβ=σ²(X^TX+σ²Λ)⁻¹ (β−µβ)^>Σ⁻¹_β (β−µβ) =β^>Σ⁻¹_β β−2β^>Σ⁻¹_β µβ+µ^>_βΣ⁻¹_β µβ

Conditioning a Gaussian: p(xa|xb) =N(xa|µa|b,Σa|b) µa|b=µa+ ΣabΣ⁻¹_bb(xb−µb),Σa|b= Σaa−ΣabΣ⁻¹_bbΣba

Λaa= (Σaa−ΣabΣ⁻¹_bbΣba)⁻¹,Λab=−ΛaaΣabΣ⁻¹_bb

Gaussian Process

Joint distribution of [y, yn+1] is given by

y|X, σ²∼ N(0,X^TΛ⁻¹X+σ²I), kernelized version:

p y

yn+1

|xn+1,X, σ²

=N

0,

Cn k k^T c

Cn=K+σ²In c=k(xn+1, xn+1) +σ² k=k(xn+1,X) K=k(X,X)

p a1

a2

=N a1

a2

| u1

u2

,

Σ11 Σ12

Σ21 Σ22

a1,u1∈R^e; Σ11∈R^e×e PSD; Σ12∈R^e×f PSD a2,u2∈R^d; Σ22∈R^f×f PSD; Σ2,1∈R^f×e PSD

Predictive density:

p(yn+1|xn+1,X,y) =N(µn+1, σ_n+1² ) µn+1=k^TC⁻¹n y σ²n+1=c−k^TC⁻¹n k

Numerical Estimation Techniques Cross-Validation

fitted models fˆ^−ν∈arg minf∈F 1

|Z\Z_ν|

P

i6∈Z_ν(yi−f(xi))² pred. error Rˆ^cv=_n¹P

i≤n(yi−fˆ^−κ(i)(xi))² unbiasedness ^N(k−1)_k ≥m(exam2018 k-fold CV)

Bootstrap

It works if R^strn ( ˆF ,Fˆ^∗)−R^strn (F,Fˆ)→^P 0 bootst. avg risk: Rˆ^∗=_B¹_n¹PB

b=1

Pn

i=1Q(yi,fˆ^∗b(xi)) Sols for overlap: C⁻ⁱ:={j∈[B] :xi6∈ Z^∗j}

Rˆ⁽¹⁾=_n¹Pn i=1

1

|C⁻ⁱ|

P

b∈C⁻ⁱl(yi, f^∗b(xi)) ( ˆR^∗fit on trainset) Rˆ^.632= 0.368 ˆR^∗+ 0.632 ˆR⁽¹⁾

Rˆ^(0.632+)= (1−w)ˆ ∗Rˆ^∗+ ˆwRˆ⁽¹⁾ ˆ

w= ^0.632

1−0.368 ˆG,Gˆ= ^{Rˆ(1)−Rˆ∗}

ˆ

γ−Rˆ^∗ ,ˆγ=_n¹₂PN i=1

PN

j=1l(yi,fˆ(xj))

Jackknife

Sˆn−1⁻ⁱ (x1, ..., xi−1, xi+1, ..., xn) = ˆSn−1(x1, ..., xi−1, xi+1, ..., xn) S˜n:= 1

n

n

X

i=1

Sˆn−i⁻ⁱ bias^{J K} := (n−1)( ˜Sn−Sˆn) (Jackknife) Debiasd estm. Sˆ^{J K} = ˆSn−bias^{J K}

Tests and criteria

LetX1, ..., Xn∼Q(x) i.i.d. andH0:Q=P0 H1:Q=P1. Test g(x1, ..., xn) =

(0(accepted) ^P_P^0(x1,...,xn)

1(x₁,...,x_n) > T 1(rejected) ^P_P^0(x1,...,xn)

1(x₁,...,xn) ≤T

Thenα^∗=E⁰[g(x1, ..., xn)] andβ^∗= 1−E¹[g(x1, ..., xn)] Assume that we know the log likelihood function (loss) of the model Bayes Factor p(X|Mk)/p(X|Ml) (i)

(i)>1 takeMk p(X|Mk) =R

p(X|θk,Mk)p(θk|Mk)dθk

BIC(minimise) −2 log(ˆp(X|θˆk,Mk)) +k⁰logn Laplace approx. (k⁰= #free params inMk) logp(X|Mk) = log p(X|θˆk,Mk)−log(n)k⁰/2 +O(1) MDL −logp(X|θk)−logp(θk)

AIC −2 log(ˆp(X|θˆk)) + 2k

KL D(p||ˆp) =−R

p(x) log

ˆ p(x|θˆ_k)

p(x)

dx TIC −2 log(ˆp(X|θˆk)) + 2trace[I1(θk)J₁⁻¹(θk)]

AIC is asymptotically equivalent to LOOCV for ordinary linear regression models.

Linear Discriminant Functions

Gradient Descent ak+1=ak−ηk∇J(ak)

J(ak+1)≈J(ak) +∇J^T(ak+1−ak) +¹₂(ak+1−ak)^TH(ak+1−ak) η^{OP T} = _∇J^||∇J||_TH∇J²

Newton’s Rule ak+1=ak−H⁻¹∇J(ak) Percep loss J(a) =P

˜

x∈X˜^mc(−a^T˜x) Percep update ak+1=ak+ηkP

˜ x∈X˜^mcx˜ γ= min_i∈X˜^mc(ˆa^Tx˜i) β²= max_i∈X˜^mc||˜xi||² Max steps (γ)⁻²β²||ˆa||²

Bayesian view:

Prior P(Y =y) =πy

Posterior density p(y|x) = P^πyp(x|y) zπzpz(x)

c(x) =

(y P

zL(z, y)p(z|x) =minρ≤kP

L(z, ρ)p(z|x)≤d D else

Outlier classi.: πOpO(x)≥max{(1−d)p(x),maxzπzpz(x)}

Fisher’s Linear Discriminant Analysis (LDA) sample avg mα=_n¹

α

P

x∈X_αx, nα=|Xα| projected avg m˜α=_n¹

α

P

x∈X_αw^Tx=w^Tmα

class scatter Σα=P

xα∈X_α(x−mα)(x−mα)^T within scatter ΣW =P

1≤α≤kΣα

projected scatter Σ˜α=w^TΣαw

Fisher’s Separation J(w) = ^wT(m1−m_wΣ^{2)(m1−m2)Tw)}

Ww

yields w∝Σ⁻¹_W(m1−m2)

Mean scatter ΣB= (m1−m2)(m1−m2)^T resultΣ⁻¹_WΣBw =^wTΣBw

w^TΣwww

Lagrangian Optimization

minf(w) w∈Ω⊆R^d s.t. gi(w)≤0 1≤i≤k

hj(w) = 0 1≤j≤m L(w,α,β) =f(w) +

k

X

i=1

αigi(w) +

m

X

j=1

βjhj(w)

∂L

∂w|w=w^∗= 0 max

α,β θ(α,β) withθ(α,β) = inf

w L(w,α,β) s.t. αi≥0

Duality gap ∆ :=L(w^∗, α^∗, β^∗)−θ(α^∗, β^∗)

Strong duality, i.e. convex obj. fctf & convex domain, then the duality gap is zero.

KKT Conditions: f∈C¹ andgi, hi are affine, thenw^∗is an optimum ifα^∗,β^∗satisfy

∂L(w^∗,α^∗,β^∗)

∂w = 0∂L(w^∗,α^∗,β^∗)

∂β = 0

α^∗_igi(w^∗) = 0,gi(w^∗)≤0, α^∗_i ≥0

SVM

Soft Margin Geometric problem formulation Primal minw,ξ1

2w^Tw+CPn i=1ξi

zi(w^Ty_i+w0)≥1−ξi ξi≥0 Dual maxαP

i≤nαi−¹₂P

i≤n

P

j≤nαiαjzizjy^T_iy^T_j C≥αi≥0,P

i≤nziαi= 0

Solution w₀^∗= (maxi:z_i=−1w^∗Ty_i+ mini:z_i=1w^∗Ty_i)/2 w^∗=P

i∈SVα^∗iziy_i g^∗(y) =P

i∈SVziα^∗iy^T_iy+w0^∗

By the KKT condition,ξi(αi−C) = 0, non-zero slack variable can only occur ifαi=C.

The optimal margin is given by: w^>w=P

i∈SVα^∗_i Multi-class SVM:w^T = (w^T₁, ...,w^T_n).

Primal minw,ξ1

2w^Tw+CP

i≤nξi

ξi≥0 (w^T_ziy_i+wz_i,0)−maxz6=z_i(w^T_zy_i+wz,0)≥1−ξi

Structured SVM:

Primal minw,ξ1

2w^Tw+CP

i≤nξi s.t.ξi≥0 w^TΨ(zi,y_i) −maxz6=z_i[∆(z, zi) +w^TΨ(z,y_i)]≥ −ξi

w^TΨ(zi,y_i) −w^TΨ(z,y_i)≥∆(z, zi)−ξi ∀i,∀z6=zi

Dualminw,ξ −¹₂Pn

i=1

Pn j=1

P

z_k∈KαikαjkΨi(zk)^>Ψj(zk) +Pn

i=1

P

z_k∈Kαik∆i(zk)

s.t. C≥P

z_k∈Kαik≥0,αik≥0,∀i,∀k Prediciton h(y) = arg maxz∈K

w^>ψ(z,y)

Ensemble

If we combine different regressors: V[ ˆf(x)]≈^σ_B² bias[ ˆf(x)] =_B¹ PB

i=1bias[ ˆfi(x)]

Boosting: Weighted models and weighted training data instead of bootstrapping.

b←

n

X

i=1

w_i^(b)I{c_b(x_i)6=y_i}/

n

X

i=1

w^(b)_i

αb←log1−b

b

= log

p(y= 1|x) p(y=−1|x)

(log-odds ratio)

∀i wi←wiexp(αbI{c_b(x_i)6=y_i}) ˆ

cB(x) =sign

B

X

b=1

αbcb(x)

!

avg exp loss= _N¹ PN

i=1exp(−yiˆcB(xi)) ErrAdaBoost= exp(−h(x)sign(PB

b=1αbyb(x)))

PAC Learning

error error(h) =Px∼D[c(x)6=h(x)]

(, δ) criterion: PX,Y[R(ˆcn)≤R(c^Bayes) +]>1−δ Strong PAC L.: holds for arbitrarily small

Weak PAC L.: non-trivially large PAC learnability P[R(ˆcn)≤ε]≥1−δ

efficiently Aruns in poly time in ¹_ε and ¹_δ Results:

R(ˆc^∗_n)−infc∈CR(c) ≤2 sup_c∈C|Rˆn(c)−R(c)|

P[sup_c∈C|Rˆn(c)−R(c)|> ] ≤2Nexp(−2n²) Implying: R(c)≤Rˆn(c) +p

(logN−log(δ/2))/2n X shattered byAif

{X∩A|A∈ A} contains all subsets ofX VC DimofA= max{n:∃X s.t.|X shattered byA,|X|=n}

score score(A, X) =|{X∩ A|A∈ A}|

shattering coeff s(A, n) = maxX:|X|=nscore(A, X) IfVA>2 (VA= VC dim. of A): s(A, n)≤n^VA

P[R(c^∗n)−infc∈CR(c)> ] ≤8s(A, n) exp(−n²/32)

Non Paramteric Bayesian Methods

Beta function B(a, b) = ^Γ(a)Γ(b)_Γ(a+b) , a, b >0 Γ(a) =R∞

0 e^−xx^a−1dx Beta(x|a, b) = ^xa−1_B(a,b)^(1−x)b−1, x∈[0,1]

Dir(x|α) =

Qn k=1x^αk_k ⁻¹

B(α)

Finite Gaussian Mix p(xi|θ) =PK

k=1ρkN(xi|µk, σk)

Stick breaking process (GEM distribution):

βk∼Beta(1, α) ρk=β2(1−Pk−1

i=1ρi), k= 1,2, ...

Chinease Restaurant Process P(P) =^α|P|

α⁽ⁿ⁾

Q

τ∈P(|τ| −1)! E[#k] =PN i=1

α

α+i ∼O(αlogN) P[Customern+ 1 joins tableτ ∈ P ∪ {∅}|P] =

( _|τ|

α+n τ∈ P

α α+n ow.

Dirichlet Mixture Model

Base Measures µk∼ N(µ0, σ0)

cluster prob. ρ= (ρ1, ρ2, ...)∼GEM(α) Category assignment zi∼Categorical(ρ) Data Sample xi∼ N(µz_i, σ) De Finetti’s Theorem

p(X1, ..., Xn) =R Q

p(xi|G)dP(G) Gibbs Sampling

p(zi=k|z−i,x, α,µ)∝p(zi=k|z−i, α)

| {z }

P rior

p(xi|x−i, zi=k,z−i,µ)

| {z }

Likelihood

p(zi=k|z−i,x, α,µ) =

( _Nk,−i

α+N−1p(xi|x−i,k,µ) For existingk

α

α+N−1p(xi|µ) Otherwise p(xi,x−i,k|µ) =

Z

p(xi|µk)



 Y

j6=i

p(xj|µk)



p(µ_k|µ0, σ0)dµ_k

Gaussian-mixtures and EM estimation

Parameters θ={πc, µc, σ²c}^kc=1

k Gaussian Mix. P(xi|θ) =Pk

c=1πcP(xi|c, µc, σ²_c) Pk

c=1πc= 1

log likelihood L(X |θ) = logP(X |θ) =Pn

i=1logP(xi|θ)

=Pn

i=1logPk

c=1πcP(xi|c, µc, σ²c)

Define binary latent variablesMic∈ {0,1}whereMicindicates that xiis generated by componentc. log likelihood

L(X, M|θ) = logQn i=1

Qk

c=1(πcP(xi|c, µc, σ²_c))^Mic L(X, M|θ) =Pn

i=1

Pk

c=1Miclog(πcP(xi|c, µc, σ_c²))

Expectation over the latent va. γic:=EM|X,θ[Mic] Q(θ) :=EM|X,θ[L(X, M|θ)] =Pn

i=1

Pk

c=1γiclog(πcP(xi|c, µc, σc²)) EM-estimation algo •E-stepcomputeγic,θconst

•M-stepcomputeθ,γicconst

E-step: EM|X,θ[Mic] = 1·P(Mic= 1|xi, θ) + 0·P(Mic= 0|xi, θ)

=P(Mic= 1|xi, θ) =P(c|xi, θ) =^P(xi|c,θ)P(c|θ) P(x_i|θ)

= ^{πcP(xi|c,µc,σ}

2 j) Pk

j=1πjP(xi,|j,µ_j,σ²_j)

M-step:

(i)µc fromarg maxθQ(θ):

∂

∂µcQ(θ) = 0 =⇒ µc=

Pn i=1γ_icx_i P_n

i=1γ_ic

(ii)σcfromarg maxθQ(θ):

∂

∂σ_cQ(θ) = 0 =⇒ σ_c²=

P_n

i=1γ_ic(x_i−µ_c)² P_n

i=1γ_ic

(ii)πcfromarg maxθQ(θ): constraintP

cπc= 1 L(θ, λ) =−Q(θ) +λ(Pk

c=1πc−1) ⇒_∂π^∂

cL(θ, λ) = 0

⇔Pn

i=1γic=λπc⇔Pk c=1

Pn

i=1 γic=λPc i=1πc

⇔Pn i=1

Pk

c=1γic=Pn

i=11 =λ ⇔ πc=

P_n i=1γ_ic

n