Practical Course on Pattern Recognition

Practical Course on Pattern Recognition

Prof. Dr. Stefan Posch Dipl.Bioinform. Andr´e Gohr (andre.gohr@informatik.uni-halle.de)

Institute of Computer Science University Halle

Series 10

Exercise 10.1(4 points)

Define the back-prop learning rule for a multilayer perceptron that also allows connec- tions (edges between neurons) between non-adjacent layers. But all connections stay feed-forward.

Solution 10.1

Notation: ^kNi denotes the ith neuron in layer k. Layer k has M^k neurons at all.

w^k_Ni(^qNj) denotes the weight of the edge ingoing into neuron^kNioutgoing from neuron

qNj if any. The multilayer perceptron consists of Llayers. LayerLis the output-layer.

P^k_Ni denotes the set of ”predecessor” neurons of^kNi that are neurons having an outgo- ing edge being directed towards neuron^kNi. D^k_Nidenotes the set of ”direct descendant”

neurons of^kNi.

Further on σ(h^k_Ni) = y^k_Ni denotes the answer of neuron ^kNi being in activation state h^k_Ni. x^k_Ni(^qNj) =y^qNj denotes the input of neuron ^kNi coming from neuron ^qNj. The activation state h^k_Ni of neuron ^kNi in layer k is determined by all weighted inputs (weighted answers of ”predeseccor” neurons):

h^k_Ni = P

dNz∈P_{k N}

i

x^k_Ni(^dNz)w^k_Ni(^dNz) = P

dNz∈P_{k N}

i

y^d_Nz w^k_Ni(^dNz).

~

y_L = (y^L_N1, . . . , y^L_N

M L) is the vector of outputs of the perceptron. The error-function is denoted by E(~yL, ~t,w) =

M^L

P

i=1

E(y^L_Ni, ti,w). ~t= (t1, . . . , t_M^L) is the vector of target outputs the perceptron should give for a given data-set. wdenotes the set of all weights of the perceptron.

1

Weights of edges ingoing into any neuron^LN_i of layerL:

4w^L_Ni(^kNj) = −ε ∂E(~y, ~tL,w)

∂w^L_Ni(^kN_j) with: ^kNj ∈ P^L_Ni

= −ε

M^L

X

r=1

∂E(y^L_Nr, tr,w)

∂w^L_Ni(^kN_j)

= −ε

M^L

X

r=1

∂E(yLNr, t_r,w)

∂y^L_Nr

∂yLNr

∂h^L_Nr

∂hLNr

∂w^L_Ni(^kN_j)

= −ε

M^L

X

r=1

∂E(y^L_Nr, tr,w)

∂y^L_Nr σ⁰(h^L_Nr)

∂ P

dNz∈P_LNr

y^d_Nz w^L_Nr(^dN_z)

∂w^L_Ni(^kN_j)

∂ P

dNz∈P LNr

y_dNzw_{k N}

i(^dNz)

∂w_LN

i(^kNj) is always equal to zero unless^dN_z =^kN_j and^LN_r =^LN_i. Hence we get:

4w^L_Ni(^kN_j) = −ε ∂E(y^L_Ni, ti,w)

∂y^L_Ni σ⁰(h^L_Ni)y^k_Nj

= −ε δ^L_Niy^k_Nj (1)

with: δ^L_Ni = ^∂E(y_∂y^LNi,ti,w)

LNi

σ⁰(h^L_Ni).

Weights of edges ingoing into any neuron^L−1N_i of layerL−1:

4w^L−1_Ni(^kN_j) = −ε ∂E(~y, ~t_L,w)

∂w^L−1_Ni(^kN_j) with: ^kN_j ∈ P^L−1_Ni

= −ε

M^L

X

r=1

∂E(y^L_Nr, t_r,w)

∂y^L_Nr

∂y^L_Nr

∂h^L_Nr

∂h^L_Nr

∂w^L−1_Ni(^kN_j)

= −ε

M^L

X

r=1

∂E(y^L_Nr, t_r,w)

∂y^L_Nr σ⁰(hLNr) X

dNz∈P_LNr

∂ydNzw^L_Nr(^dN_z)

∂y^d_Nz

∂ydNz

∂w^L−1_Ni(^kN_j)

= −ε

M^L

X

r=1

δ^L_Nr X

dNz∈P_LNr

∂y^d_Nzw^L_Nr(^dN_z)

∂y^d_Nz

∂y^d_Nz

∂h^d_Nz

∂h^d_Nz w^L−1_Ni(^kNj)

= −ε

M^L

X

r=1

δ^L_Nr X

dNz∈P_LNr

w^L_Nr(^dN_z)σ⁰(h^d_Nz) X

sNt∈P_dNz

∂y^s_Ntw^d_Nz(^sN_t) w^L−1_Ni(^kN_j)

∂y_sNtw_dNz(^sNt)

wL−1Ni(^kNj) is always equal to zero unless ^sN_t = ^kN_j and ^dN_z = ^L−1N_i. Hence we get:

2

4w^L−1_Ni(^kN_j) = −ε

M^L

X

r=1

δ^L_Nr w^L_Nr(^L−1N_i)σ⁰(h^L−1_Ni)y^k_Nj

4w^L−1_Ni(^kNj) = −ε y^k_Njσ⁰(h^L−1_Ni)

M^L

X

r=1

δ^L_Nr w^L_Nr(^L−1Ni)

All outgoing edges of neuron ^L−1N_i are ingoing into neurons of layer L since layer L is the last layer. One may state this fact more general: All outgoing edges of neuron

L−1N_i are ingoing into one neuron of D^L−1_Ni.

4w^L−1_Ni(^kN_j) = −ε y^k_Nj σ⁰(h^L−1_Ni) X

zNv∈D_L−1

Ni

δ^z_Nvw_z^N_v(^L−1N_i)

= −ε y^k_Nj σ⁰(h^L−1_Ni)δ^L−1_Ni (2)

with: δ^L−1_Ni = P

zNv∈DL−1Ni

δ^zNv w^zNv(^L−1Ni).

Weights of edges ingoing into any neuron ^mN_i of layer m in analogy to the previous derivation (especially equation (2)):

4w^mNi(^kNj) =−ε y^k_Njσ⁰(h^mNi)δ^mNi with: ^kNj ∈ P^mNi (3)

with: δ^m_Ni = P

zNv∈D_mN

i

δ^z_Nvw^z_Nv(^mN_i).

3