Intelligent Systems Discriminative Learning, Neural Networks

(1)

WS2014/2015, 28.01.2015

Intelligent Systems

Discriminative Learning, Neural Networks

Carsten Rother, Dmitrij Schlesinger

(2)

28.01.2015 Intelligent Systems: Discriminative Learning, Neural Networks

1. Discriminative learning

2. Neurons and linear classifiers:

1) Perceptron-Algorithm

2) Non-linear decision rules

3. Feed-Forward Neural Networks:

1) Architecture

2) Modeling abilities

3) Learning — Error Back-Propagation

2

Outline

(3)

28.01.2015

Intelligent Systems: Discriminative Learning, Neural Networks 3

Discriminant Functions

• Let a parameterized family of probability distributions be given.

• Each particular p.d. leads to a classifier (for a fixed loss).

• The final goal is the classification (applying the classifier).

Generative approach:

1. Learn the parameters of the probability distribution (e.g. ML) 2. Derive the corresponding classifier (e.g. Bayes)

3. Apply the classifier for test data Discriminative approach:

1. Learn the unknown parameters of the classifier directly 2. Apply the classifier for test data

If the family of classifiers is “well parameterized”, it is not necessary to consider the underlying probability distribution at all !!!

(4)

28.01.2015 Intelligent Systems: Discriminative Learning, Neural Networks

Assume, we know the probability model: two Gaussians of equal variance, i.e. , ,

Assume, at the end we want to do the Maximum A-posteriori Decision Consequently (see the previous lecture), the classifier is a separating plane (a linear classifier).

Let us search during the learning just for a “good” separating plane instead to learn the unknown parameters of the underlying probability model

4

Example: two Gaussians

k 2 {1, 2} x 2 Rⁿ p(k, x) = p(k) · p(x|k)

p(x|k) = 1 (p

2⇡ )ⁿ exp

 ||x µ_k||² 2 ²

(5)

28.01.2015

Neuron

Hunan vs. Computer

(two nice pictures from Wikipedia)

(6)

28.01.2015

Neuron (McCulloch and Pitt, 1943)

Input:

Weights:

Activation:

Output:

Step-function

Sigmoid-function (differentiable!!!)

x 2 Rⁿ w 2 Rⁿ b 2 R

y = f(y⁰ b) = f(hw, xi b)

f(y⁰) =

⇢ 1 if y⁰ > 0 0 otherwise

f(y⁰) = 1

1 + exp( y⁰) hx, wi 7 b

(7)

28.01.2015

Geometric interpretation

Let be normalized, i.e.

is the length of the projection of onto .

Separating plane:

Neuron implements a linear classifier hx, wi = ||x|| · ||w|| · cos

w ||w|| = 1

) ||x|| · cos

x w

hx, wi = const

(8)

28.01.2015

A special case — boolean functions

Input:

Output:

Find and so, that

Disjunction, other boolean functions, but XOR x = (x₁, x₂), x_i 2 {0, 1}

y = x₁&x₂

w b step(w₁x₁ + w₂x₂ b) = x₁&x₂

x₁ x₂ y

0 0 0

0 1 0

1 0 0

1 1 1

w₁ = w₂ = 1, b = 1.5

(9)

28.01.2015

The (one possible) learning task

Given: training data

Find: so that for all

For a step-neuron: system of linear inequalities

Solution is not unique in general !!!

L = (x¹, y¹), (x², y²) . . . (x^L, y^L) , x^l 2 Rⁿ, y^l 2 {0, 1} w 2 Rⁿ, b 2 R f(hx^l, wi b) = y^l

l = 1, . . . , L

⇢ hx^l, wi > b if y^l = 1 hx^l, wi < b if y^l = 0

(10)

28.01.2015

“Preparation 1”

Eliminate the bias:

The trick − modify the training data

Example in 1D

non-separable without the bias separable without the bias x = (x₁, x₂, . . . , x_n) ) x˜ = (x₁, x₂, . . . , x_n, 1)

w = (w₁, w₂, . . . , w_n) ) w˜ = (w₁, w₂, . . . , w_n, b) hx^l, wi 7 b ) hx˜^l, w˜i 7 0

(11)

28.01.2015

“Preparation 2”

Remove the sign:

The trick − the same

All in all:

ˆ

x^l = ˜x^l for all l with y^l = 1 ˆ

x^l = x˜^l for all l with y^l = 0

⇢ hx^l, wi > b if y^l = 1

hx^l, wi < b if y^l = 0 ) hxˆ^l, w˜i > 0

(12)

28.01.2015

Perceptron Algorithm (Rosenblatt, 1958)

Solution of a system of linear inequalities:

1. Search for an equation that is not satisfied, i.e.

2. If not found − Stop else update

go to 1.

• The algorithm terminates in a finite number of steps if a solution exists (the training data are separable)

• The solution is a convex combination of the data points hx^l, wi  0

w^new = w^old + x^l

(13)

28.01.2015

An example problem

Consider another decision rule for a real valued feature :

It is not a linear classifier anymore but a polynomial one.

The task is again to learn the unknown coefficients given the training data

Is it also possible to do that in a “Perceptron-like” fashion ?

x 2 R

a_nxⁿ + a_n ₁xⁿ ¹ + . . . + a₁x + a₀ = X

i

a_ixⁱ 7 0

a_i

(x^l, y^l) . . . , x^l 2 R, y^l 2 {0, 1}

(14)

28.01.2015

An example problem

The idea: reduce the given problem to the Perceptron-task.

Observation: although the decision rule is not linear with respect to , it is still linear with respect to the unknown coefficients

The same trick again − modify the data:

In general, it is very often possible to learn non-linear decision rules by the Perceptron algorithm using an appropriate transformation of the

input space (more examples at seminars).

Extension — Support Vector Machines, Kernels

x a_i

w = (a_n, a_n ₁, . . . , a₁, a₀)

˜

x = (xⁿ, xⁿ ¹, . . . , x, 1)

) X

i

a_ixⁱ = hx, w˜ i

(15)

28.01.2015

Many classes

Before: two classes − a mapping Now: many classes − a mapping How to generalize ? How to learn ?

Two simple (straightforward) approaches:

The first one: “one vs. all” − there is one binary classifier per class, that separates this class from all others.

The classification is ambiguous in some areas.

Rⁿ ! {0, 1}

Rⁿ ! {1, 2, . . . , K}

(16)

28.01.2015

Many classes

Another one:

“pairwise classifiers” − there is a classifier for each class pair

The goal:

• no ambiguities,

• parameter vectors

Less ambiguous, better separable.

However:

binary classifiers instead of in the previous case.

Fisher Classifier

K(K 1)/2 K

K )

(17)

28.01.2015

Fisher classifier

Idea: in the binary case the output is the “more likely to be 1” the greater is the scalar product → generalization:

The input space is partitioned into the set of convex cones.

Geometric interpretation (let be normalized)

Consider projections of an input vector onto vectors

y hx, wi

y = arg max

k hx, w_ki

w_k

x

(18)

28.01.2015

Feed-Forward Neural Networks

Output level i-th level

First level Input level

Special case: , Step-neurons – a mapping y_ij = f⇣X

j⁰

w_ijj⁰y_i _1j⁰ b_ij⌘

m = 1 Rⁿ ! {0, 1}

(19)

28.01.2015

What we can do with it ?

One level – single step-neuron – linear classifier

(20)

28.01.2015

What we can do with it ?

Two levels, “&”-neuron as the output – intersection of half-spaces

If the number of neurons is not limited, all convex subspaces can be implemented with an arbitrary precision.

(21)

28.01.2015

What we can do with it ?

Three levels– all possible mappings as union of convex subspaces:

Three levels (really even less) are enough to implement all possible mappings !!!

Rⁿ ! {0, 1}

(22)

28.01.2015

Learning – Error Back-Propagation

Learning task:

Given: training data

Find: all weights and biases of the net.

Error Back-Propagation is a gradient descent method for Feed- Forward-Networks with Sigmoid-neurons

First, we need an objective (error to be minimized)

Now: derive, build the gradient and go.

F (w, b) = X

l

k^l y(x^l; w, b) ² ! min

w,b

(x^l, k^l), . . . , x^l 2 Rⁿ, k^l 2 R

(23)

28.01.2015

Error Back-Propagation

We start from a single neuron and just one example . Remember:

Derivation according to the chain-rule:

F (w, b) = (k y)²

y = 1

1 + exp( y⁰) y⁰ = hx, wi = X

j

x_jw_j

@F (w, b)

@w_j = @F

@y · @y

@y⁰ · @y⁰

@w_j =

= (y k) · exp( y⁰)

(1 + exp( y⁰))² · x_j = · d(y⁰) · x_j

(x, k)

(24)

28.01.2015

Error Back-Propagation

In general: compute “errors” at the -th level from all -s at the -th level – propagate the error.

The Algorithm (for just one example ):

1. Forward: compute all and (apply the network), compute the output error ;

2. Backward: compute errors in the intermediate levels:

3. Compute the gradient and apply it:

For many examples – just sum them up.

(i + 1)

i

(x, k)

y y⁰

n = y_n k

ij = X

j⁰

i+1j⁰ · d(y_i+1j⁰ 0) · w_i+1j⁰_j

@F

@w_ijj⁰ = _ij · d(y_ij⁰ ) · y_i _1j⁰

(25)

28.01.2015

A special case – Convolutional Networks

Local features – convolutions with a set of predefined masks (more detailed in lectures “Computer Vision”).

(26)

28.01.2015

Convolutional Networks

Yann LeCun, Koray Kavukcuoglu and Clement Farabet Convolutional Networks and Applications in Vision

(27)

28.01.2015

Neural Networks — Summary

1. Discriminative learning: learn decision strategies directly without to consider the underlying probability model at all

3. Neurons are linear classifiers.

4. Learning by the Perceptron-algorithm.

5. Many non-linear decision rules can be transformed to linear ones by the corresponding transformation of the input space.

6. Classification into more than two classes is possible either by naïve approaches (e.g. by a set of “one-vs-all” simple classifiers) or more principled by Fisher-Classifier.

5. Feed-Forward Neural Networks implement (arbitrary complex) decision strategies.

6. Error Back-Propagation for learning (gradient descent).

7. Some special cases are useful in Computer Vision.