based on a prizewinning paper by

talk by

Sebastian Limbach

based on a prizewinning paper by

Thomas Brox, Andrés Bruhn, Nils Papenberg and Joachim Weickert

Motivation

A novel variational approach Coarse-to-fine warping

Summary

O u t l i n e

O u t l i n e

What is optical flow estimation?

displacement field between two images correspondence problem

understanding of details of existing methods and quality of new methods have increased dramatically

M o t i v a t i o n

M o t i v a t i o n

M o t i v a t i o n

M o t i v a t i o n

Applications

Motion planning e.g. collision avoidance Computer Vision

Video compression (although mpeg uses much simpler algorithms)

...

M o t i v a t i o n

M o t i v a t i o n

The new approach...

...combines concepts from several methods for optical flow estimation

...avoids many shortcomings of those methods ...outperforms all methods from the literature so far

...is very robust with respect to noise and parameter variations

M o t i v a t i o n

M o t i v a t i o n

Basic Ideas

Given: Image sequence

Required: Displacement vectors

T h e v a r i a t i o n a l a p p r o a c h T h e v a r i a t i o n a l a p p r o a c h

I  x ,y ,t 

 u  x , y , t  , v  x , y , t  , 1  ^T

Variational method

different constraints on displacement vectors ->

energy functional

looking for functions u and v which minimize the energy functional

T h e v a r i a t i o n a l a p p r o a c h

T h e v a r i a t i o n a l a p p r o a c h

grey value constancy assumption

linearisation by Taylor Series expansion yields the famous optical flow constraint

no linearisation in this approach gradient constancy assumption again no linearisation

C o n s t r a i n t s C o n s t r a i n t s

I  x ,y ,t = I  x  u , y  v , t  1  I _x u  I _y v  I _t = 0

∇ I  x , y , t =∇ I  x  u , y  v , t  1 

Smoothness assumption

no interaction between neighbouring pixels so far

leads to problems where the flow field vanishes or cannot fully be determined (aperture

problem )

further assumption: (piecewise) smoothness of the flow field

C o n s t r a i n t s

C o n s t r a i n t s

E n e r g y f u n c t i o n a l E n e r g y f u n c t i o n a l

 x : = x , y , t  ^T

Putting it all together

The energy functional integrates over the whole image domain Ω

E _Data  u , v = ∫ _ ^{  ∣ I  x   w − I  x  ∣ 2  ∣ ∇ I  x  w −∇ I  x  ∣ 2  d  x}

w :  = u , v ,1  ^T

E n e r g y f u n c t i o n a l E n e r g y f u n c t i o n a l

Putting it all together

grey value constancy assumption

E _Data  u , v = ∫ _ ^{  ∣ I  x   w − I  x  ∣ 2  ∣ ∇ I  x  w −∇ I  x  ∣ 2  d  x}

E n e r g y f u n c t i o n a l E n e r g y f u n c t i o n a l

≥ 0

Putting it all together

gradient constancy assumption

, weight between both assumptions

E _Data  u , v = ∫ _ ^{  ∣ I  x   w − I  x  ∣ 2  ∣ ∇ I  x  w −∇ I  x  ∣ 2  d  x}

E n e r g y f u n c t i o n a l E n e r g y f u n c t i o n a l

  s ² =  ^{s 2  2 , = 0.001}

Putting it all together

to get a more robust energy, an increasing, convex function is applied

E _Data  u , v = ∫ _ ^{  ∣ I  x   w − I  x  ∣ 2  ∣ ∇ I  x  w −∇ I  x  ∣ 2  d  x}

E n e r g y f u n c t i o n a l E n e r g y f u n c t i o n a l

∇ : =∂ x , ∂ y  ^T , ∇ ₃ : =∂ x , ∂ y , ∂ t  ^T Putting it all together

smoothness assumption

penalising the total variation of the flow field spatial / spatio-temporal gradient

E _Smooth  u , v = ∫ _ ^{ } ∣ ^∇ 3 u ∣ ^{2 } ∣ ^∇ 3 v ∣ ^{2  d  x}

E n e r g y f u n c t i o n a l E n e r g y f u n c t i o n a l

Putting it all together

regularisation parameter α > 0

goal is to find functions u and v that minimise this energy

the calculus of variations states that minimising functions must fulfil the Euler-Lagrange

equations

E  u , v = E _Data  E _Smooth

M i n i m i s a t i o n M i n i m i s a t i o n

Some abbreviations

I _x : =∂ _x I  x  w  I _y : =∂ _y I  x   w 

I _z : = I  x  w − I  x  I _xx : =∂ _xx I  x   w 

I _xy : =∂ _xy I  x   w  I _yy : =∂ _yy I  x   w 

I _xz : =∂ _x I  x  w −∂ _x I  x 

I _yz : =∂ _y I  x   w −∂ _y I  y 

M i n i m i s a t i o n M i n i m i s a t i o n

Euler-Lagrange-Equations

non-convex functions non-linear functions

 '  I _z ²  I ² _xz  I _yz ² ⋅ I _x I _z  I _xx I _xz  I _xy I _yz 

− div  '  ∣ ^∇ 3 u ∣ ^{2 } ∣ ^∇ 3 v ∣ ^{2  ∇} 3 u = 0

 '  I _z ²  I ² _xz  I _yz ² ⋅ I _y I _z  I _yy I _yz  I _xy I _xz 

− div  '  ∣ ^∇ 3 u ∣ ^{2 } ∣ ^∇ 3 v ∣ ^{2  ∇} 3 v = 0

N u m e r i c a l A p p r o x i m a t i o n N u m e r i c a l A p p r o x i m a t i o n

Fixed point iteration

index k indicates iteration step w ^k = (u ^k , v ^k , 1) ^T

starting with initial values, we are looking for new values w ^k+1 in each step

still non-linear

 '  I _z ^{k  1}  ²   I _xz ^{k  1}  ²  I ^k _yz ^{ 1}  ² ⋅ I _x ^k I _z ^{k  1}  I _xx ^k I _xz ^{k  1}  I _xy ^k I _yz ^{k  1} 

− div  '  ∣ ^∇ ₃ ^{u k  1} ∣ ^{2 } ∣ ^∇ ₃ ^{v k  1} ∣ ^{2  ∇} ₃ ^{u k  1 = 0}

N u m e r i c a l A p p r o x i m a t i o n N u m e r i c a l A p p r o x i m a t i o n

Taylor Series Expansion

first order expansions for I _?z are applied

linearisation in the numerical scheme instead of the model assumptions

unknowns splitted:

u ^{k  1} = u ^k  du ^k , v ^{k  1} = v ^k  dv ^k I _z ^{k 1} ≈ I _z ^k  I _x ^k du ^k  I _y ^k dv ^k

I _xz ^{k  1} ≈ I ^k _xz  I _xx ^k du ^k  I ^k _xy dv ^k

I _yz ^{k 1} ≈ I ^k _yz  I _xy ^k du ^k  I ^k _yy dv ^k

N u m e r i c a l A p p r o x i m a t i o n N u m e r i c a l A p p r o x i m a t i o n

Second fixed point iteration

to remove non-linearity in ψ', a second fixed point iteration is applied

applied to discrete image data we end up at a linear system of equations in the unknown

increments du ^k,l+1 and dv ^k,l+1

can be solved with common numerical methods

(Gauss-Seidel, SOR iterations, ...)

W a r p i n g W a r p i n g

Warping

pyramid of images with a scaling factor η

algorithm starts at the coarsest scale with initial values

inner fixed point iteration on this scale gives dw ⁰ and therefore w ¹ = w ⁰ + dw ⁰

on the next finer scale the second frame is warped by w ¹ using bilinear interpolation

the increments du ¹ and dv ¹ are computed and

one gets w ² etc.

W a r p i n g W a r p i n g

I

(x,y,t)

w

I

(x,y,t+1)

coarse

initial values

W a r p i n g W a r p i n g

I

(x,y,t)

w

I

(x,y,t+1) dw

initial values

coarse

W a r p i n g W a r p i n g

I

(x,y,t)

w

I

(x,y,t+1) dw

w

+

= initial values

coarse

W a r p i n g W a r p i n g

I

(x,y,t+1) I

(x,y,t)

w

I

(x,y,t+1) dw

w

+

= coarse

fine

initial values

I

(x+u

,y+v

,t+1)

warping

W a r p i n g W a r p i n g

I

(x,y,t+1) I

(x,y,t)

w

I

(x,y,t+1) dw

w

+

=

I

(x+u

,y+v

,t+1) I

(x,y,t)

dw

coarse

fine

initial values

warping

W a r p i n g W a r p i n g

What has been reached?

by splitting the unknowns and using the coarse- to-fine warping we have to solve a difference problem at each scale

we end up solving a series of convex problems instead of the non-convex initial problem

this strategy is needed to avoid local minima

the warping strategy is theoretically justified

W a r p i n g W a r p i n g

Connection to previous approaches (1)

many optic flow methods use the linearised

optical flow constraint (vs. non-linearised OFC) convex energy functional, but the model cannot handle large displacements well (vs. non-

convex energy functional)

therefore the coarse-to-fine warping is applied -> only the (small) difference dw has to be

computed

at each scale, a separate linearised energy

functional is minimised

W a r p i n g W a r p i n g

Connection to previous approaches (2)

similar results in both approaches

both approaches result in equivalent Euler- Lagrange equations

-> the “linearised OFC / warping” - approach

minimises a non-linearised constancy assumption

by means of fixed point iteration on w

E v a l u a t i o n E v a l u a t i o n

Benchmark results (1)

Source: T. Brox, A. Bruhn, N. Papenberg, J. Weickert: “High Accuracy Optical

Flow Estimation Based on a Theory for Warping”

E v a l u a t i o n E v a l u a t i o n

Benchmark results (2)

Source: T. Brox, A. Bruhn, N. Papenberg, J. Weickert: “High Accuracy Optical

Flow Estimation Based on a Theory for Warping”

E v a l u a t i o n E v a l u a t i o n

Benchmark results (3)

Source: T. Brox, A. Bruhn, N. Papenberg, J. Weickert: “High Accuracy Optical

Flow Estimation Based on a Theory for Warping”

S u m m a r y S u m m a r y

The new variational approach

uses non-linearised model assumptions linearisations in the numerical scheme

coarse-to-fine warping applied for better

approximation of the global minimum of the

non-convex energy functional

S u m m a r y S u m m a r y

Warping Methods

if the optical flow constraint is linearised, the functional is easier to solve

to deal with larger displacements, a coarse-to- fine warping is applied

one can show that this strategy in fact solves a non-linearised optical flow constraint

-> theoretical foundation for the warping

strategy

R e f e r e n c e s R e f e r e n c e s

T. Brox, A. Bruhn, N. Papenberg, J. Weickert: “High Accuracy Optical Flow Estimation Based on a

Theory for Warping”, Proc. 8 ^th European Conference on Computer Vision, vol. 4, pp. 25-36, Berlin,

Heidelberg: Springer Verlag 2004.

E. Mémin, P. Pérez: “Hierarchical estimation and

segmentation of dense motion fields”, International Journal of Computer Vision, vol. 46(2), pp. 129-155, 2002.

A. Bruhn: “Correspondence Problems in Computer Vision”, lecture notes, Summer Term 2007.

B. Burgeth: “Image Processing and Computer

Vision”, lecture notes, Winter Term 2006/2007.