Energy Functional Energy Functional

Mumford

Mumford - - Shah Shah

Energy Functional Energy Functional

Zoltan

Zoltan Kato Kato

http://www.cab.u

http://www.cab.u - - szeged.hu/~kato/variational/ szeged.hu/~kato/variational/

Introduction Introduction

! ! Proposed in their influential paper by Proposed in their influential paper by

! ! David David Mumford Mumford

! ! http:// http:// www.dam.brown.edu/people/mumford www.dam.brown.edu/people/mumford / /

! ! Jayant Jayant Shah Shah

! ! http://www.math.neu.edu http:// www.math.neu.edu/~shah/ /~shah/

Optimal Approximations by Piecewise Smooth Optimal Approximations by Piecewise Smooth

Functions and Associated

Functions and Associated Variational Variational Problems

Problems . . Communications on Pure and Applied Communications on Pure and Applied Mathematics, Vol. XLII, pp 577

Mathematics, Vol. XLII, pp 577 - - 685, 1989 685, 1989

Images as functions Images as functions

! ! A gray A gray - - level image represents the light level image represents the light intensity recorded in a plan domain

intensity recorded in a plan domain R R

! ! We may introduce coordinates We may introduce coordinates x x , , y y

! ! Let Let g(x,y g(x,y ) ) denote the intensity recorded at the denote the intensity recorded at the point

point ( ( x,y x,y ) ) of of R R

! ! The function The function g(x,y g(x,y ) ) defined on the domain defined on the domain R R is is called an image.

called an image.

What kind of function is g?

What kind of function is g?

! ! The light reflected by the The light reflected by the surfaces

surfaces S S

of various of various objects

objects O O

will reach the will reach the domain

domain R R in various open in various open subsets

subsets R R

! ! When When O O

appears as the appears as the background to the sides of background to the sides of O O

then the open sets then the open sets R R

and and R R

will have a will have a

common boundary (

common boundary ( edge edge ) )

! ! One usually expects One usually expects g(x,y g(x,y ) ) to be discontinuous along to be discontinuous along

this boundary

this boundary

and Associated VariationalVariationalProblemsProblems. . Communications on Pure and Applied Mathematics, Communications on Pure and Applied Mathematics, Vol. XLII, pp 577

Vol. XLII, pp 577--685, 1989685, 1989

Other discontinuities Other discontinuities

! ! Surface orientation of visible objects (cube) Surface orientation of visible objects (cube)

! ! Surface markings Surface markings

! ! Illumination (shadows, uneven light) Illumination (shadows, uneven light)

Piece

Piece - - wise smooth g wise smooth g

! ! In all cases, we expect In all cases, we expect g(x,y g(x,y ) ) to be piece to be piece - - wise wise smooth to the first approximation.

smooth to the first approximation.

! ! It is well It is well modelled modelled by a set of smooth functions by a set of smooth functions f f

defined on a set of disjoint regions

defined on a set of disjoint regions R R

covering covering R R . .

! ! Problems: Problems:

! ! Textured objects (regions perceived homogeneous but Textured objects (regions perceived homogeneous but lots of discontinuities in intensity)

lots of discontinuities in intensity)

! ! Sahdows Sahdows are not true discontinuities are not true discontinuities

! ! Partially transparent objects Partially transparent objects

! ! Noise Noise

! ! Still widely and Still widely and succesfully succesfully applied model! applied model!

Segmentation problem Segmentation problem

! ! Consists in computing a decomposition of Consists in computing a decomposition of the domain of the image

the domain of the image g(x,y g(x,y ) )

1. 1. g g varies varies smootly smootly and/or slowly within and/or slowly within R R _{i i} 2. 2. g g varies discontinuously and/or rapidly varies discontinuously and/or rapidly

across most of the boundary

across most of the boundary Γ Γ between between regions

regions R R _{i i}

U ⁿ

i

R i

R

= 1

=

Optimal approximation Optimal approximation

! ! Segmentation problem may be restated as Segmentation problem may be restated as

! ! finding optimal approximations of a general function finding optimal approximations of a general function g g

! ! by piece by piece - - wise smooth functions wise smooth functions f f , whose restrictions , whose restrictions f f

to the regions

to the regions R R

are are differentiable differentiable

! ! Many other applications: Many other applications:

! ! Speech recognition Speech recognition

! ! Sonar, radar or laser range data Sonar, radar or laser range data

! ! CAT scans CAT scans

! ! etc etc … …

Optimal segmentation Optimal segmentation

! ! Mumford Mumford and Shah studied 3 and Shah studied 3 functionals functionals which which measure the degree of match between an image measure the degree of match between an image

g(x,y

g(x,y ) ) and a segmentation. and a segmentation.

! ! First, they defined a general functional First, they defined a general functional E E (the (the famous

famous Mumford Mumford - - Shah functional): Shah functional):

! ! R R

will be disjoint connected open subsets of the planar will be disjoint connected open subsets of the planar domain

domain R R , each one with a piece , each one with a piece - - wise smooth boundary wise smooth boundary

! ! Γ Γ will be the union of the boundaries. will be the union of the boundaries.

C C ^Γ

=

= n

i

R i

R

1

Mumford

Mumford - - Shah functional Shah functional

! ! Let Let f f differentiable on differentiable on ∪ ∪ R R _{i i} and allowed to and allowed to be discontinuous across

be discontinuous across Γ Γ . .

! ! The smaller The smaller E E , the better , the better (f, (f, Γ Γ ) ₎ segments segments g g 1. 1. f f approximates approximates g g

2. 2. f f (hence (hence g g ) does not vary much on ) does not vary much on R R _{i i} s s

3. 3. The boundary The boundary Γ Γ be as short as possible. be as short as possible.

! ! Dropping any term would cause Dropping any term would cause inf inf E=0 E=0 . .

Γ +

∇ +

−

=

Γ µ ∫∫ f g dxdy ∫∫ − Γ f dxdy ν

f

E R R

2 2

2 ( )

) ,

(

Cartoon image Cartoon image

! ! (f, (f, Γ Γ ) ₎ is simply a cartoon of the original image is simply a cartoon of the original image g g . .

! ! Basically Basically f f is a new image with edges drawn sharply. is a new image with edges drawn sharply.

! ! The objects are drawn The objects are drawn smootly smootly without texture without texture

! ! (f, (f, Γ Γ ) ) is essentially an idealization of is essentially an idealization of g g by the sort of by the sort of image created by an artist.

image created by an artist.

! ! Such cartoons are perceived correctly as representing Such cartoons are perceived correctly as representing the same

the same scane scane as g as g " " f f is a simplification of the scene is a simplification of the scene containing most of its essential features.

containing most of its essential features.

Cartoon image example

Cartoon image example

Related problems Related problems

! ! D. D. Geman Geman & S. & S. Geman Geman : Stochastic relaxation, : Stochastic relaxation,

Gibbs distribution and the Bayesian restoration of Gibbs distribution and the Bayesian restoration of

images.

images. IEEE Trans. on PAMI 6, pp 721 IEEE Trans. on PAMI 6, pp 721 - - 741, 741, 1984.

1984.

! ! MRF model MRF model

! ! A. Blake & A. A. Blake & A. Zisserman Zisserman : Visual Reconstruction. : Visual Reconstruction.

MIT Press, 198 MIT Press, 198 7 7

! ! Weak membrane model Weak membrane model

! ! M. M. Kass Kass , A. , A. Witkin Witkin & D. & D. Terzopoulos Terzopoulos : Snakes: : Snakes:

Active contour Models.

Active contour Models. International Journal of International Journal of Computer Vision, vol. 1, pp 321

Computer Vision, vol. 1, pp 321 - - 332, 1988 332, 1988 . .

! ! Active contour model Active contour model

approximation approximation

! ! A special case of A special case of E E where where f= f= a a _{i i} is constant is constant on each open set

on each open set R R _{i i} . .

! ! Obviously, it is minimized in Obviously, it is minimized in a a _{i i} by setting by setting a a _{i i} to the mean of

to the mean of g g in in R R _{i i} : :

Γ +

−

=

Γ ∑ ∫∫

−

2 2

2 ( , ) ( )

µ µ ν

R

i

i

dxdy a

g f

E

) ) (

(

i R

R

i area R

gdxdy g

mean

a

i

= ∫∫

=

approximation approximation

! ! It can be proven that minimizing It can be proven that minimizing E E _{0 0} is well is well posed:

posed:

! ! For any continuous For any continuous g g , there exists a , there exists a Γ Γ made up made up of of finit finit number of singular points joined by a number of singular points joined by a finit finit

number of arcs on which

number of arcs on which E E

atteins atteins a minimum. a minimum.

! ! It can also be shown that It can also be shown that E E _{0 0} is the natural is the natural limit functional of

limit functional of E E as as µ µ # # 0 ₀

Γ +

−

=

Γ ∑ ∫∫

0

( ) ( ( ))

µ ν

i i

R R

i

dxdy g

mean g

E

Relation to the

Relation to the Ising Ising model model

! ! If we further restrict If we further restrict f f to to

! ! take only values of +/ take only values of +/ - - 1, 1,

! ! assume that assume that g g and and f f are defined on a lattice are defined on a lattice

! ! then then E E

becomes the energy of the becomes the energy of the Ising Ising model

model . .

Relation to the

Relation to the Ising Ising model model

! ! Γ Γ is the path between all pairs of lattice points on is the path between all pairs of lattice points on which

which f f changes sign: changes sign:

∑

∑ ^{− + −}

=

) , ( ), , (

2 ,

2

*

0

( ) ( ( , ) ( , ))

( ( , ) ( , ))

l k j i j

i

l k f

j i f j

i g j

i f f

E

Figure

FigurefromfromD. MumfordD. Mumford& J. & J.

ShahShah: : Optimal Optimal Approximations by Approximations by Piecewise Smooth Piecewise Smooth Functions and Associated Functions and Associated Variational

VariationalProblems. Problems. Communications on Pure and Communications on Pure and Applied Mathematics, Vol.

Applied Mathematics, Vol.

XLII, pp 577

XLII, pp 577--685, 1989685, 1989

Weak Weak string string

! ! Fitting Fitting an an elastic elastic spline spline with with possible possible breaks breaks ( ( line line process

process or or local local edges edges ) )

! ! Remove Remove noise noise

! ! Approximate Approximate with with smooth smooth curves curves

! ! Breaks Breaks where where smoothness smoothness is is not not satisfied satisfied

Image

Image fromfromCMU 15CMU 15--385 Computer 385 Computer VisionVisioncourse, course, SpringSpring2002 2002 bybyTaiTaiSing Sing LeeLee

Energy

Energy of of a a weak weak string string

! ! α α is is the the cost cost of of inserting inserting a a break break ( ( local local edge edge element

element ) ) l l _{i i}

! ! l l _{i i} may may take take binary binary values values [0,1] [0,1]

! ! l l _{i i} is is turned turned on on when when ( ( f f _{i+1 i+1} - - f f _{i i} ) ) ^{2 2} > > α α / / λ λ

∑

∑

∑ ^{− + − − +}

=

i i i

i i

i i

i

i

d f f l l

f f

E ( ) ( )

λ (

)

( 1 ) α

Weak Weak membrane membrane model model

Image

Image fromfromCMU 15-CMU 15-385 Computer 385 Computer VisionVisioncourse, course, SpringSpring2002 2002 bybyTaiTaiSing LeeSing Lee

∑

∑

∑

∑

∑

+ +

+

−

− +

+

−

− +

−

=

+

+

j i

c j

i

j i j

i j

i j

i

j i j

i

j i j

i

j i j

i j

i

j i j

i j

i j i j

i

j i V h

v v

f f

h f

f d

f v

h f

E

, ,

, ,

, ,

2 ,

1 ,

, ,

2 ,

, 1 ,

2 ,

, ,

, ,

) , ( )

( )

1 ( ) (

) 1

( ) (

) (

) ,

, (

κ α

λ

constraint constraint

! ! V V

(i,j (i,j ) ) energy energy term term : :

! ! Low Low for for 0, 2 0, 2 lines lines

! ! Medium Medium for for 3 lines 3 lines

! ! High High for for 1, 4 1, 4 lines lines

Image

Image fromfromCMU 15-CMU 15-385 Computer 385 Computer VisionVisioncourse, course, SpringSpring2002 2002 bybyTaiTaiSing LeeSing Lee

the the Euler Euler equation equation

! ! The The extrema extrema of of a a function function f(x f(x ) ) are are attained attained where where f’ = 0

f’ = 0

! ! Similarly Similarly , , the the extrema extrema of of the the functional functional E(u E(u ) ) are are obtained

obtained where where E’ = 0 E’ = 0 . .

! ! E = ( E = ( ∂ ∂ E / E / ∂ ∂ u) u) is the is the first first variation variation . .

! ! Assuming Assuming a common a common formulation formulation where where u(x):[0,1] u(x):[0,1] # # R R , , u(0)=a

u(0)=a and and u(1)=b u(1)=b , ,

the the basic basic problem problem is is to to minimize minimize : :

! ! The The necessary necessary condition condition for for u u to to be be an an extremum extremum of of E(u E(u ) ) is is the the

Euler

Euler equation equation

of of a a one one dimensional dimensional problem problem . .

∫

=

0

( , ' ) )

( u F u u dx

E

'  = 0



 





∂

− ∂

∂

∂

u F dx

d u

F

Gradient

Gradient descent descent

! ! The The Euler Euler equation equation can can be solved be solved by by numerically numerically solving solving (1). (1).

! ! Can Can be be formulated formulated as as an an evolution evolution equation

equation (t ( t – – time time ): ):

!! ForFor exampleexample, , updatingupdating ff whilewhile ll is is fixed (

fixed (stringstring):):

Image

Image fromfromCMU 15CMU 15--385 Computer 385 Computer VisionVisioncourse, course, SpringSpring2002 2002 bybyTaiTaiSing Sing LeeLee

) 3 (

) 2 ( ) 1

)(

( 2 ) (

2

) 1 (

1

1

i t

i t

i i

i i

i i

i i

i i

f f E

f f

l f

f d

f f E

f E dt

df

∂

− ∂

=

−

=

∆

−

−

−

−

∂ =

∂ ∂

− ∂

=

+

+

µ λ

Works

Works only only for for convex convex functions

functions ! !

Simulated

Simulated Annealing Annealing example example

! ! Works Works for for non non - - convex convex energy

energy functions functions

! ! λ λ =6 =6 , , α α =0.04 =0.04

Images

ImagesfromfromCMU 15-CMU 15-385 Computer 385 Computer VisionVisioncourse, course, SpringSpring 2002

2002 bybyTaiTaiSing LeeSing Lee

Line process (l) Surface signal (f)

Graduated

Graduated non non - - convexity convexity

! ! Proposed Proposed by by Blake Blake & & Zisserman Zisserman for for the the weak weak membrane’s membrane’s energy energy

! ! Basic idea: Basic idea:

1. 1. Approximate Approximate the the original original energy energy functional functional by by a a convex convex one one

2. 2. Do Do a a gradient gradient descent descent on on the the approximation approximation 3. 3. Gradually Gradually morph morph back back the the approximation approximation into into

the the original original energy energy while while repeating repeating step step 2. 2.

! ! In In case case of of a a weak weak membrane membrane energy energy , , the the morphing

morphing can can be be parametrized parametrized ! !

Graduated

Graduated non non - - convexity convexity

! ! GNC GNC runs runs downhill downhill on on each each of of a a sequence sequence of of

functions functions

! ! It It reaches reaches a a global global optimum

optimum assuming assuming a a seuence

seuence of of

approximating

approximating and and locally

locally convex convex funtions funtions exist

exist . .

Image

Image fromfromCMU 15-CMU 15-385 Computer 385 Computer VisionVisioncoursecourse, , SpringSpring2002 by2002 byTaiTaiSing LeeSing Lee

the the weak weak membrane membrane energy energy

! ! F F ^{(0) (0)} =E =E the the original original functional

functional , ,

! ! F F ^{(1) (1)} =F* =F* the the convex convex approximation

approximation

∑

∑ ^{− + −}

=

i

i i

p i

i i

p

f d g f f

F

( )

(

)

 

 



≥

<

≤

−

−

<

=

r t

if

r t

q if

r t

c

q t

if t

t g

|

|

|

| 2

/ )

| (|

|

| )

(

2 2 )

(

α α λ

q r r c

membrane string

c p

c c

2

2

2 1

4 / 1

2 /

* 1 ,

/

*

λ α λ ^ _ ⁼

 

  +

=

 

= 

=

Contours

Contours are are defined

defined as as

the the set set of of i i for for which

which

|f |f

- - f f

|>0 |>0

the the weak weak membrane membrane energy energy

! ! It It is is shown shown that that F F

is is convex

convex for for p p ≥ ≥ 1 1

! ! F F

can can be be minimized minimized using

using gradient gradient descent descent

! ! As As p p # # 0 ₀

! ! Increased Increased localization localization of of boundaries boundaries ( ( l l ) )

! ! Gradual Gradual anisotropic anisotropic smoothing

smoothing of of surface surface ( ( f f ) )

! ! Parameters Parameters used used for for the the test: test: λ λ =6 =6 , , α α =0.03 =0.03

Image

Image fromfromCMU 15CMU 15--385 Computer 385 Computer VisionVisioncourse, course, SpringSpring2002 by2002 byTaiTaiSing Sing LeeLee

ff ll

equivalence equivalence

! ! Formal Formal equivalence equivalence between between the the two two approaches

approaches . . For For example example : :

! ! Taking Taking exponential exponential (~ (~ Hammersley Hammersley - - Clifford Clifford ) )

! ! T T ~ ~ uncertainty uncertainty („ („ temperature temperature ”) ”)

∑

∑

∑ ^{− + − − +}

=

i

i i

i i

i i

i

i

d f f l l

f f

E ( ) ( )

λ (

)

( 1 ) α

4 4 4

4 3 4

4 4

4 2 1

4 43 4

42

1

/ ) )

1 ( ) (

(

(Gaussian) term

data

/ ) / (

)

(

∏

∏

−

=

i

T l

l f

f i

T d

T f f

E _{i i i i i i}

e e

e ^{λ α}

equivalence equivalence

! ! Size Size of of the the neighborhood neighborhood in in the the MRF ( MRF ( or or Gibbs

Gibbs field field ) ) corresponds corresponds to to the the degree degree of of derivatives

derivatives in in the the energy energy functional functional

! ! Membrane Membrane : : (f (f

- - f f

) )

! ! Thin Thin plate plate : : (f (f

- - 2f 2f

+f +f

) )