Active Contours Active Contours

Snakes:

Snakes:

Active Contours Active Contours

Zoltan

Zoltan Kato Kato

http://www.cab.u

http://www.cab.u--szeged.hu/~kato/variational/szeged.hu/~kato/variational/

Introduction Introduction

ƒ ƒ Proposed by Proposed by

ƒ ƒ Michael Michael Kass Kass

ƒ ƒ Andrew Andrew Witkin Witkin

ƒƒ http://www.ri.cmu.edu/people/witkin_andrew.htmlhttp://www.ri.cmu.edu/people/witkin_andrew.html

ƒ ƒ Demetri Demetri Terzopoulos Terzopoulos

ƒƒ http://mrl.nyu.edu/~dthttp://mrl.nyu.edu/~dt//

Snakes: Active Contour Models Snakes: Active Contour Models . .

International Journal of Computer Vision, International Journal of Computer Vision,

Vol. 1, pp 321

Vol. 1, pp 321 - - 331, 1988. 331, 1988.

What is a snake?

What is a snake?

ƒƒ An energy minimizing An energy minimizing splinespline guided by external constraint guided by external constraint forces and pulled by image forces toward features:

forces and pulled by image forces toward features:

ƒƒ Edge detectionEdge detection

ƒƒ Subjective contoursSubjective contours

ƒƒ Motion trackingMotion tracking

ƒƒ Stereo matchingStereo matching

ƒƒ ….….

Images taken from the GVF website:

Images taken from the GVF website: http://iacl.ece.jhu.edu/projects/gvfhttp://iacl.ece.jhu.edu/projects/gvf//

Snake behavior Snake behavior

ƒ ƒ A snake falls into the closest A snake falls into the closest local local energy energy minimum.

minimum.

ƒ ƒ The local minima of the snake energy comprise The local minima of the snake energy comprise the set of alternative solutions

the set of alternative solutions

ƒ ƒ A higher level knowledge is needed to choose A higher level knowledge is needed to choose the the „correct one” „correct one” from these solutions from these solutions

ƒƒ HighHigh--level reasoninglevel reasoning

ƒƒ User interactionUser interaction

ƒ ƒ These high These high - - level methods can level methods can interact interact with the with the contour model by pushing it toward an

contour model by pushing it toward an appropriate local minimum

appropriate local minimum

Snake behavior Snake behavior

ƒƒ They rely on other mechanisms They rely on other mechanisms to place them

to place them nearnear the desired the desired contour.

contour.

ƒƒ The existence of such an The existence of such an initializerinitializer is is application dependent.

application dependent.

ƒƒ Even in the case of manual Even in the case of manual initialization, snakes are quite initialization, snakes are quite

powerful in refining the user’s input.

powerful in refining the user’s input.

ƒƒ Basically, snakes are trying to Basically, snakes are trying to match a deformable model to an match a deformable model to an

image by means of energy image by means of energy

minimization.

minimization. Image taken from the GVF website: Image taken from the GVF website:

http://

http://iacl.ece.jhu.edu/projects/gvfiacl.ece.jhu.edu/projects/gvf//

Snake energy Snake energy

ƒ ƒ Parametric representation: Parametric representation: v(s v(s )=( )=( x(s),y(s x(s),y(s )) ))

ƒƒ EE_intint = internal energy due to bending. Serves to impose = internal energy due to bending. Serves to impose piecewise smoothness constraint.

piecewise smoothness constraint.

ƒƒ EE_imageimage = image forces pushing the snake toward image = image forces pushing the snake toward image features (edges, etc…).

features (edges, etc…).

ƒƒ EE_concon = external constraints are responsible for putting = external constraints are responsible for putting the snake near the desired local minimum. It may come the snake near the desired local minimum. It may come

from:

from:

ƒƒ Higher level interpretationHigher level interpretation

ƒƒ User interaction, etc…User interaction, etc…

∫ ^{+ +}

=

0

int

( v ( s )) E ( v ( s )) E ( v ( s )) ds E

E

Internal energy Internal energy

ƒ ƒ The snake is a The snake is a controlled continuity controlled continuity spline spline

ƒƒ Regularizes the problemRegularizes the problem

ƒƒ The first order derivative The first order derivative vv_ss(s(s)) makes the makes the splinespline act like act like a a membrane („elasticity”).membrane („elasticity”).

ƒƒ The second order derivative The second order derivative vv_ssss(s(s)) makes it act like a makes it act like a thinthin--plate plate („rigidity”).(„rigidity”).

ƒƒ αα(s(s)) andand ββ(s)(s) controls the relative importance of controls the relative importance of membrane and thin

membrane and thin--plate termsplate terms

ƒƒ Setting Setting ββ(s)=0(s)=0 for a point allows the snake to become secondfor a point allows the snake to become second-- order discontinuous and develop a corner.

order discontinuous and develop a corner.

2 / )

| ) (

| ) (

| ) (

| ) (

(

int

s v s s v s

E = α

+ β

Image forces Image forces

ƒ ƒ Attracts the snake to features (data term) Attracts the snake to features (data term)

ƒ ƒ lines lines : the simplest functional is the image : the simplest functional is the image intensity:

intensity: E E

= = I(x,y I(x,y ) )

ƒƒ Depending on the sign of Depending on the sign of ww_lineline, the snake will be , the snake will be attracted to the lightest or darkest

attracted to the lightest or darkest nerbynerby contour contour

ƒ ƒ edges edges : one can simply set : one can simply set E E

= = - - | | ∇ ∇ I(x,y)| I(x,y)|

ƒƒ attracts the snake to large intensity gradients.attracts the snake to large intensity gradients.

ƒ ƒ terminations terminations : discussed later : discussed later

term term

edge edge

line line

image

w E w E w E

E = + +

Snake convergence Snake convergence

ƒ ƒ If part of a snake finds a low If part of a snake finds a low - - energy energy feature

feature Î Î the spline the spline term will pull term will pull neighboring parts toward a possible neighboring parts toward a possible

continuation of the feature found.

continuation of the feature found.

ƒ ƒ In fact, this places a large energy well In fact, this places a large energy well around a good local minimum

around a good local minimum

Video taken from the website:

Video taken from the website:

http://www

http://www--2.cs.cmu.edu/afs/cs/user/aw/www/gallery.html2.cs.cmu.edu/afs/cs/user/aw/www/gallery.html

Scale space Scale space

ƒƒ Minimization by scaleMinimization by scale-- continuation:

continuation:

1.1. Spatial smootingSpatial smooting the edge or the edge or line functional

line functional

ƒƒ EE_edgeedge= = --(G(G_σσ∗∇∗∇²² I)I)²², where , where GG_σσ is is a Gaussian with

a Gaussian with σσ standard standard deviation

deviation

ƒƒ Minima lie on zero crossings Minima lie on zero crossings of of GG_σσ∗∇∗∇²² I I (~edges)(~edges)

2.2. Snake comes to equilibrium on Snake comes to equilibrium on a blurry energy

a blurry energy

3.3. Slowly reduce the blurringSlowly reduce the blurring

Image taken from

Image taken from M. M. KassKass& A. & A. WitkinWitkin& D. Terzopoulos& D. Terzopoulos: Snakes: : Snakes:

Active Contour Models.

Active Contour Models. International Journal of Computer Vision, International Journal of Computer Vision, Vol. 1, pp 321

Vol. 1, pp 321--331, 1988331, 1988..

Zero crossings

Termination functional Termination functional

ƒ ƒ Attracts the snake toward termination of line Attracts the snake toward termination of line segments and corners.

segments and corners.

ƒ ƒ Let Let C(x,y C(x,y )= )= G G

(x,y) (x,y) ∗ ∗ I(x,y)) I(x,y))

(smoothed image) (smoothed image)

ƒ ƒ Let Let θ θ =tan =tan

(C (C

/C /C

) ) the gradient angle the gradient angle

ƒƒ n=(n=(coscos θθ, sin , sin θθ)) unit vector along gradientunit vector along gradient

ƒƒ nn_⊥⊥=(=(--sin sin θθ, , coscos θθ)) perpendicular to gradientperpendicular to gradient

ƒ ƒ E E

is defined using curvature of level lines in is defined using curvature of level lines in C(x,y

C(x,y ) ) : :

(

)

2 2 2

2

2

/ /

y x

y xx y

x xy x

yy

term

C C

C C

C C C

C C

n C

n C

E n

+

+

= −

∂

∂

∂

= ∂

∂

= ∂

⊥

θ

Subjective contour Subjective contour

ƒƒ Combining Combining EE_edgeedge and and EE_termterm, we can create a snake , we can create a snake attracted to edges and terminations

attracted to edges and terminations

ƒƒ The shape of the snake between the edges and lines in the The shape of the snake between the edges and lines in the illusion is completely determined by the

illusion is completely determined by the splinespline smoothness smoothness termterm

ƒƒ The same snake can find traditional edges in natural The same snake can find traditional edges in natural images

images

Image taken from

Image taken from M. M. KassKass& A. & A.

Witkin

Witkin& D. & D. TerzopoulosTerzopoulos: Snakes: : Snakes:

Active Contour Models.

Active Contour Models. International International Journal of Computer Vision, Vol. 1, Journal of Computer Vision, Vol. 1, pp 321

pp 321--331, 1988331, 1988..

hysteresys hysteresys

ƒƒ Snake tracking a Snake tracking a moving subjective moving subjective

contour contour

ƒƒ The snake bends The snake bends until the internal until the internal

spline

spline forces forces

overpower image overpower image

forces forces

ƒƒ Then the snake Then the snake

falls off the line and falls off the line and

returns to a returns to a

smoother shape

smoother shape Image taken from Image taken from M. M. KassKass& A. & A. WitkinWitkin& D. Terzopoulos& D. Terzopoulos: Snakes: Active Contour Models. : Snakes: Active Contour Models. International International Journal of Computer Vision, Vol. 1, pp 321

Journal of Computer Vision, Vol. 1, pp 321--331, 1988331, 1988..

Motion tracking Motion tracking

ƒ ƒ Once a snake finds a Once a snake finds a feature, it „locks on”

feature, it „locks on”

ƒ ƒ If the feature begins If the feature begins to move, the snake to move, the snake will track the same will track the same

local minimum local minimum

ƒƒ Fast motion could Fast motion could cause the snake to cause the snake to

flip into a different flip into a different

minimum minimum

Image taken from

Image taken from M. M. KassKass& A. Witkin& A. Witkin& D. & D. TerzopoulosTerzopoulos: Snakes: Active Contour Models. : Snakes: Active Contour Models. International International Journal of Computer Vision, Vol. 1, pp 321

Journal of Computer Vision, Vol. 1, pp 321--331, 1988331, 1988..

Snake energy minimization Snake energy minimization

ƒ ƒ When When α α (s (s ) ) and and β β (s) (s) are are constant

constant , we get two , we get two independent Euler

independent Euler - - Lagrange equations. Lagrange equations.

ƒ ƒ When When α α (s (s ) ) and and β β (s) (s) are not constant are not constant then then it is simpler to use a discrete formulation:

it is simpler to use a discrete formulation:

0 0

∂ = + ∂

+

∂ = + ∂

+

y y E

y

x x E

x

ssss ext ss

ext ssss

ss

β α

β α

4 2

1 1

2 2

1 int

1

int

2 /

| 2

| 2

/

|

| )

(

)) (

), (

( )

, (

), ( )

(

h v

v v

h v

v i

E

ih y

ih x

y x

v i

E i

E E

i i

i i

i i

i

i i

i n

i

ext snake

+

−

−

=

+

− +

−

=

=

= +

= ∑

β

α

Snake energy minimization Snake energy minimization

ƒ ƒ Let Let f f

(i (i ) = ) = ∂ ∂ E E

/ / ∂ ∂ x x

where derivatives are where derivatives are approximated by finite approximated by finite

differences if they differences if they

cannot be computed cannot be computed

analitically analitically . .

ƒ ƒ The corresponding The corresponding Euler equations

Euler equations

ƒ ƒ In matrix form where In matrix form where A A is a

is a pentadiagonal pentadiagonal banded matrix:

banded matrix:

0 ))

( ),

( (

) 2

(

) 2

( 2

) 2

(

) (

) (

2 1

1

1 1

1 2

1

1 1

1

= +

+

− +

+

−

− − +

+

−

−

−

+ +

+

+

−

−

−

−

+ +

−

i f

i f

v v

v

v v

v

v v

v

v v

v v

y x

i i

i i

i i

i i

i i

i i

i i

i i

i i

β β β

α α

0 )

, (

0 )

, (

= +

= +

y x

f Ay

y x

f Ax

y x

Snake energy minimization Snake energy minimization

ƒ ƒ Taking into account the derivatives requires Taking into account the derivatives requires changing

changing A A at each iteration. Speed up: at each iteration. Speed up:

ƒ ƒ We assume that We assume that f f

and and f f

are constant during a are constant during a time step

time step Î Î explicit Euler method explicit Euler method w.r.t w.r.t . the . the external forces.

external forces.

ƒ ƒ internal forces are specified by internal forces are specified by A A Î Î we can we can evaluate the time derivative at

evaluate the time derivative at t t rather than rather than t t - - 1 1

) (

) ,

(

) (

) ,

(

1 1

1

1 1

1

−

−

−

−

−

−

−

−

= +

−

−

= +

t t

t t

y t

t t

t t

x t

y y

y x

f Ay

x x

y x

f Ax

γ

γ

Snake energy minimization Snake energy minimization

ƒ ƒ At equilibrium, the time derivative vanishes. At equilibrium, the time derivative vanishes.

ƒ ƒ The Euler equations can be solved by The Euler equations can be solved by matrix inversion:

matrix inversion:

ƒ ƒ The inverse can be calculated by The inverse can be calculated by LU LU decomposition in

decomposition in O(n O(n ) ) time. time.

)) ,

( (

) (

)) ,

( (

) (

1 1

1 1

1 1

1 1

−

−

− −

−

−

− −

− +

=

− +

=

t t

y t

t

t t

x t

t

y x

f y

I A

y

y x

f x

I A

x

γ γ

γ

γ