Efficient Nonlocal Regularization for Optical Flow (ECCV 2012)

Efficient Nonlocal Regularization for Optical Flow (ECCV 2012)

P. Kr¨ahenb¨uhl, V. Koltun (Stanford University)

Lilli Kaufhold, December 18th

Optical Flow

Given: images I₁,I₂ ∈R^N

Wanted: displacement vectoru ∈R^2N such that pixelI₁(x) corresponds toI2(x+u)

Different Constancy Assumptions:

Grey Value: I₂(x+u)−I₁(x) = 0 Gradient: ∇I₂(x+u)− ∇I₁(x) = 0 Hessian: H₂I2(x+u)− H₂I1(x) = 0 Laplacian: ∆I₂(x+u)−∆I₁(x) = 0

Optical Flow

Minimise an energy functional of the form E(u) =E_Data(u) +λ_LE_L(u)

data term: E_Data(u) =X

i

kI₂(x_i +u_i)−I₁(x_i)k²

smoothness term: E_L(u) =X

i

X

j∈N_i

Ψ (u_i −u_j)² w_ij

Ψ(·): penaliser function Ψ(x²) = (x²+²)^0.45

wi,j: weighting function determining the influence pixelj has on pixeli

Content

1 Nonlocal Method

2 Minimisation Linearisation

Efficient computation

3 Influence of Parameters

4 Results

Nonlocal Regularisation

Energy Functional:

E(u) =E_Data(u) +λ_LE_L(u) +λ_NE_N(u)

nonlocal regulariser: E_N(u) =X

i

X

j6=i

Ψ (u_i−u_j)² w_ij

Ψ(·):penaliser function Ψ(x²) = (x²+²)^0.45

wi,j: weighting function determining the influence pixelj has on pixeli

Nonlocal Regularisation

Visualisation of nonlocal weights:

w_i,j = exp (−kp_i −pjk²

2σ²_x −kc_i−cjk² 2σ_c² )

Figure: Higher intensity corresponds to higher weight.

Minimisation

E(u) = X

i

kI₂(xi+ui)−I1(xi)k²+λL

X

i

X

j∈N_i

Ψ (ui−uj)² wij

+λN

X

i

X

j6=i

Ψ (ui−uj)² wij

Minimisation Strategy:

Compute ∇E(u) Solve ∇E(u) = 0

Difficulties:

energy functional is not convex nor linear

normal minimisation techniques are too slow for the nonlocal term E_data andE_L easier to handle→ only focus on E_N

Linearisation

Nonlocal regulariser

E_N(u) =X

i

X

j6=i

Ψ (ui−uj)² wij

Computation of∇E_N(u):

∂

∂ui

E_N(u) = 2X

j6=i

w_ijΨ⁰ (u_i−u_j)²

(u_i−u_j)

Linearisation:

∂

∂ui

E_N(u^l+1) = 2X

j6=i

w_ijΨ⁰

(u_i^l−u_j^l)²

(u^l+1_i −u^l_j⁺¹)

Linear System

Resulting Linear Equation: (A+B)u^l+1=wB with Au^l+1 = ∇E_N(u)

Bu^l+1−w_B = ∇E_data(u) +∇E_L(u)

The entries ofAare given by:

A_ij = −w_ijΨ⁰

(u_i^l −u_j^l)²

for i 6=j Aii = X

j6=i

wijΨ⁰

(u_i^l−u_j^l)²

Jacobi-Method:

u_i^l+1,k+1= (Aii+Bii)⁻¹·



wB i −X

j6=i

(Aij +Bij)u_j^l+1,k



 for all i

Computational Problems

Problem: Each iteration is very expensive due to the following terms:

A_ii = X

j6=i

w_ijΨ⁰

(u^l_i −u^l_j)²

= X

j6=i

exp (−^kpi_2σ^−p₂^jk2

x −^kci_2σ^−c₂^jk2

c )Ψ⁰

(u^l_i −u^l_j)²

X

j6=i

Aiju^l+1,k_j = X

j6=i

−w_ijΨ⁰

(u_i^l −u_j^l)²

u_j^l+1,k

= X

j6=i

−exp (−^kpi_2σ^−p2^jk2

x −^kci_2σ^−c2^jk2

c )Ψ⁰

(u_i^l−u_j^l)²

u_j^l+1,k

Idea: Approximate Ψ with Gaussian kernels and perform an efficient Gaussian convolution.

Reduction to Gaussian Convolution

Approximation with exponential mixtures

Ψ(x²)≈µω,σ(x²) :=T −

K

X

n=1

ωnexp(− x² 2σ²_n)

⇒Ψ⁰(x²)≈

K

X

n=1

ωn

2σ_n² exp(− x² 2σ_n²)

Minimise

Z ∞

−∞

µ_ω,σ(x²)−Ψ(x˜ ²)2

dx with truncated penalty function ˜Ψ

in order to optimise the parametersω andσ.

Influence of Parameters: K

Figure: Penalty function Ψ(x²) = (x²+²)^0.45 and its Approximation

Influence of Parameters: σ

, σ

and λ

E_N(u) =λ_NX

i

X

j6=i

Ψ (u_i −u_j)²

exp (−^kpi−pjk2

2σ²_x −^kci−cjk2

2σ²_c )

Figure: top row: endpoint and angular errors for varyingσ_c andλ_N bottom row: endpoint and angular errors for varyingσ_x andλ_N

Results

Figure: top left: first image frametop right: ground truth flowbottom left: result without nonlocal regularisation bottom right: result with nonlocal regularisation

Results

Figure: Screenshot of the current Middlebury benchmark ranking [http://vision.middlebury.edu/flow/eval/results/results-a1.php]

Summary

Use a nonlocal smoothness term in addition to the commonly used local one

Approximate the penalty function with Gaussian kernels and use a fast Gaussian convolution method

Can be computed very efficiently

Nonlocality improves the results of this optical flow method Can maybe even improve the best method there currently is?

References

Philipp Kr¨ahenb¨uhl and Vladlen Koltun: Efficient Nonlocal Regularization for Optical Flow. European Conference on Computer Vision, 2012.

N. Papenberg, A. Bruhn, T. Brox, S. Didas, and J. Weickert:

Highly accurate optic flow computation with theoretically justified warping. International Journal of Computer Vision, 2006.

A. Adams, J. Baek, and M. A. Davis: Fast high-dimensional filtering using the permutohedral lattice. Computer Graphics Forum, 2010.

Approximation of Charbonnier penalizer for K = 3

Average endpoint and angular error

endpoint and angular errors for σ

and λ

endpoint and angular errors for σ

and λ

first image frame

ground truth flow

result without nonlocal regularisation

result with nonlocal regularisation