Algorithms and Applications:

Computer Vision I -

Algorithms and Applications:

Robust Two-View Geometry

Carsten Rother

01/12/2013

Null space (correction from last lecture)

• Right null space: 𝐴 𝒙 = 𝟎 where 𝐴 = 𝑚 × 𝑛 with 𝑚 < 𝑛 this is given by the (n-m) last column vectors in 𝑉

• Example:

A =

1 2

−1 2

−1 2

−1 2

1 2

1 2

Then SVD is:

1 2

−1

√2

−1 2

−1 2

1

√2 1 2

0 0 0

=

1

√2

−1

√2 0

−1

√2 1

√2 0 0 0 1

1 0 0 0 1 0 0 0 0

1

√2 0 ⁻¹

√2

0 1 0

−1

√2 0 ⁻¹

√2

1

√2 0 −1

√2 0 1 0 0 0 0

𝒙 = 𝜆

−1

√2 0

−1

√2

= 𝜆 1 0 1

Nullspace

and 𝐴𝑥 = 0

Reminder: The tasks from last lecture

• Two-view transformations we look at today:

• Homography 𝐻: between two views

• Camera matrix 𝑃 (mapping from 3D to 2D)

• Fundamental matrix 𝐹 between two un-calibrated views

• Essential matrix 𝐸 between two calibrated views

• Derive geometrically: 𝐻, 𝑃, 𝐹, 𝐸, i.e. what do they mean?

• Calibration: Take primitives (points, lines, planes, cones,…) to compute 𝐻, 𝑃, 𝐹, 𝐸 :

• What is the minimal number of points to compute them (very important for next lecture on robust methods)

• If we have many points with noise: what is the best way to computer them: algebraic error versus geometric error ?

• Can we derive the intrinsic (𝐾) an extrinsic (𝑅, 𝑇) parameters from 𝐻, 𝑃, 𝐹, 𝐸?

Epipolar Geometry (Repeat)

• Epipolar line: Constrains the location where a particular point (here 𝑝₁) from one view can be found in the other.

Epipolar Geometry

• Epipolar lines:

• Intersect at the epipoles

• In general not parallel

The maths behind it: Fundamental/Essential Matrix

derivation on black board

𝑋

(𝑅, 𝑇)

~

~

Camera 0 Camera 1

The maths behind it: Fundamental/Essential Matrix

The 3 vectors are in same plane (co-planar):

1) 𝑇 = 𝐶₁ − 𝐶₀ 2) 𝑋 − 𝐶₀

3) 𝑋 − 𝐶₁

Set camera matrix: 𝑥₀ = 𝐾₀ 𝐼 0 𝑋 and 𝑥₁ = 𝐾₁𝑅⁻¹ 𝐼 −𝐶₁ 𝑋 The three rays are:

1) T

2) 𝐾₀⁻¹𝑥₀

3) 𝑅𝐾₁⁻¹𝑥₁ + 𝐶₁ − 𝐶₁ = 𝑅𝐾₁⁻¹𝑥₁

We know that:

~ ~ ~

~ ~

~ ~

~

~ ~

~ ~

~

𝑋

(𝑅, 𝑇)~

~

Camera 0 Camera 1

The maths behind it: Fundamental/Essential Matrix

• In an un-calibrated setting (𝐾’𝑠 not known):

𝑥₀^𝑇𝐾₀^−𝑇 𝑇 _× 𝑅(𝐾₁⁻¹𝑥₁) = 0

• In short: 𝑥₀^𝑇𝐹𝑥₁ = 0 where F is called the Fundamental Matrix

(discovered by Faugeras and Luong 1992, Hartley 1992)

• In an calibrated setting (𝐾’s are known):

we use rays: 𝑥_𝑖 = 𝐾_𝑖⁻¹𝑥_𝑖

then we get: 𝑥₀^𝑇 𝑇 _× 𝑅𝑥₁ = 0

In short: 𝑥₀^𝑇𝐸𝑥₁ = 0 where E is called the Essential Matrix

(discovered by Longuet-Higgins 1981)

~

~

Fundamental Matrix: Properties

• We have 𝑥₀^𝑇𝐹𝑥₁ = 0 where F is called the Fundamental Matrix

• It is det 𝐹 = 0. Hence F has 7 DoF Proof: 𝐹 = 𝐾₀^−𝑇 𝑇 _× 𝑅𝐾₁⁻¹

𝐹 has Rank 2 since 𝑇 _× has Rank 2

~

Check: det( 𝑥 _×) = 𝑥₃ 𝑥₃0 − 𝑥₁𝑥₂ + 𝑥₂ 𝑥₁𝑥₃ + 𝑥₂0 = 0

𝒙 _× =

0 −𝑥₃ 𝑥₂ 𝑥₃ 0 −𝑥₁

−𝑥₂ 𝑥₁ 0

~

Fundamental Matrix: Properties

• For any two matching points (i.e. have the same 3D point) we have: 𝑥₀^𝑇𝐹𝑥₁ = 0

• Compute epipolar line in camera 1 of a point 𝑥₀: 𝑙₁^𝑇 = 𝑥₀^𝑇𝐹 (since 𝑥₀^𝑇𝐹𝑥₁ = 𝑙₁^𝑇𝑥₁ = 0)

• Compute epipolar line in camera 1 of a point 𝑥₀:

𝑙₀ = 𝐹𝑥₁ (since 𝑥₀^𝑇𝐹𝑥₁ = 𝑥₀^𝑇𝑙₀ = 0) 𝑋

(𝑅, 𝑇)~

~

Camera 0 Camera 1

𝑋~

Fundamental Matrix: Properties

Camera 0

• For any two matching points (i.e. have the same 3D point) we have: 𝑥₀^𝑇𝐹𝑥₁ = 0

• Epipole 𝑒₀ is a projection of 𝐶₁ into camera 0.

It is: 𝑒₀^𝑇 𝐹𝑥_𝑖 = 0 for all points 𝑥_𝑖 (since all lines 𝑙₀ = 𝐹𝑥₁ go through 𝑒₀^𝑇)

This means: 𝑒₀^𝑇𝐹 = 0

So 𝑒₀ is left nullspace of 𝐹 (can be computed with SVD)

• Epipole 𝑒₁ is right null space of 𝐹 (𝐹𝑒₁ = 0)

How can we compute 𝐹 (2-view calibration) ?

• Each pair of matching points gives one linear constraint 𝑥^𝑇𝐹𝑥^′ = 0 in 𝐹. For (𝑥, 𝑥^′) we get:

• Given 𝑚 ≥ 8 matching points (𝑥^′, 𝑥) we can compute the F in a simple way.

𝑓₁₁ 𝑓₁₂ 𝑓₁₃ 𝑓₂₁ 𝑓₂₂ 𝑓₂₃ 𝑓₃₁ 𝑓₃₂ 𝑓₃₃

= 0

𝑥₁𝑥₁^′ 𝑥₁𝑥₂^′ 𝑥₁𝑥₃^′ 𝑥₂𝑥₁^′ 𝑥₂𝑥₂^′ 𝑥₂𝑥₃^′ 𝑥₃𝑥₁^′ 𝑥₃𝑥₂^′ 𝑥₃𝑥₃^′ .

. .

How can we compute 𝐹 (2-view calibration) ?

Method (normalized 8-point algorithm):

1) Take 𝑚 ≥ 8 points

2) Compute T, and condition points: 𝑥 = 𝑇𝑥; 𝑥’ = 𝑇’𝑥’

3) Assemble 𝐴 with 𝐴𝑓 = 0, here A is of size 𝑚 × 9, and 𝑓 vectorized 𝐹 4) Compute 𝑓^∗ = 𝑎𝑟𝑔𝑚𝑖𝑛_𝑓 𝐴𝑓 subject to 𝑓 = 1

use SVD to do this.

5) Get 𝐹 of unconditioned points: 𝑇^𝑇𝐹𝑇′ (note: (𝑇𝑥)^𝑇𝐹 𝑇′𝑥^′ = 0)

4) Make 𝑟𝑎𝑛𝑘 𝐹 = 2

Can we compute 𝐹 with just 7 points?

Method (7-point algorithm):

1) Take 𝑚 = 7 points.

2) Assemble 𝐴 with 𝐴𝑓 = 0, here A is of size 7 × 9, and 𝑓 vectorized 𝐹 3) Compute 2D right null space: 𝛼𝐹₁ + 1 − 𝛼 𝐹₂ (use SVD to do this) 4) Derive cubic polynomial equation: det(𝛼𝐹₁ + 1 − 𝛼 𝐹₂) = 0.

This gives one or three real solutions.

Note an 8^th point would say which of these 3 solutions is the correct one.

We will see in later that 7-point algorithm is the best choice for robust case

Can we get 𝐾’𝑠, 𝑅, 𝑇 from 𝐹 ?

• Assume we have 𝐹 = 𝑥₀^𝑇𝐾₀^−𝑇 𝑇 _× 𝑅𝐾₁⁻¹ can we get out 𝐾₁, 𝑅, 𝐾₀, 𝑇 ?

• 𝐹 has 7 DoF

• 𝐾₁, 𝑅, 𝐾₀, 𝑇 have together 16 DoF

• Not directly possible. Only with assumptions such as:

• External constraints

• Camera does not change over several frames

• This is an important topic (10 years of research) called

auto-calibration or self-calibration. We look at it in detail in next lecture.

~

~

~

Coming back to Essential Matrix

• In a calibrated setting (𝐾’s are known):

we use rays: 𝑥_𝑖 = 𝐾_𝑖⁻¹𝑥_𝑖

then we get: 𝑥₀^𝑇 𝑇 _× 𝑅𝑥₁ = 0

In short: 𝑥₀^𝑇𝐸𝑥₁ = 0 where E is called the Essential Matrix

• 𝐸 has 5 DoF, since 𝑇 has 3DoF, 𝑅 3DoF but unknown overall scale

• 𝐸 has also Rank 2

~

~

𝑋

(𝑅, 𝑇)~

~

How to compute 𝐸

• We have: 𝑥₀^𝑇𝐸𝑥₁ = 0 (reminder: 𝑥₀^𝑇𝐹𝑥₁ = 0)

• Given 𝑚 ≥ 8 matching run 8-point algorithm (as for 𝐹)

• Given 𝑚 = 7 run 7-point algorithm and get 1 or 3 solutions

• Given 𝑚 = 5 run 5-point algorithm to get up to 10 solutions.

This is the minimal case since 𝐸 has 5 DoF.

• 5-point algorithm history:

• Kruppa, “Zur Ermittlung eines Objektes aus zwei Perspektiven mit innere

Orientierung,” Sitz.-Ber. Akad. Wiss., Wien, Math.-Naturw. Kl., Abt. IIa, (122):1939- 1948, 1913.

found 11 solutions

• M. Demazure, “Sur deux problemes de reconstruction,” Technical Report Rep. 882, INRIA, Les Chesnay, France, 1988

showed that only 10 valid solutions exist

• D. Nister, “An Efficient Solution to the Five-Point Relative Pose Problem,” IEEE Conference on Computer Vision and Pattern Recognition, Volume 2, pp. 195-202, 2003

fast method which gives out 10 solutions of a 10 degree polynomial

Can we get 𝑅, 𝑇 from 𝐸 ?

• Assume we have 𝐸 = 𝑇 _× 𝑅, can we get out 𝑅, 𝑇 ?

• E has 5 DoF

• 𝑅, 𝑇 have together 6 DoF

• We can get 𝑇 up to scale, and a unique 𝑅

~

~

~

~

~

𝑋

(𝑅, 𝑇)~

~

Can we get 𝑅, 𝑇 from 𝐸 ?

• How to get T :

~

𝐸 = 𝑇 _×𝑅 = 𝑈𝐷𝑉^𝑇 = 𝒖₀ 𝒖₁ 𝑇

1 0 0 0 1 0 0 0 0

𝒗₀^𝑇 𝒗₁^𝑇 𝒗₂^𝑇

~ ~

~

~

• How to get R:

It Is (for some 𝒔_𝑖): 𝑇 _× = 𝑆𝑍𝑅_+/−90^𝑜𝑆^𝑇 = 𝒔₀ 𝒔₁ 𝑇

1 0 0 0 1 0 0 0 0

0 −1 0

1 0 0

0 0 1

𝒔₀^𝑇 𝒔₁^𝑇 𝑇^𝑇 We have: 𝐸 = 𝑇 _×𝑅 = 𝑆𝑍𝑅_+/−90^𝑜𝑆^𝑇𝑅 = 𝑈𝐷𝑉^𝑇

𝑈 𝐷 𝑉^𝑇

~

~

~

~

So we get: 𝑅_+/−90^𝑜𝑆^𝑇𝑅 = 𝑉^𝑇

That gives: 𝑅 = 𝑈𝑅_+/−90^{𝑇 𝑜}𝑉^𝑇 (note 𝑆 = 𝑈)

𝑇

This is𝑅₊₉₀^𝑜 This fixes the norm of 𝑇 to 1

~

𝑆𝑉𝐷 𝑓𝑟𝑜𝑚 𝐸

First Ambiguity: 𝐸 = 𝑇 _×(−1)(−1)𝑅

𝑇 = 𝒔~ ₀ × 𝒔₁

Visualization of the 4 solutions for 𝑅

The property that points must lie in front of the camera is known as chirality (Hartley 1998) correct one since points in

front of cameras!

What can we do with 𝐹, 𝐸 ?

• 𝐹/𝐸 encode the geometry of 2 cameras

• Can be used to find matching points (dense or sparse) between two views (next lecture)

• 𝐹/𝐸 encodes the essential information to do 3D reconstruction

Fundamental and Essential Matrix: Summary

• Derive geometrically 𝐹, 𝐸 ∶

• F for un-calibrated cameras

• E for calibrated cameras

• Calibration: Take measurements (points) to compute 𝐹, 𝐸

• 𝐹 minimum of 7 points -> 1 or 3 real solutions.

• 𝐹 many points -> least square solution with SVD

• 𝐸 minimum of 5 points -> 10 solutions

• 𝐸 many points -> least square solution with SVD

• Can we derive the intrinsic (𝐾) an extrinsic (𝑅, 𝑇) parameters from 𝐹, 𝐸?

-> 𝐹 next lectures time

-> 𝐸 yes can be done (translation up to scale)

• What can we do with 𝐹, 𝐸 ?

-> essential tool for 3D reconstruction

In last lecture we asked (for rotating camera)…

Question 1: If a match is completly wrong then is a bad idea

Question 2: If a match is slighly wrong then might not be perfect.

Better might be a geometric error:

𝑎𝑟𝑔𝑚𝑖𝑛_ℎ 𝐴𝒉 𝑎𝑟𝑔𝑚𝑖𝑛_ℎ 𝐴𝒉

𝑎𝑟𝑔𝑚𝑖𝑛_ℎ 𝑯𝑥 − 𝑥′

Robust model fitting

RANSAC:

Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and

Automated Cartography

Martin A. Fischler and Robert C. Bolles (June 1981).

[Side credits: Dimitri Schlesinger]

Example Tasks

Search for a straight line in a clutter of points

i.e. search for parameters and for the model

𝑥 𝑦

Example Tasks

Estimate the fundamental matrix

i.e. parameters satisfying

given a training set of correspondent pairs

Two sources of errors

1. Noise: the coordinates deviate from the true ones according to some “rule” (probability) – the father away the less confident

2. Outliers: the data have nothing in common with the model to be estimated

Ignoring outliers can lead to a wrong

estimation.

→ The way out – find outliers explicitly, estimate the model from inliers only

Task formulation

Let be the input space and be the parameter space The training data consist of data points

Let an evaluation function be given that checks the consistency of a point with a model .

• straight line

• fundamental matrix

The task is to find the parameter that is consistent with the majority of the data points: 𝑦^∗ = 𝑎𝑟𝑔𝑚𝑖𝑛_{𝑦 𝑖}𝑓(𝑥^𝑖, 𝑦)

𝑓 𝑥, 𝑦, 𝑎, 𝑏 = 0 𝑖𝑓 𝑎𝑥 + 𝑏𝑦 − 1 ≤ 𝑡 (𝑒. 𝑔. 0.1)

1 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒

𝑓 𝑥_𝑙, 𝑥_𝑟, 𝐹 = 0 𝑖𝑓 𝑥_𝑙𝐹𝑥_𝑟 ≤ 𝑡 (𝑒. 𝑔. 0.1)

1 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒

Inlier Outlier

First Idea

A naïve approach: enumerate all parameter values

→ Hough Transform (very time consuming, not possible at all for parameters of high dimension).

𝑦^∗ = 𝑎𝑟𝑔𝑚𝑖𝑛_𝑦

𝑖

𝑓(𝑥^𝑖, 𝑦)

How to compute:

Reminder: Hough Transform (Lecture 3)

Goal: find point with many

“votes” in accumulator space

Hough transform

Image with just 3 points

All lines that go through the 3 points Image with points

Observation

• the parameter space have very low counts

• Idea: do not try all values but only some of them.

• Which ones?

Accumulator space number of inliers (sketched)

2 1 2 3 2 1 1 4

2 1 3 3 2 1 2 4

2 3 1 4 2 4 1 2

3 3 3 4 4 1 3 3

3 3 4 3 5 3 4 2

9 ¹⁰ 9 3 1 3 4 4

20 25 4 2 3 1 2

50 35 5 10 5 2 4 4

Θ 𝑟

200

Hough transform

Data-driven Oracle

An oracle is a function that predicts a parameter given the minimum amount of data points (𝑑-tuple):

Examples: a straight line can be estimated from 2 points, the

fundamental matrix from 8 (or even 7) points (correspondences) etc.

Do not enumerate all parameter values but all d-tuples of data points

That is then 𝑛^𝑑 number of tests, e.g. 𝑛² for lines

The optimization is performed over a discrete domain.

𝑦^∗ = 𝑎𝑟𝑔𝑚𝑖𝑛_𝑦

𝑖

𝑓(𝑥^𝑖, 𝑦)

RANSAC

Basic RANSAC Idea: Do not try all subsets, but sample them randomly:

How many times do you have to sample in order to reliable estimate the true model?

Can be done in parallel!

Repeat many times

select d-tuple, e.g. (𝑥¹, 𝑥²) for lines

compute parameter 𝑦, e.g. line y = 𝑔 𝑥¹, 𝑥² evaluate 𝑓′ 𝑦 = _𝑖 𝑓(𝑥^𝑖, 𝑦)

If 𝑓′ 𝑦 ≤ 𝑓′ 𝑦^∗

set 𝑦^∗ = 𝑦 and keep value 𝑓′ 𝑦^∗

Example Video

Side note: Can we compute 𝐹 with just 7 points?

Method (7-point algorithm):

1) Take 𝑚 = 7 points.

2) Assemble 𝐴 with 𝐴𝑓 = 0, here A is of size 7 × 9, and 𝑓 vectorized 𝐹 3) Compute 2D right null space: 𝛼𝐹₁ + 1 − 𝛼 𝐹₂ (use SVD to do this) 4) Derive cubic polynomial equation: det(𝛼𝐹₁ + 1 − 𝛼 𝐹₂) = 0.

This gives one or three real solutions.

Note an 8^th point would say which of these 3 solutions is the correct one.

We will see in next lecture that 7-point algorithm is the best choice for robust case Answer: try all 3 solutions

Convergence

Observation: it is necessary to sample any -tuple of inliers just once in order to estimate the model correctly.

Let 𝜀 be the probability of outliers.

The probability to sample 𝑑 inliers is 1 − 𝜀 ^𝑑 (here 0.8² = 0.64) The probability of a “wrong” 𝑑-tuple is (here 0.36) The probability to sample 𝑛 times only wrong tuples is

(here 0.36²⁰ = 0.0000000013)

The probability to sample the “right” tuple at least once during the process (i.e. to estimate the correct model according to assumptions)

𝜀 ~ 0.2

200 1000

(here 99.999999866 %)

Convergence

𝑛

𝑝𝑟𝑜𝑏𝑎𝑏𝑖𝑡𝑦

𝜀 (outliers)

The choice of the Oracle is crucial

Example – the fundamental matrix:

a) 8-point algorithm

Probability: 70% (𝑛 = 300; 𝜖 = 0.5; 𝑑 = 8) b) 7-point algorithm

Probability: 90% (𝑛 = 300; 𝜖 = 0.5; 𝑑 = 7) Number of trials to get p% accuracy (here 99%)

𝑛 = log 1 − 𝑝

log(1 − 1 − 𝜀 ^𝑑) 𝑝 = 1 − 1 − 1 − 𝜀 ^{𝑑 𝑛}

𝑑 𝜀

Extension: Adaptive number of samples 𝑛

Choose 𝑛 in an adaptive way:

1) Set 𝑛 = ∞ and 𝜀 = 0.9 (large value) 2) During RANSAC adapt 𝑛:

1) Re-compute 𝜀 from current best solution 𝜀 = outliers / all points

2) Compute new 𝑛:

𝑛 = ^{log 1−𝑝}

log(1− 1−𝜀 ^𝑑)

Confidence Interval

The choice of a right confidence interval is crucial.

Large confidence,

“right” model, 2 outliers

Large confidence,

“wrong” model, 2 outliers

Small confidence,

Almost all points are outliers (independent of the model)

For a line: 𝑓 𝑥, 𝑦, 𝑎, 𝑏 = 1 𝑖𝑓 𝑎𝑥 + 𝑏𝑦 − 1 ≤ 𝑡 (𝑒. 𝑔. 0.1)

0 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒

better: 𝑓 𝑥, 𝑦, 𝑎, 𝑏 = 1 𝑖𝑓 𝑑 (𝑥, 𝑦), 𝑙(𝑎, 𝑏) ≤ 𝑡 (𝑒. 𝑔. 0.1)

0 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒

(Euclidean distance:

point to line)

Examples:

Heuristic guess on Confidence interval

Assume Gaussian noise for a point with 𝜎 as standard deviation

You can derive that 𝑑 𝑥, 𝑙 ≤ 𝜎 3.84 = 𝑡 to have a 95% chance that inlier is inside interval (see page 119 HZ)

𝑙

MSAC (M-Estimator SAmple Consensus)

If a data point is an inlier the penalty is not 0, but it depends on the

“distance” to the model.

Example for the fundamental matrix:

becomes

→ the task is to find the model of the minimum average penalty

[P.H.S. Torr und A. Zisserman 1996]

“robust function”

𝑡)

𝑓 𝑥_𝑙, 𝑥_𝑟, 𝐹 = 0 𝑖𝑓 𝑥_𝑙𝐹𝑥_𝑟 ≤ 𝑡 (𝑒. 𝑔. 0.1)

1 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒

𝑓 𝑥_𝑙, 𝑥_𝑟, 𝐹 = 𝑥_𝑙𝐹𝑥_𝑟 𝑖𝑓 𝑥_𝑙𝐹𝑥_𝑟 ≤ 𝑡 (𝑒. 𝑔. 0.1) 𝑡 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒

Extension … again randomization

Evaluation of a hypothesis , i.e. is often consuming Randomized RANSAC:

instead to check all data points 1. Sample 𝑚 points from

2. If all of them are good, check all others as before

3. If there is at least one bad point among 𝑚, reject the hypothesis It is possible that good hypotheses are rejected.

However it saves time (bad hypotheses are recognized fast)

→ one can sample more often

In last lecture we asked…

Question 1: If a match is completly wrong then is a bad idea Answer: RANSAC with 𝑑 = 4

Question 2: If a match is slighly wrong then might not be perfect.

Better might be a geometric error:

Answer: next slides

𝑎𝑟𝑔𝑚𝑖𝑛_ℎ 𝐴𝒉

𝑎𝑟𝑔𝑚𝑖𝑛_ℎ 𝐴𝒉

𝑎𝑟𝑔𝑚𝑖𝑛_ℎ 𝑯𝑥 − 𝑥′

Reminder last Lecture: Homography for rotating camera

image

Algorithm:

1) Take 𝑚 ≥ 4 point matches (𝑥, 𝑥’) 2) Assemble 𝐴 with 𝐴𝒉 = 0

3) compute 𝒉^∗ = 𝑎𝑟𝑔𝑚𝑖𝑛_𝒉 𝐴𝒉 subject to 𝒉 = 1, use SVD to do this.

Refine Hypothesis 𝐻 with inliers

(The following is also applicable to 𝐹, 𝐸, etc. computation) Algebraic error: 𝑎𝑟𝑔𝑚𝑖𝑛_ℎ 𝐴𝒉

First geometric error: 𝐻^∗ = 𝑎𝑟𝑔𝑚𝑖𝑛_{𝐻 𝑖} 𝑑(𝑥_𝑖^′, 𝐻𝑥_𝑖)

That is not very symmetric

𝑥_𝑖 𝐻𝑥_𝑖 𝑥_𝑖^′

Where 𝑑(𝑎, 𝑏)is (sqaured) 2D distance: 𝑎 − 𝑏 ²

Refine Hypothesis 𝐻 with inliers

(The following is also applicable to 𝐹, 𝐸, etc. computation) Algebraic error: 𝑎𝑟𝑔𝑚𝑖𝑛_ℎ 𝐴𝒉

First geometric error: 𝐻^∗ = 𝑎𝑟𝑔𝑚𝑖𝑛_{𝐻 𝑖} 𝑑(𝑥_𝑖^′, 𝐻𝑥_𝑖)

Where 𝑑(𝑎, 𝑏)is (sqaured) 2D distance: 𝑎 − 𝑏 ²

Symmetric geometric error: 𝐻^∗ = 𝑎𝑟𝑔𝑚𝑖𝑛_{𝐻 𝑖} 𝑑 𝑥_𝑖^′, 𝐻𝑥_𝑖 + 𝑑 𝑥_𝑖, 𝐻⁻¹𝑥′_𝑖

𝑥_𝑖 𝐻⁻¹𝑥′_𝑖 𝐻𝑥_𝑖 𝑥_𝑖^′

Refine Hypothesis 𝐻 with inliers

(The following is also applicable to 𝐹, 𝐸, etc. computation) Algebraic error: 𝑎𝑟𝑔𝑚𝑖𝑛_ℎ 𝐴𝒉

First geometric error: 𝐻^∗ = 𝑎𝑟𝑔𝑚𝑖𝑛_{𝐻 𝑖} 𝑑(𝑥_𝑖^′, 𝐻𝑥_𝑖)

Where 𝑑(𝑎, 𝑏)is (sqaured) 2D distance: 𝑎 − 𝑏 ²

Symmetric geometric error: 𝐻^∗ = 𝑎𝑟𝑔𝑚𝑖𝑛_{𝐻 𝑖} 𝑑 𝑥_𝑖^′, 𝐻𝑥_𝑖 + 𝑑 𝑥_𝑖, 𝐻⁻¹𝑥′_𝑖 Ideal (Gold standard error):

{𝐻^∗, 𝑥_𝑖^′_𝑠} = 𝑎𝑟𝑔𝑚𝑖𝑛_{𝐻,𝑥

𝑖′𝑠} 𝑖𝑑 𝑥_𝑖, 𝑥^{^}_𝑖 ² + 𝑑 𝑥_𝑖^′, 𝑥^{^′}_𝑖 ² 𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑥^{^}_𝑖^′ = 𝐻𝑥^{^}_𝑖

𝑥_𝑖 𝑥_𝑖^′

The true 3D point 𝑋 is searched for

𝑥_𝑖 ^

^ 𝑥_𝑖^′

𝐻

Comment: same as MLE with isotropic Gaussian noise (see page 103 HZ)

^

^ _^

Full Method (HZ page 123)

[see details on page 114ff in HZ]

Example

Input images

~500 interest points

Example

268 putative matches

Found 117 outliers

Found 151 inliers

262 inliers After guided matching

A few word on iterative continuous optimization

So far we had linear (least square) optimization problems:

𝑥^∗ = 𝑎𝑟𝑔𝑚𝑖𝑛_𝑥 𝐴𝑥

For non-linear (arbitrary) optimization problems:

𝑥^∗ = 𝑎𝑟𝑔𝑚𝑖𝑛_𝑥 𝑓(𝑥)

• Iterative Estimation methods (see Appendix 6 in HZ; page 597ff)

• Gradient Descent Method

(good to get roughly to solution)

• Newton Methods (e.g. Gauss-Newton):

second order Method (Hessian). Good to find accurate result

• Levenberg – Marquardt Method:

mix of Newton method and Gradient descent

Red Newton’s method;

green gradient descent

Application: Automatic Panoramic Stitching

An unordered set of images:

Application: Automatic Panoramic Stitching

... automatically create a panorama

Application: Automatic Panoramic Stitching

... automatically create a panorama

Application: Automatic Panoramic Stitching

... automatically create a panorama

An Example: 6DoF Pose Estimation of arbitrary objects

Preparation Step (training step):

Collect a database of 3D objects and virtually point them

3 different lighting conditions Virtual RGB paint

An Example: 6DoF Pose Estimation of arbitrary objects

[Current Research project in CVLD]

At runtime (test-time)

1. From Kinect RGBD image predict object coordinates, i.e. virtual paint

(use Random Forest – see later lecture)

2. Run RANSAC to determine initial 6D Pose (𝑑 = 3)

(in 3D, 3 points are enough to define a 6DoF pose uniquely)

3. Refinement:

6DoF 3D Model fit

An Example: 6DoF Pose Estimation of arbitrary objects

“smart” sampling of candidates

“refinement”

is 3D model fit

3 3D points

An Example: 6DoF Pose Estimation of arbitrary objects

[Current Research project in CVLD]

Next Lecture

Final Lecture on Sparse Geometry:

• Multi-view geometry

• Triangulation of points (from calibrated cameras)

• Self-calibration (in case of un-calibrated cameras)

• Bundle adjustment

• Alternative Methods: Factorization, Reference Plane