Algorithms and Applications:

Computer Vision I -

Algorithms and Applications:

Two-View Geometry

Carsten Rother

03/12/2013

This is the lecture with most math

… for any error on slides please contact:

v

Roadmap: matching 2 Images (appearance & geometry)

• Find interest points (including different scales)

• Find orientated patches around interest points to capture appearance

• Encode patch in a descriptor

• Find matching patches according to appearance (similar descriptors)

• Verify matching patches according to geometry

The tasks for this lecture

• Two-view transformations we look at today:

• Homography 𝐻: between two views

• Camera matrix 𝑃 (mapping from 3D to 2D)

• Fundamental matrix 𝐹 between two un-calibrated views

• Essential matrix 𝐸 between two calibrated views

• Derive geometrically: 𝐻, 𝑃, 𝐹, 𝐸 , i.e. what do they mean?

• Calibration: Take primitives (points, lines, planes, cones,…) to compute 𝐻, 𝑃, 𝐹, 𝐸 :

• What is the minimal number of points to compute them (very important for next lecture on robust methods)

• If we have many points with noise: what is the best way to computer them: algebraic error versus geometric error ?

• Can we derive the intrinsic (𝐾) an extrinsic (𝑅, 𝑇) parameters from 𝐻, 𝑃, 𝐹, 𝐸?

• What can we do with 𝐻, 𝑃, 𝐹, 𝐸? (e.g. augmented reality)

Singular Value Decomposition (SVD)

• Reminder: separable filters

Apply SVD to the kernel matrix:

Note:

ℎ _𝑥 ∗ ℎ _𝑦

ℎ _𝑥 ℎ _𝑦 ℎ _𝑥

ℎ _𝑦

Properties of SVD

• In short: 𝐴 = 𝑈𝐷𝑉

• If 𝐴 = 𝑚 × 𝑛 with 𝑚 ≥ 𝑛 then 𝑈 = 𝑚 × 𝑛 , D = n × 𝑛, 𝑉 = 𝑛 × 𝑛

• If 𝐴 = 𝑚 × 𝑛 with 𝑚 < 𝑛 then add zero-valued rows to A to make it 𝑛 × 𝑛

• 𝑈 and 𝑉 form an orthogonal basis: 𝑈

𝑈 = 𝑉

𝑉 = 𝐼

• It is: 𝑈𝑥 = 𝑥 (an example U is a rotation matrix)

• Implementation: 𝐴 = 𝑚 × 𝑛 needs: 4𝑚

𝑛 + 8𝑚𝑛

+ 9𝑛

[Golub-89]

• It is: 𝐴

𝐴 = 𝑉𝐷𝑈

𝑈𝐷𝑉

= 𝑉𝐷

𝑉

then the entries of 𝐷

are eigenvalues of 𝐴

𝐴 and eigenvectors are columns of 𝑉

Singular values; sorted: 𝜎

≥ 𝜎

Null space (correction from last lecture)

• Right null space: 𝐴 𝒙 = 𝟎 where 𝐴 = 𝑚 × 𝑛 with 𝑚 < 𝑛 this is given by the (n-m) last column vectors in 𝑉

• Example:

A =

1 2

−1 2

−1 2

−1 2

1 2

1 2

Then SVD is:

1 2

−1

√2

−1

−1 2 2

1

√2 1

0 0 02

=

1

√2

−1

√2 0

−1

√2 1

√2 0 0 0 1

1 0 0 0 1 0 0 0 0

1

√2 0 ⁻¹_√2 0 1 0

−1

√2 0 ⁻¹_√2

1

√2 0 −1 0 1 √20 0 0 0

𝒙 = 𝜆

−1

√2 0

−1

√2

= 𝜆 1 0 1

Nullspace

and 𝐴𝑥 = 0

Least Square Problem (homogenous linear system)

• Find a solution to 𝐴𝒙 = 0 where 𝐴 = 𝑚 × 𝑛 with 𝑚 ≥ 𝑛

• Solution 𝒙 = 0 is not interesting

• If 𝒙 is solution then any 𝑘𝒙. So we look for 𝒙 = 1

• 𝐴𝒙 = 0 is never exactly zero (due to noise in measurements).

Hence, solve in least square sense:

𝒙 ^∗ = 𝑎𝑟𝑔𝑚𝑖𝑛 _𝒙 𝐴𝒙 subject to 𝒙 = 𝟏

• 𝒙 ^∗ is the last column of 𝑉, where 𝐴 = 𝑈𝐷𝑉 ^𝑇

(blackboard)

Proof

Topic 1: Homography 𝐻

• Derive geometrically 𝐻

• Calibration: Take measurements (points) to compute 𝐻

• How do we do that with a minimal number of points?

• How do we do that with many points?

• Can we derive the intrinsic (𝐾) an extrinsic (𝑅, 𝑇) parameters from 𝐻?

• What can we do with 𝐻 ?

Definition Homography

• Definition: A projectivity (or homography) ℎ is an invertible mapping ℎ from 𝑃

to 𝑃

such that three points 𝑥

, 𝑥

, 𝑥

lie on the same line if an only if ℎ(𝑥

), ℎ(𝑥

), ℎ(𝑥

) do.

• Theorem: A mapping ℎ from 𝑃

to 𝑃

is a homography if and only if there exists a non-singular 3 × 3 matrix H with ℎ(𝑥) = 𝐻𝑥

• In equations: x′ = 𝐻𝑥

• 𝐻 has 8 DoF

𝑥 𝑦 1

Transformation matrix 𝐻 ℎ

ℎ

ℎ

ℎ

ℎ

ℎ

ℎ

ℎ

ℎ

𝑦′𝑥′

1

=

Definition Homography

• Definition: A projectivity (or homography) ℎ is an invertible mapping ℎ from 𝑃

to 𝑃

such that three points 𝑥

, 𝑥

, 𝑥

lie on the same line if an only if ℎ(𝑥

), ℎ(𝑥

), ℎ(𝑥

) do.

• Theorem: A mapping ℎ from 𝑃

to 𝑃

is a homography if and only if there exists a non- singular 3 × 3 matrix H with ℎ 𝑥 = 𝐻𝑥

One direction of proof:

• If 𝑥

, 𝑥

, 𝑥

lie on a line 𝑙 then 𝑙

𝑥

= 0.

• Let H be a non-singular 3 × 3 matrix. Then 𝑙

𝐻

𝐻𝑥

= 0. Hence the points: 𝐻𝑥

lie

on a transformed line 𝑙

= 𝑙

𝐻

Geometric Derivation: Homography

𝑥 𝑦 1

Transformation matrix 𝐻 ℎ

ℎ

ℎ

ℎ

ℎ

ℎ

ℎ

ℎ

ℎ

𝑥′𝑦′

1

=

Mapping via a plane

Notation: x (homogenous 2D), x (inhomogenous 2D), 𝑿 (homogenous 3D), 𝑿(inhomogenous 3D)

Homography from a rotating camera

Put it toghter: 𝒙

= 𝑲

𝑹𝑲

𝒙

Hence 𝑯 = 𝑲

𝑹𝑲

is a homography (general 3x3 matrix) with 8 DoF

~

𝑲 = ^𝑓 0 𝑚𝑓 𝑝 ^{𝑠 𝑝}

0 0 1

𝒙 = 𝑲 𝑹 (𝑰

| − 𝑪) 𝑿

Camera 0: 𝒙

= 𝑲

𝑿 (in 3D ∶ 𝑲

𝒙

= 𝑿) Camera 1: 𝒙

= 𝑲

𝑹 𝑿

~

~

~

~

~

Homography of a projection via a plane

(See geometric derivation on page 49)

Definition: A projectivity (or homography) ℎ is an invertible mapping ℎ from 𝑃

to 𝑃

such that three points 𝑥

, 𝑥

, 𝑥

lie on the same line if an only if ℎ(𝑥

), ℎ(𝑥

), ℎ(𝑥

) do so.

We know (see Theorem): All Homographies can be written as ℎ(𝑥) = 𝐻𝑥

We see visually that lines map to lines

So we can write: 𝑥

= 𝐻𝑥

How to compute/calibrate 𝐻

• We have 𝜆𝑥′ = 𝐻𝑥

• H has 8 DoF

• We get for each pair of matching points (𝑥′, 𝑥) the 3 equations:

1) ℎ

𝑥

+ ℎ

𝑥

+ ℎ

𝑥

= 𝜆𝑥

2) ℎ

𝑥

+ ℎ

𝑥

+ ℎ

𝑥

= 𝜆𝑥

• This gives 2 linear independent equations be taking the ratios.

Here 1) divide by 2) gives:

3) ℎ

𝑥

+ ℎ

𝑥

+ ℎ

𝑥

= 𝜆𝑥

′

𝑥

𝑥

, 𝑥

𝑥

, 𝑥

𝑥

, −𝑥

𝑥

, −𝑥

𝑥

, −𝑥

𝑥

ℎ

, ℎ

, ℎ

, ℎ

, ℎ

, ℎ

= 0

How to compute/calibrate 𝐻

ℎ

ℎ

ℎ

ℎ

ℎ

ℎ

ℎ

ℎ

ℎ

= 0

• Put it together:

• We need a minimum of 4 points to get 𝐴ℎ = 0 with 𝐴 𝑖𝑠 8 × 9 matrix, and ℎ 𝑖𝑠 9 × 1 vector

• Solution for ℎ is the right null space of 𝐴 (This solution will be useful in next lecture)

𝑥

𝑥

𝑥

𝑥

𝑥

𝑥

−𝑥

𝑥

−𝑥

𝑥

−𝑥

𝑥

0 0 0 0 0 0 𝑥

𝑥

𝑥

𝑥

𝑥

𝑥

−𝑥

𝑥

−𝑥

𝑥

−𝑥

𝑥

.

. .

Often we have many, slightly wrong point-matches

image

Algorithm:

1) Take 𝑚 ≥ 4 point matches (𝑥, 𝑥’) 2) Assemble A with Ah = 0

3) compute 𝒉 ^∗ = 𝑎𝑟𝑔𝑚𝑖𝑛 _ℎ 𝐴𝒉 subject to 𝒉 = 𝟏, use SVD to do this.

We know how to do: 𝒙 ^∗ = 𝑎𝑟𝑔𝑚𝑖𝑛 _𝒙 𝐴𝒙 subject to 𝒙 = 𝟏

A more numerically stable solution

• Coefficients of an equation system should be in the same order of magnitude so as not to lose significant digits

• In pixels: 𝑥

𝑥

’ ~ 1𝑒6

• Conditioning: scale and shift points to be in [-1..1] (or +/- √2 )

• A general rule, not only for homography computation

• How to do it:

A more numerically stable solution

Algorithm:

1) Take 𝑚 ≥ 4 point matches (𝑥, 𝑥’)

2) Compute T, and condition points: 𝑥 = 𝑇𝑥; 𝑥’ = 𝑇’𝑥’

3) Assemble A with Ah = 0

4) compute 𝒉

= 𝑎𝑟𝑔𝑚𝑖𝑛

𝐴𝒉 subject to 𝒉 = 𝟏, use SVD to do this.

4) Get H of unconditioned points: 𝑇

𝐻𝑇 ^{(Note: 𝑇}

^𝑥

^{= 𝐻𝑇𝑥)}

Motivation for next lecture

Question 1: If a match is completly wrong then is a bad idea

Question 2: If a match is slighly wrong then might not be perfect.

Better might be a geometric error:

𝑎𝑟𝑔𝑚𝑖𝑛

𝐴𝒉 𝑎𝑟𝑔𝑚𝑖𝑛

𝐴𝒉

𝑎𝑟𝑔𝑚𝑖𝑛

𝑯𝑥 − 𝑥′

Can we get 𝐾’𝑠 and 𝑅 from 𝐻 ?

• Assume we have 𝐻 = 𝐾 ₁ 𝑅𝐾 ₀ ⁻¹ of a rotating camera, can we get out 𝐾 ₁ , 𝑅, 𝐾 ₀ ?

• 𝐻 has 8 DoF

• 𝐾 ₁ , 𝑅, 𝐾 ₀ have together 13 DoF

• Not directly possible, only with assumptions on K.

(No application needs such a decomposition)

• For other transfromations such decompositions are possible

What can we do with 𝐻 ?

• Panoramic stitching with rotating camera (exercise later)

Warp images into a canonical view: 𝑥′ = 𝐻𝑥

What can we do with 𝐻 ?

What can we do with 𝐻 ?

• Plane-based augmented realty

Homography 𝐻 : Summary

• Derive geometrically 𝐻

• Calibration: Take measurements (points) to compute 𝐻

• Minimum of 4 points. Solution: right null space of 𝐴ℎ = 0

• Many points. Use SVD to solve 𝒉

= 𝑎𝑟𝑔𝑚𝑖𝑛

𝐴𝒉

• Can we derive the intrinsic (𝐾) an extrinsic (𝑅, 𝑇) parameters from 𝐻?

-> hard. Not discussed much

• What can we do with 𝐻 ?

-> augmented reality on planes, panoramic stitching

Topic 2: Camera Matrix P

• Derive geometrically 𝑃

• Calibration: Take measurements (points) to compute 𝑃

• How do we do that with a minimal number of points?

• How do we do that with many points?

• Can we derive the intrinsic (𝐾) an extrinsic (𝑅, 𝑇) parameters from 𝑃?

• What can we do with 𝑃?

Geometric Derivation: Camera Matrix (Reminder)

• Camera matrix P has 11 DoF

• Intrinsic parameters

• Principal point coordinates (𝑝 _𝑥 , 𝑝 _𝑦 )

• Focal length 𝑓

• Pixel magnification factors 𝑚

• Skew (non-rectangular pixels) 𝑠

• Extrinsic parameters

• Rotation 𝑹 (3DoF) and translation 𝐂 (3DoF) relative to world coordinate system

𝒙 = 𝑲 𝑹 (𝑰

| − 𝑪) 𝑿 ~

~

𝑲 =

𝑓 𝑠 𝑝

0 𝑚𝑓 𝑝

0 0 1

𝒙 = 𝑷 𝑿

How can we compute/calibrate 𝑃 ?

𝒙 = 𝑲 𝑹 (𝑰

| − 𝑪) 𝑿 𝒙 = 𝑷 𝑿

Important move in all directions: 𝑥, 𝑦, 𝑧

Calibration pattern

How can we compute/calibrate 𝑃 ?

• We have 𝜆𝑥′ = 𝑃𝑋

• 𝑃 has 11 DoF

• We get for each point pair (𝑥′, 𝑋) 3 equations, but only 2 linear independent once, by taking ration (to get rid of 𝜆)

• We need a minimum of 6 Points to get 12 equations

Algorithm (DLT - Direct Linear Transform):

1) Take 𝑚 ≥ 6 points.

2) Condition points 𝑋, 𝑥′ using 𝑇, 𝑇′

3) Assemble A with 𝐴𝑝 = 0 (𝐴 𝑖𝑠 𝑚 × 12 and 𝑝 is vectorized 𝑃) 4) compute 𝑝

= 𝑎𝑟𝑔𝑚𝑖𝑛

𝐴𝑝 subject to 𝑝 = 1

use SVD to do this.

5) Get out unconditioned 𝑃 = 𝑇

𝑃𝑇 (note 𝑇’𝑥’ = 𝑃 𝑇𝑋)

Note: a version with minimal number of points (6) is same as with many points

Half-way slide

3 minutes break. Stand Up. Please ask me Questions.

An alternative: Lagrange Multiplier

Maximize 𝑓 𝑥, 𝑦 subject to 𝑔 𝑥, 𝑦 = 𝑐

Lagrange function: 𝐿 𝑥, 𝑦, 𝜆 = 𝑓 𝑥, 𝑦 + 𝜆 (𝑔 𝑥, 𝑦 − 𝑐) An optimal solution is obtained for:

𝜕𝐿(𝑥,𝑦,𝜆)

𝜕𝑥

= 0 ;

= 0 ;

= 0 (these are 3 constraints and 3 unknowns)

Note:

= 𝑔 𝑥, 𝑦 − 𝑐 = 0 (the original constraint)

An alternative: Lagrange Multiplier

• Why does

= 0 ;

= 0 give an optimal solution for 𝑓

• Lagrange function: 𝐿 𝑥, 𝑦, 𝜆 = 𝑓 𝑥, 𝑦 + 𝜆 (𝑔 𝑥, 𝑦 − 𝑐)

• We have:

𝜕𝐿(𝑥,𝑦,𝜆)

𝜕𝑥

=

+ 𝜆

𝜕𝑥

= 0

𝜕𝐿(𝑥,𝑦,𝜆)

𝜕𝑦

=

+ 𝜆

= 0

• This means that gradients are aligned (

,

𝜕𝑦

) = - 𝜆 (

,

𝜕𝑦

)

Gradient 𝑓 Gradient 𝑔

(blackboard)

Lagrange Multiplier: Example

This gives:

+

-1 = 0; hence: 𝜆 = +/−

Maximize 𝑓 𝑥, 𝑦 = 𝑥 + 𝑦 subject to x

+ y

= 1

Lagrange function: 𝐿 𝑥, 𝑦, 𝜆 = 𝑥 + 𝑦 + 𝜆 (𝑥

+ 𝑦

− 1)

𝜕𝐿(𝑥,𝑦,𝜆)

𝜕𝑥

= 1 + 2𝜆𝑥 = 0 (gives: 𝑥 =

)

𝜕𝐿(𝑥,𝑦,𝜆)

𝜕𝑦

= 1 + 2𝜆𝑦 = 0 (gives: y =

)

𝜕𝐿(𝑥,𝑦,𝜆)

𝜕𝜆

= 𝑥

+ 𝑦

− 1 = 0

Two Solutions:

,

,

,

We require:

How can we compute/calibrate 𝑃 ?

• Goal: compute 𝑝

= 𝑎𝑟𝑔𝑚𝑖𝑛

𝐴𝑝 subject to 𝑝 = 1

• Let us rewrite 𝑝 = 1 as 1 − 𝑝

𝑝 = 0

• We have to solve the following problem:

𝜕

𝜕𝑝

𝑝

𝐴

𝐴𝑝 + 𝜆 1 − 𝑝

𝑝 = 0 (note: 𝐴𝑝 = 𝑝

𝐴

𝐴𝑝)

• This gives after some transformation: 𝐴

𝐴𝑝 = 𝜆𝑝

• The solutions for 𝑝 are the eigenvectors of 𝐴

𝐴 and 𝜆 the corresponding eigenvalues

• The least squared error is then:

𝑎𝑟𝑔𝑚𝑖𝑛

𝐴𝑝 = 𝑎𝑟𝑔𝑚𝑖𝑛

𝑝

𝐴

𝐴𝑝 = 𝑎𝑟𝑔𝑚𝑖𝑛

𝑝

𝜆𝑝

note 𝐴^𝑇𝐴𝑝 = 𝜆𝑝

• Assume 𝑃 is known, can we get out 𝐾, 𝑅, 𝐶 ?

• P has 11 DoF

• 𝐾, 𝑅, 𝐶 have together 5+3+3=11 DoF -> possible

• How to do it:

1) The camera center 𝐶 is the right nullspace of P 𝑃𝐶 = 𝐾 𝑅 (𝐶 − 𝐶) = 0

2) 𝑃 = [𝐾𝑅| − 𝐾𝑅𝐶] ; 𝐴 = 𝐾𝑅

can be done with unique RQ decomposition, where R is upper- triangular matrix and Q a rotation matrix (see HZ page 579)

How can we get K,R,C from P

𝒙 = 𝑲 𝑹 (𝑰

| − 𝑪) 𝑿 ~ 𝒙 = 𝑷 𝑿

~

~

~

~

~

~

~

What can we do with 𝑃 ?

• Many things can be done with an externally and internally calibrated camera

• Robot navigation, augmented reality, photogrammetry …

camera

Manipulator

Camera Matrix 𝑃 : Summary

• Derive geometrically 𝑃

• Calibration: Take measurements (points) to compute 𝑃

• 6 or more points. Use SVD to solve 𝒑

= 𝑎𝑟𝑔𝑚𝑖𝑛

𝐴𝒑

• Can we derive the intrinsic (𝐾) an extrinsic (𝑅, 𝑇) parameters from 𝐻?

-> yes, use SVD and RQ decomposition

• What can we do with 𝑃 ?

-> very many things (robotic, photogrammetry, augmented reality, …)

𝒙 = 𝑲 𝑹 (𝑰

| − 𝑪) 𝑿 ~

𝒙 = 𝑷 𝑿

Topic 3: Fundamental/Essential Matrix 𝐹/𝐸

• Derive geometrically 𝐹/𝐸

• Calibration: Take measurements (points) to compute 𝐹/𝐸

• How do we do that with a minimal number of points?

• How do we do that with many points?

• Can we derive the intrinsic (𝐾) an extrinsic (𝑅, 𝑇) parameters from 𝐹/𝐸?

• What can we do with 𝐹/𝐸?

Reminder from Lecture 4: The 3D case

Illustration of the general 3D case

Appearance based matching:

Geometry based matching:

1) Assume sensible camera model. We will see that: Given 7 matching

3D points defines how other 3D points match.

3D Geometry

Non-moving scene

𝑃’

Rigidly (6D) -moving scene

Both cases are equivalent for the following derivations

Epipolar Geometry

Epipolar Geometry

Epipolar Geometry

Epipolar Geometry

Epipolar Geometry

Epipolar plane: Plane through both camera centers and world point.

Epipolar Geometry

Epipolar Geometry

• Epipolar lines:

• Intersect at the epipoles

• In general not parallel

Example: Converging Cameras

Example: Motion Parallel to Camera

• We will use this idea when it comes to stereo matching (in 2 lectures)

Example: Forward Motion

• Epipoles have same coordinate in both images

• Points move along lines radiating from epipole

The maths behind it: Fundamental/Essential Matrix

The 3 vectors are in same plane (co-planar):

1) 𝑇 = 𝐶

− 𝐶

2) 𝑋 − 𝐶

3) 𝑋 − 𝐶

Set camera matrix: 𝑥

= 𝐾

𝐼 0 𝑋 and 𝑥

= 𝐾

𝑅

𝐼 −𝐶

𝑋 The three rays are:

1) T 2) 𝐾

𝑥

3) 𝑅𝐾

𝑥

+ 𝐶

− 𝐶

= 𝑅𝐾

𝑥

We know that:

𝐾

𝑥

𝑇

𝑅(𝐾

𝑥

) = 0 which gives: 𝑥

𝐾

𝑇

𝑅(𝐾

𝑥

) = 0

~ ~ ~

~ ~

~ ~

~

~

~ ~

~ ~

The maths behind it: Fundamental/Essential Matrix

• In an un-calibrated setting (𝐾’𝑠 not known):

𝑥 ₀ ^𝑇 𝐾 ₀ ^−𝑇 𝑇 _× 𝑅(𝐾 ₁ ⁻¹ 𝑥 ₁ ) = 0

• In short: 𝑥 ₀ ^𝑇 𝐹𝑥 ₁ = 0 where F is called the Fundamental Matrix

(discovered by Faugeras and Luong 1992, Hartley 1992)

• In an calibrated setting ( 𝐾 ’s are known):

we use rays: 𝑥 _𝑖 = 𝐾 _𝑖 ⁻¹ 𝑥 _𝑖

then we get: 𝑥 ₀ ^𝑇 𝑇 _× 𝑅𝑥 ₁ = 0

In short: 𝑥 ₀ ^𝑇 𝐸𝑥 ₁ = 0 where E is called the Essential Matrix

(discovered by Longuet-Higgins 1981)

~

~

Fundamental Matrix: Properties

• We have 𝑥 ₀ ^𝑇 𝐹𝑥 ₁ = 0 where F is called the Fundamental Matrix

• It is det 𝐹 = 0. Hence F has 7 DoF Proof: 𝐹 = 𝐾 ₀ ^−𝑇 𝑇 _× 𝑅𝐾 ₁ ⁻¹

𝐹 has Rank 2 since 𝑇 _× has Rank 2

~

Check: det( 𝑥

) = 𝑥

𝑥

0 − 𝑥

𝑥

+ 𝑥

𝑥

𝑥

+ 𝑥

0 = 0

𝒙

=

0 −𝑥

𝑥

𝑥

0 −𝑥

−𝑥

𝑥

0

~

Fundamental Matrix: Properties

• For any two matching points (i.e. have the same 3D point) we have: 𝑥 ₀ ^𝑇 𝐹𝑥 ₁ = 0

• Compute epipolar line in camera 1 of a point 𝑥 ₀ : 𝑙 ₁ ^𝑇 = 𝑥 ₀ ^𝑇 𝐹 (since 𝑥 ₀ ^𝑇 𝐹𝑥 ₁ = 𝑙 ₁ ^𝑇 𝑥 ₁ = 0)

• Compute epipolar line in camera 1 of a point 𝑥 ₀ :

𝑙 ₀ = 𝐹𝑥 ₁ (since 𝑥 ₀ ^𝑇 𝐹𝑥 ₁ = 𝑥 ₀ ^𝑇 𝑙 ₀ = 0)

𝑋

~

~

Fundamental Matrix: Properties

Camera 0

Camera 1

• For any two matching points (i.e. have the same 3D point) we have: 𝑥 ₀ ^𝑇 𝐹𝑥 ₁ = 0

• Epipole 𝑒 ₀ is a projection of 𝐶 ₁ into camera 0.

It is: 𝑒 ₀ ^𝑇 𝐹𝑥 _𝑖 = 0 for all points 𝑥 _𝑖 (since all lines 𝑙

= 𝐹𝑥

go through 𝑒

)

This means: 𝑒 ₀ ^𝑇 𝐹 = 0

So 𝑒 ₀ is left nullspace of 𝐹 (can be computed with SVD)

• Epipole 𝑒 ₁ is right null space of 𝐹 (𝐹𝑒 ₁ = 0)

How can we compute 𝐹 (2-view calibration) ?

• Each pair of matching points gives one linear constraint 𝑥

𝐹𝑥

= 0 in 𝐹 . For 𝑥, 𝑥′ we get:

• Given 𝑚 ≥ 8 matching points (𝑥

, 𝑥) we can compute the F in a simple way.

𝑓

𝑓

𝑓

𝑓

𝑓

𝑓

𝑓

𝑓

𝑓

𝑥

𝑥

𝑥

𝑥

𝑥

𝑥

𝑥

𝑥

𝑥

𝑥

𝑥

𝑥

𝑥

𝑥

𝑥

𝑥

𝑥

𝑥

= 0

.

. .

How can we compute 𝐹 (2-view calibration) ?

Method (normalized 8-point algorithm):

1) Take 𝑚 ≥ 8 points

2) Compute T, and condition points: 𝑥 = 𝑇𝑥; 𝑥’ = 𝑇’𝑥’

3) Assemble 𝐴 with 𝐴𝑓 = 0, here A is of size 𝑚 × 9, and 𝑓 vectorized 𝐹 4) Compute 𝑓

= 𝑎𝑟𝑔𝑚𝑖𝑛

𝐴𝑓 subject to 𝑓 = 1

use SVD to do this.

5) Get 𝐹 of unconditioned points: 𝑇

𝐹𝑇′ (note: (𝑇𝑥)

𝐹 𝑇′𝑥

= 0)

4) Make 𝑟𝑎𝑛𝑘 𝐹 = 2

How to make 𝐹 Rank 2

• (Again) use SVD:

Set last singular value 𝜎

to 0 then A has Rank 𝑝 − 1 and not p (assume A had full Rank p before)

Proof: diagonal matrix has Rank 𝑝 − 1 hence A has Rank 𝑝 − 1

Can we compute 𝐹 with just 7 points?

Method (7-point algorithm):

1) Take 𝑚 = 7 points.

2) Assemble 𝐴 with 𝐴𝑓 = 0, here A is of size 7 × 9, and 𝑓 vectorized 𝐹 3) Compute 2D right null space: 𝛼𝐹

+ 1 − 𝛼 𝐹

(use SVD to do this) 4) Derive cubic polynomial equation: det(𝛼𝐹

+ 1 − 𝛼 𝐹

) = 0.

This gives one or three real solutions.

Note an 8

point would say which of these 3 solutions is the correct one.

We will see in next lecture that 7-point algorithm is the best choice for robust case