Image Matching and

(1)

Computer Vision I - Image Matching and

Image Formation

Carsten Rother

10/12/2014

(2)

Roadmap for next five lecture

• Appearance based matching (sec. 4.1)

• How do we get an RGB image? (sec 2.2-2.3)

• The Human eye

• The Camera and Image formation model

• Projective Geometry - Basics (sec. 2.1.1-2.1.4)

• Geometry of a single camera (sec 2.1.5, 2.1.6)

• Pinhole camera

• Lens effects

• Geometry of two cameras (sec. 7.2)

• Robust Geometry estimation for two cameras (sec. 6.1.4)

• Multi-View 3D reconstruction (sec. 7.3-7.4)

10/12/2014 2 Computer Vision I: Image Formation Process

(3)

Matching two Images

v

• Find interest points

• Find orientated patches around interest points to capture appearance

• Encode patch appearance in a descriptor

• Find matching patches according to appearance (similar descriptors)

• Verify matching patches according to geometry (later lecture)

(4)

Examples: Appearance-based matching

10/12/2014

Computer Vision I: Image Formation Process 4

(5)

Applications

• 3D reconstruction:

• Augmented Realty:

• Panoramic Stitching:

(6)

Roadmap: matching two Images (appearance & geometry)

10/12/2014

v

• Find interest points

• Find orientated patches around interest points to capture appearance

• Encode patch appearance in a descriptor

• Find matching patches according to appearance (similar descriptors)

• Verify matching patches according to geometry (later lecture!)

(7)

Reminder Lecture 3: Harris Corner Detector

Idea was a so-called auto-correlation function:

Compute:

1.

2.

(8)

Roadmap: matching two Images (appearance & geometry)

10/12/2014

v

• Find interest points

• Find orientated patches around interest points to capture appearance

• Encode patch appearance in a descriptor

• Find matching patches according to appearance (similar descriptors)

• Verify matching patches according to geometry (later lecture)

(9)

How to deal with orientation

Orientate with image gradient:

(10)

Choose a patch around each point

10/12/2014

How to deal with scale?

(11)

Choose a patch around each point

(12)

Choose a patch around each point

10/12/2014

How to deal with scale?

(13)

Reminder: Edge detection via image gradient

Final result with canny edge detector

Result of

(using small sigma) ( (𝐷

_𝑥

∗ 𝐺) ∗ 𝐼 , (𝐷

_𝑦

∗ 𝐺) ∗ 𝐼)

Image Image gradient:

Result of

(using large sigma)

(14)

Alterative Edge Detector via LoG Operator

10/12/2014 14

Find zero-crossing

• To find an edge we first smooth

• is called the LoG (Laplacian of Gaussian operator)

• 1D example: • 2D example

(Mexican hat):

Computer Vision I: Image Formation Process

(15)

Alterative: Edge detection with LoG Filter

Called Laplacian of Gaussian (LoG)

(16)

Scale selection (illustration)

10/12/2014

𝛻

²

𝐺(𝜎) ∗ 𝐼 = 𝜕

²

(𝐺(𝜎) ∗ 𝐼)

𝜕𝑥

²

+ 𝜕

²

(𝐺(𝜎) ∗ 𝐼)

𝜕𝑦

²

𝒇 is Laplacian of Gaussian (LoG) operator.

Measures an average edge-ness in all directions

(details on page 191)

(17)

Scale selection (illustration)

(18)

Scale selection (illustration)

10/12/2014

(19)

Scale selection (illustration)

(20)

Scale selection (illustration)

10/12/2014

We could match up these curves and find unique corresponding points

(21)

Scale selection (illustration)

Simpler: Find maxima / minima in the curves, respectively

(22)

Roadmap: matching 2 Images (appearance & geometry)

10/12/2014

v

• Find interest points

• Find orientated patches around interest points to capture appearance

• Encode patch appearance in a descriptor

• Find matching patches according to appearance (similar descriptors)

• Verify matching patches according to geometry

(23)

v

SIFT feature

• 4*4=16 cells

• Each cell has an 8 bin histogram (smoothed across cells)

• In total: 16*8 values, i.e. 128D vector

64 pixels

64 pi xels

A cell has 16x16 pixels

(here 8x8 for illustration only)

(blue circle shows center weighting)

(24)

SIFT feature is very popular

10/12/2014

• Fast to compute

• Can handle large changes in viewpoint well (up to 60

^𝑜

out of plane rotation)

• Can handle photometric changes (even day versus night)

(25)

Many other feature descriptors

• MOPS [Brown, Szeliski and Winder 2005]

• SURF [Herbert Bay et al. 2006]

• DAISY

[Tola, Lepetit, Fua 2010]

• Shape Context

• ….

DAISY

(26)

Roadmap: matching 2 Images (appearance & geometry)

10/12/2014

v

• Find interest points

• Find orientated patches around interest points to capture appearance

• Encode patch appearance in a descriptor

• Find matching patches according to appearance (similar descriptors)

• Verify matching patches according to geometry (later lecture)

(27)

Find matching patches fast

N patches (e.g. N = 1000) Goal:

1) Find for each patch in left image the closest in right image

2) Accept all those matches where descriptors are similar enough N patches

(e.g. N = 1000)

Methods:

• Naïve: 𝑁

²

tests (here 1 Million)

• Hashing (not discussed)

• Kd-tree; on average NlogN tests (here ~10,000)

Hashing

(Hashing

Function) Index for patch DB

(28)

Subtask: Find one patch

10/12/2014

?

Database image

Query patch

(29)

Nearest Neighbor Search

• Tracking in Video

The video is the Database

• Whole image has one descriptor

• Database is an image collection

• Image retrieval

(30)

Kd-tree (d stands for dimension)

10/12/2014

• Build the tree over database image:

1) Cycle over dimensions: x,y,z,x,y,z,….

2) Put in axis-aligned hyper-planes (split at median of point set)

• Result: balanced tree

• Nearest Neighbour search (to come)

Example in 3D

[Invented by Jon Louis Bentley 1975]

(31)

4 Examples

1 2

3

5

6

(32)

Examples

10/12/2014

1 2

3

5 4,5,6 1,2,3

Dimension 1 0

4 5 10

8 Di men sion 2

d1>5

Kd-tree

4 6

(33)

Examples

1 2

3

5 1 2,3

d1>5

d2>4.8 d2>6.5

0 4

8 Di men sion 2

Dimension 1

5 10

5 4,6

Kd-tree

4 6

(34)

Examples

10/12/2014

1 2

3

5

1

d1>5

d2>4.8 d2>6.5

0 4

8 Di men sion 2

Dimension 1

5 10

5

d1>1

2 3

Kd-tree

4 6

d1>5.5

4 6

(35)

Examples: nearest neighbor search

1 2

3 5 4

1

d1>5

d2>4.8 d2>6.5

0 4

8 Di men sion 2

Dimension 1

5 10

5

d1>1

2 3

query

Kd-tree

6

d1>5.5

4 6

(36)

Examples: nearest neighbor search

10/12/2014

1 2

3

5

1

d1>5

d2>4.8 d2>6.5

0 4

8 Di men sion 2

Dimension 1

5 10

5

d1>1

2 3

visited

current best

search radius

query

Kd-tree

4 6

d1>5.5

4 6

“find leave where query is in”

(37)

Examples: nearest neighbor search

1 2

3

5

1

d1>5

d2>4.8 d2>6.5

0 4

8 Di men sion 2

Dimension 1

5 10

5

d1>1

2 3

visited

“Radius not intersected”

query

Kd-tree

4 6

d1>5.5

4 6

search radius

(38)

Examples: nearest neighbor search

10/12/2014

1 2

3

5

1

d1>5

d2>4.8 d2>6.5

0 4

8 Di men sion 2

Dimension 1

5 10

5

d1>1

2 3

visited

current best

“Radius intersects.

Go down this

subtree and find in all touching pockets (here one) the

closest point. Take it as new candidate if closer than

current one”

query

Kd-tree

4 6

d1>5.5

4 6

search radius

(39)

Examples: nearest neighbor search

1 2

3

5

1

d1>5

d2>4.8 d2>6.5

0 4

8 Di men sion 2

Dimension 1

5 10

5

d1>1

2 3

visited

search

radius “better

solution found”

query

Kd-tree

4 6

d1>5.5

4 6

(40)

Examples: nearest neighbor search

10/12/2014

1 2

3

5

1

d1>5

d2>4.8 d2>6.5

0 4

8 Di men sion 2

Dimension 1

5 10

5

d1>1

2 3

visited

current best

query

“no need to visit these subtrees”

Kd-tree

4 6

d1>5.5

4 6

search radius

(41)

Examples: nearest neighbor search

1 2

3

5

1

d1>5

d2>4.8 d2>6.5

0 4

8 Di men sion 2

Dimension 1

5 10

5

d1>1

2 3

visited

query

Done since root-node is marked in both ways

Kd-tree

4 6

d1>5.5

4 6

search radius

(42)

Nearest neighbour search – pseudo code

10/12/2014

Input: query point

Step 1: Find leave node (bucket) with query point Step 2: Make hyper-sphere with radius

(current best and query point)

Step3: go up the tree and see if hyper-plane intersects hyper-sphere Step 3a: no intersection: mark tree branch as visited

since no better point can be found there.

Step 3b: intersection: If not yet marked as visited, then go down

the other branch to find potentially a better point (in all

“touching” pockets). If so, mark as current best and go to Step 2, otherwise mark as visited.

Stop criteria: root node has both branches marked as visited.

Current best

query

Hyper-plane positions Nothing

better possible

Pseudo code

On average O(log N)

(43)

Example with many points in 2D

(44)

Example with many points in 2D

10/12/2014

(45)

Example with many points in 2D

(46)

Example with many points in 2D

10/12/2014

(47)

Example with many points in 2D

(48)

Example with many points in 2D

10/12/2014

(49)

Example with many points in 2D

(50)

Example with many points in 2D

10/12/2014

(51)

Example with many points in 2D

(52)

Example with many points in 2D

10/12/2014

(53)

Example with many points in 2D

(54)

Example with many points in 2D

10/12/2014

(55)

Example with many points in 2D

(56)

Example with many points in 2D

10/12/2014

(57)

Example with many points in 2D

(58)

Example with many points in 2D

10/12/2014

(59)

Example with many points in 2D

(60)

Example with many points in 2D

10/12/2014

(61)

Example with many points in 2D

(62)

Example with many points in 2D

10/12/2014

(63)

Example with many points in 2D

(64)

Example with many points in 2D

10/12/2014

(65)

Example with many points in 2D

(66)

Example with many points in 2D

10/12/2014

(67)

Example with many points in 2D

(68)

Example with many points in 2D

10/12/2014

(69)

Example with many points in 2D

(70)

Example with many points in 2D

10/12/2014

(71)

Example with many points in 2D

(72)

Example with many points in 2D

10/12/2014

(73)

Example with many points in 2D

(74)

Example with many points in 2D

10/12/2014

(75)

Example with many points in 2D

(76)

Example with many points in 2D

10/12/2014

done

(77)

v

Roadmap: matching 2 Images (appearance & geometry)

• Find interest points (including different scales)

• Find orientated patches around interest points to capture appearance

• Encode patch in a descriptor

• Find matching patches according to appearance (similar descriptors)

• Verify matching patches according to geometry (later lecture)

(78)

Halfway

3 Minutes Break Question?

10/12/2014

(79)

Roadmap for next five lecture

• Appearance based matching (sec. 4.1)

• How do we get an RGB image? (sec 2.2-2.3)

• The Human eye

• The Camera and Image formation model

• Projective Geometry - Basics (sec. 2.1.1-2.1.4)

• Geometry of a single camera (sec 2.1.5, 2.1.6)

• Pinhole camera

• Lens effects

• Geometry of two cameras (sec. 7.2)

• Robust Geometry estimation for two cameras (sec. 6.1.4)

(80)

The Human eye

10/12/2014

• The retina contains different types of sensors:

cones “Zapfen” (colors, 6 million) and rods “Stäbchen” (gray levels, 120 million)

• The resolution is much higher in fovea centralis

• Light first passes through the layer of neurons before it reaches photo sensors (smoothing). Only in the

“fovea centralis” the light hits directly the photo sensors

• Signal goes out the other way:

Retina → Ganglion cells (1 million) → Optic nerve →

1 MPixel Camera ?

(81)

Spatial resolution

.

(82)

Spatial resolution

Image Processing: Human seeing 5

.

2MP Camera, far from the screen

(83)

Spatial resolution

.

5MP Camera, close to the screen

(84)

Spatial resolution (secrets)

10/12/2014

• The resolution is much higher In fovea centralis

• The Information is pre-processed by Ganglion cells

(Compare: 3072×2304=7MPixel, 2.4 MB RGB JPEG lossless)

• No still image, but a „Video“ (super-resolution)

• Scanning technique − Saccades

(85)

Eye Saccades

Eyes never move uniformly, but jump in saccades

(approximately 15-100 ms duration between fixation points)

Saccades are driven by the “importance” of the scene parts

(86)

The Human eye

10/12/2014

What is light?

Spectrum, i.e. a function of the wavelength

Spectral resolution of the eye is relatively bad due to projection

Different colors can be computed by adding /subtracting signal of rods

cones “Zapfen”

(87)

Roadmap for next five lecture

• Appearance based matching (sec. 4.1)

• How do we get an RGB image? (sec 2.2-2.3)

• The Human eye

• The Camera and Image formation model

• Projective Geometry - Basics (sec. 2.1.1-2.1.4)

• Geometry of a single camera (sec 2.1.5, 2.1.6)

• Pinhole camera

• Lens effects

• Geometry of two cameras (sec. 7.2)

• Robust Geometry estimation for two cameras (sec. 6.1.4)

(88)

Image formation Process

10/12/2014

(89)

Model of the Surface

BRDF (Bi-directional Reflectance Function)

light

Rendering equation:

Outgoing energy:

𝒕: time x∶ position

𝒙

Incoming energy (shining surface)

BRDF function

(4DoF only 𝜔_𝑖 , 𝜔_𝑜 considered)

Incoming light

Fore-shorting angle (always ≥ 0)

𝜔

_𝑖

𝜔

_𝑜

(90)

Example: diffuse reflectance

10/12/2014

DIFFUSE

𝑓

_𝑟

same for all 𝜔

_𝑜

Single Light source:

Shading effects come from: max(0, 𝑤

_𝑖

𝒏) Rendering equation

constant 0

constant (one 𝜔_𝑖)

0 constant

(also known as the “Lambertian world assumption”)

(91)

Examples: BRDFs

DIFFUSE SPECULAR DIFFUSE + ROUGH SPECULAR

0 Rendering equation (single light source)

^constant

(one 𝜔_𝑖)

𝑓

_𝑟

𝑓

_𝑟

𝑓

_𝑟

(for one 𝜔_𝑖 and all 𝜔_𝑜) (for one 𝜔_𝑖and all 𝜔_𝑜) (for one 𝜔_𝑖and all 𝜔_𝑜)

(92)

Inside a graphics engine

• Rendering equation

• Ray tracing

• Radiosity

10/12/2014

(93)

Example

Texture only 3D scene in blender

(94)

Capturig BRDF / BTF capture

10/12/2014

[Christopher Schwartz, Ralf Sarlette, Michael Weinmann and Reinhard Klein]

BTF – Bidirectional Texture function. At every spatial location 𝒙 a different BRDF

(95)

Camera types

• RGB cameras

• Depth cameras:

• Passive RGB stereo

• Active Structured Light stereo

• Active Time of Flight

• Lightfield cameras

(96)

Digital RGB cameras

10/12/2014

• A digital camera replaces film with a sensor array

• Each cell in the sensor array is light-sensitive diode that converts photons to electrons

• Two common types:

• Charge Coupled Device (CCD)

• Complementary metal oxide semiconductor (CMOS)

(97)

Color - Filters

(98)

Problem with de-mosaicing: color moiré

10/12/2014

(99)

Cause of color moiré

(100)

There is a lot of image processing in camera

10/12/2014

Image taken

zoom

(101)

In-camera processing

(102)

Depth camera: Time of Flight

10/12/2014

Intensity image Depth Image

PMD Camera

Sends out infra-red light and looks at changes in phase and magnitude

(103)

Depth Camera – passive RGB stereo

Easy to match hard to match

(104)

Depth Camera – Structured Light

10/12/2014