Computer Vision I -

(1)

Computer Vision I -

Algorithms and Applications:

Dense Correspondence Fields

Carsten Rother

(2)

Roadmap this lecture

• Define the Problem

• Discrete Matching – “the standard approach”

• Continuous Matching – “more accurate but more challenging ”

• Depth Map Fusion – ICP (Iterative Closest Point)

02/25/2022 Computer Vision I: Dense Correspondence Fields 2

(3)

Roadmap this lecture

• Define the Problem

• Discrete Matching – “the standard approach”

• Continuous Matching – “more accurate but more challenging ”

• Depth Map Fusion – ICP (Iterative Closest Point)

(4)

Two Images of a Static Scene

Epipolar lines for 1D

correspondence search

(5)

Simplification

Move epipoles to infinity

Rectification -

Correspondences are now in same scanline

epipol

(6)

Correspondence Search now in 1D

(7)

Stereo Camera

disparity

(8)

Depth and Disparity

Depth

(9)

Depth and Disparity

Depth changes

non- uniformly Depth

Left image

��

infinity

(10)

Disparity Range

Disparity Range can be quite different across the image

(11)

Moving Scene

Optical flow: 2D movement Scene Flow: 3D movement

2D output

(x,y) movement

3D output (x,y,z) movement (hard to visualize)

Often scene flow is done with RGBD input

We will look at a method later (patch match) which

can deal with all these cases.

(12)

Optical Flow: Small versus large baseline

Tiny displacement (slow moving object in a video sequence)

Normal/large displacement (fast moving object in a video sequence;

stereo camera)

Very large displacement

Often: Different set of techniques used such as

continuous domain and label

(13)

Roadmap this lecture

• Define the Problem

• Discrete Matching – “the standard approach”

• Continuous Matching – “more accurate but more challenging ”

• Depth Map Fusion – ICP (Iterative Closest Point)

(14)

Depth Map – discrete Formulation

• Each pixel gets a disparity label

• Edges are used later for constraining the labeling of neighboring pixels

�_�=0

�_�=�_��

�_�=1

(15)

Similarity measures (photo consistency measures)

• Pixel-Cost for a pixel

Left: Right:

• Patch-Cost for a pixel

Left: Right:

To use neighborhood is same motivation as for sparse point matching (lecture 5)

Size of patches often to

(16)

Robust similarity measures

• Robust to changes in brightness (offset and scaling)

(known as NCC – Normalized cross correlation)

�

_�

( ^�

�

) ⁼

� ∈ �

∑

_�

( ^�

��

− �

^�

) ( ^�

^�⁻^��

�

− �

^�

)

√ ∑

� ∈ �_�

( ^�

�

− �

^�

)

²

^√ ∑

� ∈ �_�

( ^�

^�⁻^��

�

− �

^�

)

²

Mean Image: where is the number of pixels in left image ^-

-

- -

-

Many different similarity measures.

See details in:

D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms.

International Journal of Computer Vision, 47(1/2/3):7-42, April-June 2002

(17)

Block matching

Find for each pixel the optimal depth independently

(18)

Block Matching

• is very simple

• is very efficient (thanks to the integral image approach!)

• can be easily parallelized

• can serve as a reasonable initialization for more elaborated techniques

(19)

Block matching: left-right consistency check

1. Match left to right and right to left .

2. Red pixels: Going from left to right and back is a different pixel location:

3. Fill red pixels in (e.g. from neighbors)

Block Matching l-r check

Original (left) Inpainted

(20)

Limitations and ambiguities

(21)

Problem: Similarity constraint is too local

Block Matching:

Non-local constraint needed!

Tsukuba Scene

Ground Truth

(22)

Smoothness constraint

Neighbouring pixel have similar disparity

Block Matching Ground Truth

(23)

Uniqueness constraint

For any point in one image, there should be at most one matching point in the other image

Left Image Right Image

disparity = 5 disparity = 6

Physically not

possible to see

both 3D points

(24)

Introduce the concept of occlusion

Left image right image

matched all pixels matched

occluded

matched

Introduce an extra label with cost : matched or not matched

(25)

Ordering Constraints

left

right object

a b c

b c a

left

right object

a b c

c

b a

Holds in “normal”

situations

Doesn’t hold in presence

of thin objects

(26)

Scanline-based approaches

Enforce these constraints for each scanline independently

(27)

Shortest Path Algorithm

Pre-compute all matching costs! So called “cost volume”

(the matching cost: ) disparity

R ig h t Im ag e

Left Image 1

n

1 n

Left occlusion

Right occlusion

(28)

Example

Left image right image

matched matched

At infinity

R ig h t Im ag e

Left Image 1

1 n

n

disparity

(29)

Example

Left image right image

matched all pixels matched

occluded

matched

R ig h t Im ag e

Left Image 1

1 n

n

disparity

(30)

Cost Volume Illustration

Red path is the optimal shortst path

zoom Dark pixels have low

matching score

(31)

Shortest Path Algorithm

Creates streaking artefacts Ground Truth

Block Matching

(32)

Extension to 2D Grid

• Do all of the above constraints but on a 2D grid.

• Inference is much harder and no longer optimal (stay tuned for Computer Vision 2)

[see inference details in Kolmogorov, Rother ECCV 06]

(33)

More complex regularizations – Surface Stereo

Computed disparity maps

Assignment of pixels to surfaces

Assumption: Describe the scene by a few low-degree surfaces (splines, planes)

(34)

More complex regularizations – Object Stereo

02/25/2022 Computer Vision I: Dense Correspondence Fields 34 reference plane +

parallax

Assumption:

Describe scene by a few Objects:

- compact in 3D - Connected in 3D - each object has a

compact color model

Depth d

Objects o

[Bleyer, Rother,Rhemann,Gelautz]

(35)

More complex regularizations – Scene Stereo

Left input

image Object labelling

proposal 1

Object labelling proposal 2

Assumption: Describe scene by a few Objects which are physically plausible:

- objects do not fly in the air

- objects do not intersect each other

(36)

Roadmap this lecture

• Define the Problem

• Discrete Matching – “the standard approach”

• Continuous Matching – “more accurate but more challenging ”

• Depth Map Fusion – ICP (Iterative Closest Point)

(37)

Halfway slide

3 minutes break

(38)

Patch Match Stereo

Left view Right view

[Bleyer, Rhemann, Rother, BMVC ’11]

(39)

Goal: Depth Map

Depth map

(40)

Window-based Matching

Local stereo matching:

rectangular region (patch) check photo-consistency

(41)

Window-based Matching

Local stereo matching:

(42)

Problems - Window-based Matching

 Fails at discontinuities

 Fails at non-fronto-parallel planes

 No continuous depth label

 Slow

(43)

Discontinuities

(44)

Discontinuities

Adaptive support weights [Yoon, CVPR ‘05]

(45)

Problems

 Fails at discontinuities

 Fails at non-fronto-parallel planes

 No continuous depth label

 Slow

(46)

Non-front to parallel planes

(47)

Non-front to parallel planes

(48)

The importance of 3D search space

(49)

Problems

 Fails at discontinuities

 Fails at non-fronto-parallel planes

 No continuous depth label

 Slow

(50)

Let’s try something

(51)

Random Correspondence

Depth map

Note we only visualize 1

out of 3 parameters

(52)

Sampling

Depth map

(takes solution if better, not proper sampling)

0 Max

depth Current

depth

Sample in limited

range of orientation

(53)

Sampling

Depth map

(takes solution if better,

(54)

Lets try a different type of “sampling”

Depth map

(55)

Sample your neighbours ;-)

Depth map

(takes solution from your 4 neighbors if

better, not proper sampling)

(56)

Sample your neighbours ;-)

Depth map

(57)

Sample your neighbours ;-)

Depth map

(58)

Sample your neighbours ;-)

Depth map

(59)

Sample your neighbours ;-)

Depth map

(60)

Patch Match Stereo Algorithm

1. Random initialization

2. Go through pixel in sequential order:

2a. Look at solutions from left/top neighbour 2b. sample around current solution

Left image – Reindeer

(Middlebury) Left and right disparity maps (intermediate step of iteration 1)

(61)

Patch Match Stereo Algorithm

(62)

Why does it work so well?

Left image – Sawtooth (Middlebury)

Image consists of 3 planes -

~80.000 guesses for yellow plane Ground truth disparities

 Randomization is in our favour

 No cost volume needed:

well suited for large images and large depth range

 The local sampling is important to get fine details

and escape local minima

(63)

Patch Match Stereo Algorithm

• If you run it forever it gives the globally optimal

• If you stop after finite time then the result has implicit smoothness (which is even what you want)!

• The method is a mix of message passing and simplified and

extended Gibbs sampling (stay tuned for Computer Vision 2)

(64)

Advertisement: 6D Flow

6D Scene Flow: rotation and translation

6D movement Input: Kinect Image

Thursday 20.12 11.00 Michael Hornacek, room INF-2024

6D output

(hard to visualize)

(65)

PatchMatch (the first publication)

(66)

Patch Match: A Dense 2D nearest Neighbor Search

02/25/2022 Computer Vision I: Dense Correspondence Fields67

Hole Filling (only skteched!)

Hole

(a) Get initial solution with some method User input

(b) Find patches for hole in the outside image

(c) Update hole patches and iterate: (b-c; or a-c) (b) The found 2D flow

We look in the same image

and encode label as 2D offset

(67)

Roadmap this lecture

• Define the Problem

• Discrete Matching – “the standard approach”

• Continuous Matching – “harder but better”

• Depth Map Fusion – ICP (Iterative Closest Point)

(68)

KinectFusion - overview

02/25/2022 Computer Vision I: Multi-View 3D reconstruction 69

(69)

Camera tracking

?

Picture of real red camera

Picture of virtual blue camera

(raycasting)

Goal: Find red camera position by moving blue virtual camera in such a way that the virtual and real picture are the same.

Known position virtual camera

unknown position Real camera

(70)

Iterative Closest Point - ICP

Given e.g. a depth image from two view points:

how to align the two point sets?

02/25/2022 Computer vision : ICP 71

applied e.g. in:

Shape inspection, Motion estimation, Appearance analysis, Texture Mapping, Tracking, …

Chen, Y. and Medioni, G. “Object Modeling by Registration of Multiple Range Images,” Proc. IEEE Conf. on Robotics and Automation, 1991

(71)

Iterative Closest Point - ICP

Basic algorithm:

1. Determine pairs of corresponding points

2. Estimate a transformation that minimizes the distances between the correspondences

3. Apply the transformation to align input and target.

4. Go back to 1.

(72)

Iterative Closest Point - ICP

We have two point sets

We are looking for mapping, e.g. a rotation R and translation t.

optimization problem:

Weightings wi,j encode point correspondences:

wi,j = 1 (corresponding points) wi,j = 0 (otherwise)

Correspondences are from Neighbour Search.

see a closed form solution to

this problem in Szeliski page 277

(73)

Iterative Closest Point - ICP

Starting from the basic idea there are a lot of variants.

Variants on the following stages of ICP have been proposed:

1.Selecting source points (from one or both point sets) 2.Choose an error metric

3.Matching to points in the other mesh 4.Weighting the correspondences

5.Rejecting certain (outlier) point pairs

6.Minimizing the error metric

(74)

Iterative Closest Point - ICP

Selecting source points:

• Use all points

• Uniform subsampling

• Random sampling

• Normal-space sampling

• Ensure that samples have normals distributed as uniformly as possible

Uniform Sampling

Uniform Sampling Normal-Space Sampling Normal-Space Sampling

(75)

Iterative Closest Point - ICP

Choose an error metric

Point-to-Plane Error Metric:

• Using point-to-plane distance instead of point-to-point lets flat

regions slide along each other

(76)

Iterative Closest Point - ICP

Matching :

Matching strategy has greatest effect on convergence and speed

• Closest point (stable, but slow and requires preprocessing)

(77)

Iterative Closest Point - ICP

Matching :

Matching strategy has greatest effect on convergence and speed

• Closest point (stable, but slow and requires preprocessing)

• Normal shooting (better for smooth clouds)

(78)

Iterative Closest Point - ICP

Matching :

Matching strategy has greatest effect on convergence and speed

• Closest point (stable, but slow and requires preprocessing)

• Normal shooting (better for smooth clouds)

• Closest compatible point (stable, slow and requires preprocessing)

• Can improve effectiveness

• Compatibility based on normals, colors, etc.

• At limit, degenerates to feature matching

(79)

Iterative Closest Point - ICP

Matching :

Matching strategy has greatest effect on convergence and speed

• Closest point (stable, but slow and requires preprocessing)

• Normal shooting (better for smooth clouds)

• Closest compatible point (stable, slow and requires preprocessing)

• Can improve effectiveness

• Compatibility based on normals, colors, etc.

• At limit, degenerates to feature matching

• Projection

• finding closest point is most expensive stage

• use a simpler algorithm to find correspondences

• for range images, can simply project points

• one to two orders of magnitude faster

(80)

Computer Vision I -