8.2 Image Rectification
8.2.1 Rectification Based on Homographies
Appendix 95
Figure 8.1: Visualization of the rectification method proposed by [Fusiello et al., 2000]. The original views are depicted in dark gray. The bright gray planeπrepresents the common image plane to which the views are projected to. Note thatR'bandR'mare identical. Rectified images are depicted by dashed lines and are located in the plane π.
96 Image Rectification object points are mapped to identical image rows across the two rectified views. Using equation 8.13 the camera matrices of the virtual views are given by
K'b=K'm=
f' 0 0 0 f' 0
0 0 1
. (8.14)
The remap functionsφare defined as
x'b =φ(xb) =Hbxb =K'bR'bR⊤bK−1b xb (8.15) x'm=φ(xm) =Hmxm=K'mR'mR⊤mK−1mxm. (8.16) In other words the homogeneous vectorx'is derived by first transferring the original vectorxto the world coordinate system and then to the pixel coordinate system defined by the virtual views (R',K',C'). So far the pixel coordinate systems of virtual views posses origins with respect to the principal axis. More common are coordinate systems with the origins located at the upper-left image corners. This can be achieved by updating K'b and K'm utilizing the dimensions of the rectified images. The dimensions can be found by transforming the old corner coordinates using equation 8.16 and 8.15. This leads tox'min,x'max,y'min and y'maxfor each of the views. The dimensions of rectified images are calculated as
colums=ceil(x'max)−f loor(x'min)
rows=ceil(y'max)−f loor(y'min). (8.17) Then interior orientations are completed by updating the camera matrices of rectified views according to
K'b =
f' 0 −x'b,min
0 f' −y'b,min
0 0 1
(8.18)
K'm=
f' 0 −x'm,min
0 f' −y'm,min
0 0 1
. (8.19)
While straight-forward to implement this approach has one significant drawback: For epipoles of two views which are close to zero, that is motion in viewing direction, dimensions of virtual views as well as distortions become huge and the method loses practicality.
Loop’s Method
Another method based on homographies was proposed by [Loop and Zhang, 1999]. Thereby homographies are computed in a way such that perspective distortions in the rectified image pairs are minimized. Let F' be the fundamental matrix of a rectified image pair:
F'=
0 0 0
0 0 −1
0 1 0
. (8.20)
The first property of this fundamental matrix is that the epipoles e' = (1,0,0) of both, base and match images are at infinity which is easily verified by
F'e'b=0=F'⊤e'm. (8.21)
Furthermore, two homogeneous image coordinates which share the same y component x'b = (x1, y,1) and x'm= (x2, y,1) satisfy the epipolar constraint:
x'⊤bF'x'm= 0 (8.22)
Appendix 97 Formulating this equation with respect to the coordinates of the original images transformed by the homo-graphiesHb andHmleads to:
x⊤bH⊤bF'Hmxm=x⊤bFxm= 0. (8.23) Now the task is to design the homographies Hb and Hm in a way such that projective distortions are minimized. Therefore let Hbe parametrized as
H=
u⊤ v⊤ w⊤
=
u1 u2 u3
v1 v2 v3
w1 w2 w3
(8.24)
This can be further composed by splitting the homography into a projective transformation, a similarity transformation and a shearing transformation
H=HsHrHp. (8.25)
More precisely the single components are designed as H=
s1 s2 s3
0 1 0
0 0 1
v2−v3w2 v3w1−v1 0 v1−v3w1 v2−v3w2 v3
0 0 1
1 0 0
0 1 0
w1 w2 1
. (8.26)
The projective transformation transfers the epipoles in the original images to epipoles at infinity in the rectified images. A pointpi= (px,i, py,i,1) is mapped byHpto (px,i/wi, py,i/wi,1) withwi=w1px,i+w2py,i. Identical weights wi would imply a purely affine transformation. The goal of the approach is to choosew1
and w2 in a way such that the projective transform is as affine as possible. Therefore the authors seek to minimize the variance of all weights and the weight of the image center:
C=
n
X
i=1
wi−wc
wc
2
=
n
X
i=1
hw⊤(pi−pc) w⊤pc
i2
= w⊤PP⊤w
w⊤pcp⊤cw (8.27)
with the 3×nmatrix
P=
p1,u−pc,u p2,u−pc,u ... pn,u−pc,u
p1,v−pc,v p2,u−pc,v ... pn,v−pc,u
0 0 ... 0
(8.28)
The cost can be computed in similar fashion for both images. The optimal projective transforms fully characterized by wb and wm can be obtained by minimizing the functional given by summing the costs Cb+Cm. However, the vectors wb and wm are not independent and are related by epipolar geometry.
Following [Hartley and Zisserman, 2004] corresponding lines or directionslb andlmacross two views satisfy
lm=F[eb]×lb (8.29)
where F is the fundamental matrix and eb is the epipole. Let zb = (α, β,0) be a direction in the first image Ib. Using the result from equation 8.29 and parameterizing the weight vector as wb = [eb]×z the correspondence is obtained bywm=Fz. Substitution into equation 8.27 leads to the functional
Cb+Cm= z⊤[e]⊤×PbP⊤b [e]×z
z⊤[e]⊤×pb,cp⊤b,c[e]×z+ z⊤F⊤PmP⊤mFz
z⊤F⊤pm,cp⊤m,cFz :=z⊤Abz
z⊤Bbz+z⊤Amz
z⊤Bmz (8.30) to be minimized. Using the general homogeneous point coordinatespx,y= [x, y,1] and the center coordinates pc= [w−12 ,h−12 ,1] the components in equation 8.30PP⊤ andpcp⊤c can be further simplified to
PP⊤ =wh 12
w2−1 0 0
0 h2−1 0
0 0 0
(8.31)
98 Image Rectification and
pcp⊤c = 1 4
(w−1)2 (w−1)(h−1) 2(w−1) (w−1)(h−1) (h−1)2 2(h−1)
2(w−1) 2(h−1) 4
(8.32)
The functional given by equation 8.30 is a non-linear optimization problem with respect to z = [α, β].
Remember that z is a direction and defined up to a scale factor, thus we can set β = 1 without loss of generality. The minimum of equation 8.30 is obtained for the first derivative with respect to αequating to zero. Starting with an initial guess derived by minimizing Cb andCm independently the minimizer of the non-linear functional can be found by iteratively solving for α. This fully specifies the weights wb andwb
and parametrizes the projective transformHPtransferring epipoles to infinity for the rectified images. In the next step the similarity transform is derived such that epipolar lines in the rectified images are horizontal.
The known fundamental matrixFof original images and the one of the rectified imagesF'(equation 8.23) are related by
F=H⊤bF'Hm (8.33)
By comparison of the single terms in this vector equation the parameters v1, v2 andv3 in Hb,r andHm,r
can be eliminated which results in the similarity transform dependent ofwb,wm andvm,3 solely
Hb,r =
F32−wb,2F33 wb,1F33−F31 0 F31−wb,1F33 F32−wb,2F33 F33+vm,3
0 0 1
(8.34)
Hm,r=
wm,2F33−F32 F31−wm,1F33 0 wm,1F33−F31 wm,2F33−F32 vm,3
0 0 1
. (8.35)
The scalar vm,3 can be chosen in a way that the minimum y-pixel coordinates are zero in either of the images. Up to now we specified transformations assuring that epipolar lines are parallel and horizontal.
Let a= [w−12 ,0,1],b= [w−1,h−12 ,1],c= [w−12 , h−1,1] and d= [0h−12 ,1]. Setting s3 = 0, the missing parameterss1ands2of shearing transform can be analytically computed by claiming that the two linesb−d and c−a are perpendicular and their aspect ratio is preserved. The final homographies are obtained by multiplication of all sub-transformsHb=Hb,sHb,rHb,pandHm=Hm,sHm,rHm,p. Once the homographies are derived the dimensions of rectified images can be computed in a similar way as within the rectification approach before. Although this rectification method offers a more controlled and interpretable formulation of the rectification process it is still based on homographies and lacks the functionality for pure forward motion configurations.