Apart from cases of occlusion — where there is no 2D–primitiveπrj in the right image such assi→jis the correct stereo — the problem becomes to select the correct correspondence out of those competing hypotheses. In order to distinguish the 2D–primitives, we will associate to each hypothesis a confidence ch
si→ji
based on the similarity between the corresponding 2D–primitives in the left and right image.
The similarity function used is akin to the one defined in section 3.2.3 in the perceptual grouping context. The notable difference is that, because neither collinearity nor co–circularity rules apply between stereo–pairs of 2D–primitives, the geometric constraint reduces to a difference in direction between the pair of primitives — illustrated in Fig. 4.3. Note that, in the general case, thecorrectcorrespondence is expected to be somewhat different from the original 2D–primitive, due to the difference in viewpoints, the sparseness of the representation, and noise. Similarity between stereo–pairs of 2D–primitives is not an exact mapping of the correctness of this correspondence assumption; however, we will show that it is an efficient criterion for identifying the correct correspondence over spurious ones.
4.2.1 Switching in the stereo case
In a similar manner than in the grouping context, the orientation–direction ambiguity needs to be resolved in order to compare two 2D–primitives over stereo. In this case, the constraint one can apply on the interpretation of the two 2D–primitives is a three–dimensional one. The two different cases are illustrated in Fig. 4.3: if the orientation of both 2D–primitives point on thesameside of the epipolar line defined by the left 2D–primitive, no switching is required. On the other hand, if the orientations points todifferent sides of this line, the two interpretations are incompatible; hence, the second 2D–primitive is switched
— as defined in section 2.2.4.
4.2.2 Geometric constraint in the stereo case
We stated before that the similarity measure used for stereo–matching is similar to the one used for the perceptual grouping of 2D–primitives. The main difference lies in the interpretation of the geometric relationship between two 2D–primitives: in the grouping context, we used a geometric constraint that
Figure 4.3: Illustration of the 3D consistency constraint applied on the interpretation of the orientation of primitives in the stereo case. The orientation of the local contour is arbitrary, but has to be chosen consistently between the left and right primitives to ensure that each side of the contour is unambigu-ously identified. Here we see that if the orientation of the primitive is above (respectively below) the epipolar line in the left image, it should also be above (below) in the right image. If not, the two 2D–
primitives’ interpretations are inconsistent: the right 2D–primitive need to beswitched— as explained in section 2.2.4.
CHAPTER 4. USING PRIMITIVES FOR STEREO–RECONSTRUCTION 76
assessed the relative position and orientation of the 2D–primitives in terms of proximity, collinearity, and co–circularity. In the stereo context, there is no equivalent to the good continuation rule; therefore we will simply consider the 2D–primitives’ orientation difference. Because the two cameras view the scene from a different perspective, the orientation of two corresponding 2D–primitives is expected to be somewhat different. Nevertheless, using orientation similarity for stereo–matching yields a performance significantly above chance, as shown in section 4.3.
In the stereo case the geometric constraint is reduced to the normalised angular distance between the 2D–primitives’ orientations:
The three other modal metrics (i.e., phase, colour and optical flow) used in the grouping context (see section 2.3) are used in the same way in the stereo–matching context. Note that, before computing the similarity, the orientation consistency needs to be ensured — as described in section 4.2.1.
We define a multi–modal similarity between a stereo–pair of 2D–primitives as a weighted combina-tion of the individual modal metrics, as follows:
ch
4.2.4 Limits of the epipolar constraint
If we consider a contour which orientation in the right image is nearly parallel to the epipolar line (e.g., an horizontal line), then all 2D–primitives in the right image along such a contour are putative correspon-dences. Furthermore, as they are all extracted from the same contour they hold very similar properties.
This makes it nigh impossible (or at best, unreliable) to identify the true correspondence amongst them
Figure 4.4: Limits of the epipolar constraint. Consider a primitiveπ0, in the left image, and its epipolar line ξ in the second image. Three 2D–primitivesπi,πj,πk are crossed by ξ, designating them all as putative correspondences ofπ0. Those three primitives are all manifestations, in the right image, of the samecontour of the scene; consequently, they are similar in all modalities, and the multi–modal constraint do not allows to choose reliably between those three candidates. Hence, when an image contour’s orientation is similar to epipolar line’s orientation, several 2D–primitives along this contour will be candidates and the correspondence problem cannot be reliably solved by local means.
— illustrated in Fig. 4.4. This is the worst case of stereo–ambiguity that can be found in general scenes, and it is unfortunately common (indeed horizontal structures are fairly common in natural scenes).
This ambiguity cannot be overcome using solely local information. One common strategy is to en-force anordering constrainton the stereo–correspondences. This requires to identify accurate endpoints on those segments, and is unstable in the event of occlusion. Furthermore, because this study focuses on local edge primitives, the endpoints are not directly available to us, and such an approach is unpractical.
Effectively, we chose to disregard primitives with an orientation differing with the one of the epipolar line by less than 10 degrees.