EXISTENCE OF TWO VIEW CHIRAL RECONSTRUCTIONS
ANDREW PRYHUBER, RAINER SINN, AND REKHA R. THOMAS
ABSTRACT. A fundamental question in computer vision is whether a set of point pairs is the image of a scene that lies in front of two cameras. Such a scene and the cameras together are known as achiral reconstructionof the point pairs. In this paper we provide a complete classification ofkpoint pairs for which a chiral reconstruction exists. The existence of chiral reconstructions is equivalent to the non-emptiness of certain semialgebraic sets. We describe these sets and develop tools to certify their non-emptiness. For up to three point pairs, we prove that a chiral reconstruction always exists while the set of five or more point pairs that do not have a chiral reconstruction is Zariski-dense. We show that for five generic point pairs, the chiral region is bounded by line segments in a Schl¨afli double six on a cubic surface with27real lines. Four point pairs have a chiral reconstruction unless they belong to two non-generic combinatorial types, in which case they may or may not.
1. INTRODUCTION
A fundamental question in computer vision is whether a set of point pairs P “ tpui,viq : i “ 1, . . . , kuis the image of a set of world pointsqi that are visible in two cameras. If we ignore the constraint (as is commonly done) that the pointsqineed to lie in front of the cameras, we get aprojective reconstruction[12]. In reality though, cameras can only see points in front of them. A reconstruction that obeys this additional constraint is known as a chiral reconstruction[1, 11]. The aim of this paper is to give a complete answer to the question: Given a set of point pairsP “ tpui,viq, i“1, . . . , ku, when doesPadmit a chiral reconstruction?
Under the assumption that the points in each image are distinct, we prove the following facts.
(1) A set of at most three point pairs always has a chiral reconstruction.
(2) A set of four point pairs has a chiral reconstruction unless the configurations are of two specific non-generic types, in which case a chiral reconstruction may not exist.
(3) Five or more point pairs can fail to have a chiral reconstruction with positive probability (in particular, even if they are in general position).
(4) For five sufficiently generic point pairs, the problem translates to finding points in semialgebraic regions on a real cubic surface whose boundaries are segments of lines in a Schl¨afli double six of real lines, creating an unexpected bridge to classical results in algebraic geometry.
The study of chirality was initiated in [11] with follow up work by Werner, Pajdla and others [21, 22, 23, 24]. There is no agreement on the name for reconstructions that are chiral. Hartley in [11] and Werner & Pajdla in [24] call them strong realizations. Werner in [21, 22, 24] calls themoriented projective reconstructions. Here and in our previous work [1], we prefer the termchiral reconstruction. In [11], Hartley shows that chiral reconstructions are a special class ofquasi-affine reconstructions. See [1, Section 5] for a detailed account of quasi-affine reconstructions in the context of our approach to chirality. Hartley’s work was done using projective geometry. In later work, such as [21], [22] and [23], the authors useoriented projective geometry[16], which also explains their use of the termoriented projected reconstructions. Following Hartley, our work uses projective geometry. Justifications of our choice of framework can be found in [1, Section 1]. We now comment briefly on how our results relate to the existing literature in computer vision and give more detailed citations in later sections.
A specific example of five point pairs in general linear position that do not admit a chiral reconstruction was given in [22] and appears again in [21]. Our result (3) shows that, in fact, the set of five point pairs without a chiral reconstruction is Zariski dense. The case of four point pairs covered in result (2) is the most involved since it does not assume genericity, and covers all possible configurations of four point pairs. We show that chiral reconstructions can fail in this case only in degenerate situations. To the best of our knowledge, both results (2) and (3) are novel. Result (1) says that three point pairs admit a chiral reconstruction unconditionally. While this is not too hard to prove, it also does not appear in the literature and completes the story.
Date: December 7, 2021.
Pryhuber and Thomas were partially supported by the NSF grant DMS-1719538.
1
arXiv:2011.07197v3 [cs.CV] 4 Dec 2021
Result (4) establishes a new connection between chiral reconstructions of five generic point pairs and the classical theory of cubic surfaces from algebraic geometry. The cubic surface inP3arises naturally in our set up and the two camera planes, modeled asP2ˆP2, can be obtained byblowing downthis surface. The papers [22] and [21] study chirality in the setting ofP2ˆP2 using tools from oriented matroids. It was shown in [22] that thechiral regions are bounded by certain conics. Our results show that these conics are obtained by blowing down aSchl¨afli double six of real lines on the cubic surface, allowing the chiral regions to be seen as semialgebraic subsets of the cubic surface bounded by lines. Both frameworks produce a sixth pair of pointspu0,v0qwhose existence was derived in [22] using the conics inP2ˆP2. Our results show that the cubic surface is theblow upof the six pointstuiuortviuin eachP2. The blow up/blow down maps create a new interpretation of the results in [22] while also offering a unified picture of chirality that transfers seamlessly between the space of cameraepipolesand the space offundamental matricesofP.
We draw from, and build on, methods from the above mentioned papers from computer vision, and our own paper [1].
Our main tools are centered in complex and real algebraic geometry as well as semialgebraic geometry.
This paper is organized as follows. Formal definitions of projective and chiral reconstructions can be found in Section 3. In Section 4, we introduce the inequalities imposed by chirality and develop tools to certify them. Along the way we prove that a set of at most three point pairs always has a chiral reconstruction.
In Section 5, we prove that a set of four point pairs has a chiral reconstruction when the point configurations in each view have sufficiently similar geometry, and in particular, when they are in general position. The bad cases fall into two non-generic combinatorial types. In particular, the probability of choosing four point pairs that fail to have a chiral reconstruction is zero.
In Section 6 we show that when k ą 4, point pairs that are in general linear position may not have a chiral reconstruction. Specific examples of this type whenk “ 5 were known to Werner [22] and as mentioned before, there are close connections between our work and that of Werner’s [21, 22, 23, 24]. We make two new contributions for the casek “5. In Section 6 we show that one can decide the existence of a chiral reconstruction whenk “ 5 andP is generic, by checking20discrete points. We use this test to show that the set of five point pairs that do not admit a chiral reconstruction is Zariski-dense. In other words, five point pairs do not have a chiral reconstruction with positive probability. A set of six or more point pairs can have a chiral reconstruction only if any subset of five point pairs among them have one. Hence, for any value ofk ą 5, there will be point configurations without a chiral reconstruction. Our second contribution in Section 7 is to show that the case ofk “ 5 is intimately related to the theory of cubic surfaces from classical algebraic geometry. Indeed,P creates a Schl¨afli double six of12lines on a cubic surface, all of whose27lines are real. These lines determine the boundary of the semialgebraic regions corresponding to chiral reconstructions.
Acknowledgements.The question addressed in this paper is a natural follow up to our previous work on chirality in [1]. We thank Sameer Agarwal for many helpful conversations. We also thank Tom´aˇs Pajdla for pointing us to several chirality papers from the literature.
2. BACKGROUND ANDNOTATION
We now introduce some background and notation that will be needed in the paper. LetPn andPnR denoten- dimensional projective space over complex and real numbers respectively. More generally, we writePpVqfor the projective space over a vector spaceV, which is the set of lines inV. We writea „ bif aandbare the same points in projective space, and reservea “ bto mean coordinate-wise equality. Aprojective camerais a matrix in PpR3ˆ4qof rank three, i.e., a3ˆ4real matrix defined only up to scaling, hence naturally a point in the projective space over the vector spaceR3ˆ4. Usually, an affine representative of a projective camera is fixed and we block- partition such a matrix asA ““
G t‰
P R3ˆ4 whereGP R3ˆ3andt PR3. ThecenterofAis the unique point cAPP3Rsuch thatAcA“0. The cameraAis a rational map from the “world”P3to the “camera plane”P2sending a “world point”qto its “image”Aq. It is defined everywhere except atcA. Consider the hyperplane at infinity inP3, L8 “ tqPP3 : nJ8q “0u, as an oriented hyperplane inR4with fixed normaln8 “ p0,0,0,1qJ. The cameraA is said to befiniteifcAis a finite point, i.e.,cA R L8. A special representative of a camera center can be obtained by Cramer’s rule where theith coordinate ofcAis the determinant of the submatrix ofAobtained by dropping the ith column. In particular, for a finite cameraA “ “
G t‰
, the Cramer’s rule center iscA “detpGqp´G´1t,1qJ. Throughout this paper we use the Cramer’s rule representation ofcA. The cameraA““
G t‰
is finite if and only if detpGq ‰0, and all cameras in this paper will be finite.
2
Theprincipal planeof a finite cameraA““ G t‰
is the hyperplaneLA:“ tqPP3 : A3,‚q“0u,whereA3,‚is the third row ofA, i.e. it is the set of points inP3that image to infinite points inP2. Note that the camera centercA lies onLA. We regardLAas an oriented hyperplane inR4with normal vectornA :“detpGqAJ3‚, which we call the principal rayofA. ThedetpGqfactor ensures that the normal vector of the principal plane does not flip sign under a scaling ofA. The depth of a finite pointqin a finite cameraAis defined as (see [12])
(1) depthpq;Aq:“
ˆ 1
|detpGq|}G3,‚}
˙pnJAqq pnJ8qq.
Note that the sign ofdepthpq;Aqis unaffected by scalingqandA. The depth of a finite pointq P P3 in a finite cameraAdefined in Equation (1) is zero if and only ifnJAq“0which happens if and only ifqlies on the principal planeLA. Otherwise,nJAq‰0andsgnpdepthpq;Aqq “sgnppnJAqqpnJ8qqqis either positive or negative. It is then natural to say that a finite pointqisin front ofthe cameraAifdepthpq;Aq ą 0, see [11]. Since only the sign of depthpq;Aqmatters, we refer to this sign as thechiralityofqinA, denoted asχpq;Aq, which is either1or´1.
The above notion of chirality was introduced by Hartley in the seminal paper [11], where he was concerned with a pair of cameras, see also [12, Chapter 21]. In [1], the definition of chirality was extended to all points inP3, finite and infinite, and defined for an arrangement of cameras. Here is the two camera version we need.
Definition 2.1. LetpA1, A2qbe a pair of finite projective cameras. Then thechiral domainofpA1, A2q, is the Eu- clidean closure inP3of the set
tqPP3|qfinite, depthpq, A1q ą0, depthpq, A2q ą0u.
A pointqPP3is said to havechirality 1 with respect topA1, A2q, denoted asχpq;pA1, A2qq “1, if and only ifqlies in the chiral domain ofpA1, A2q.
In this paper we will be concerned with a pair of finitenon-coincidentcameraspA1, A2qby which we mean that their centers are distinct. We will see that one can always takeA1 “ “
I 0‰
, and then the conditions of finite and non-coincident imply thatA2““
G t‰
whereGPGL3andt‰0. The pairpA1, A2qgives rise to the unique (up to scale) real, rank twofundamental matrixX “ rtsˆGwhere
rtsˆ“
» –
0 ´t3 t2 t3 0 ´t1
´t2 t1 0 fi fl.
The skew-symmetric matrixrtsˆrepresents the cross product withtas a linear map; that means that fort,rPR3we havetˆr“ rtsˆr“ rrsJˆt. Also,rankprtsˆq “2if and only ift‰0. We are only ever interested in properties of fundamental matrices (mostly rank and kernels) that remain unaffected by scaling and will therefore mostly consider them up to scaling which is to say as a point ofPpR3ˆ3q. We chose not to introduce a different notation to distinguish the matrix itself from the line it spans.
Theepipole pairof the cameraspA1, A2qispe1,e2q PP2ˆP2wheree1is the image of the centerc2inA1, and e2is the image of the centerc1inA2. The line joining the centersc1andc2is called thebaselineof the camera pair pA1, A2q. Note that all world points on the baseline (with the exception of the respective camera centers) image to the epipole in each camera.
3. PROJECTIVE ANDCHIRALRECONSTRUCTIONS
Throughout this paper our input is a collection of point pairsP “ tpui,viq : i “1, . . . , ku Ă P2RˆP2R. Each ui(andvi) is the homogenization of a point inR2by adding a last coordinate one. Hencepui,viqis a pair of finite points inP2RˆP2Rwith a fixed representation. We will also assume that allui(and allvi) are distinct.
In this section we formally define projective and chiral reconstructions ofP, and characterize their existence. We then set up the geometric framework within which these reconstructions will be studied in this paper.
3.1. Projective reconstructions.
Definition 3.1. Aprojective reconstructionofP consists of a pair of projective camerasA1,A2 PPpR3ˆ4q, world pointsQ “ tq1, . . . ,qku Ă P3 and non-zero scalarsw1i, w2i such that A1qi “ w1iui and A2qi “ w2ivi for i“1, . . . , k. If the cameras and world points are all finite, thenpA1, A2,Qqis called afinite projective reconstruction ofP.
3
The basics of projective reconstructions can be found in [12, Chapters 9 & 10]. Theorem 3.1 in [17] proves that if Phas a projective reconstruction then it also has a finite projective reconstruction withA1““
I 0‰ .
We now recall the necessary and sufficient conditions for the existence of a finite projective reconstruction ofP. For a pointePP2, letepu1, . . . ,ukqdenote the set of lines joiningeto eachui. The following geometric characterization is well-known [12, 18, 22, 23].
Theorem 3.2. [12, Section 9.4], [18, Section 2.4] The set of point pairsPhas a projective reconstructionpA1, A2,Qq if and only if there exist pointse1,e2PP2and a homography which sends the set ofklinese1pu1,u2, . . . ,ukqto the set ofklinese2pv1,v2, . . . ,vkqwhere the linee1uimaps to the linee2vi.
The pointse1,e2in the above theorem are the epipoles of the camera pairpA1, A2qin the reconstruction. We now give a second characterization of the existence of a projective reconstruction in terms offundamental matrices: Below, we will abuse notation and not distinguish between the matrix and the line it spans in the space of matrices (or in other words the corresponding point in projective space). All relevant properties of the matrices are invariant under scaling.
Definition 3.3. (1) Afundamental matrixofP is a rank two matrixXPPpR3ˆ3qsuch that viJXui“0, fori“1, . . . , k.
(2)
The linear equations(2)inX are called theepipolar equationsofP.
(2) Given a rank two matrixX PPpC3ˆ3qand a pairpu,vq PP2ˆP2, such thatvJXu“0, we say thatX ispu,vq-regularifvJX “0if and only ifXu“0. i.e.,uandvsimultaneously generate the right and left kernels ofX, or neither generate a kernel.
(3) AP-regular fundamental matrixis a fundamental matrix ofP that ispui,viq-regular for each point pair in P.
It is commonly believed thatPhas a projective reconstruction if and only if it has a fundamental matrix. However, a bit more care is needed as in the following theorem.
Theorem 3.4. [17, Theorem 4.6] There exists a finite projective reconstruction ofPwith two non-coincident cameras, andA1““
I 0‰
, if and only if there exists aP-regular fundamental matrix.
Remarks 3.5. (1) LetA1““ I 0‰
andA2““ G t‰
be the two finite non-coincident cameras in the projective reconstruction. Then recall thatG P GL3,t ‰ 0, and the cameras correspond to the unique fundamental matrixX “ rtsˆGup to scale. The epipoles of the cameras aree1 „G´1tande2„twhich generate the right and left kernels ofX. Further,Gdefines a homography ofP2that sendse1toe2, and the linee1uito the linee2vi. HenceGencodes the epipolar line homography of Theorem 3.2.
(2) Conversely, any (rank two) fundamental matrix X ofP can be factored asX “ rtsˆGfor somet P R3 andG PGL3and yield a pair of cameraspA1 ““
I 0‰
, A2 “ “ G t‰
qwhose epipoles generate the left and right kernels ofX. See [12, Section 9.5] for more details on the correspondence between fundamental matrices and camera pairs. These cameras can then be used to reconstruct a set of world pointsQifX is P-regular. The resulting projective reconstruction is said to beassociated toX.
(3) A fundamental matrix isP-regular if and only if for eachi, eitheruiandviare both epipoles of the cam- eras or neither are. Indeed, this is a necessary condition for a projective reconstruction since ifui „ e1, then its reconstructionqi P P3 lies on the baseline of the cameras and hence images toe2 in cameraA2
requiringvi „e2. This subtlety is often overlooked, and it is common to equate the existence of a projective reconstruction ofPto the existence of a fundamental matrix ofP.
(4) Lastly, we remark that [17, Theorem 4.6] is stated using a different notion of regularity. However, a funda- mental matrixXisP-regular in our sense if and only if it isprtsˆX,tq-regular in the sense of [17] and hence the above theorem is exactly [17, Theorem 4.6].
We now discuss the geometry encoded in Theorem 3.4 which will set the foundation for the work in this paper. Even though fundamental matrices are real, we will work overCto allow for methods from complex algebraic geometry, and will specialize toRas needed. A matrixX PPpC3ˆ3qcan be identified with a point inP8by concatenating its rows.
Under this identification we letR2 ĂP8 be the determinantal hypersurface of matrices inPpC3ˆ3qof rank at most two, andR1be its subvariety of rank one matrices. As projective subvarieties ofP8,dimR2“7and degreeR2“3 while,dimR1“4and degreeR1“6. For a point pairpu,vq PP2ˆP2, letLpu,vqdenote the hyperplane inP8:
Lpu,vq“ tX PP8 : vJXu“ xX,vuJy:“TrpXJvuJq “0u
4
wherex¨,¨ydenotes the Frobenius inner product on matrices. LetLP “Şk
i“1Lpui,viq. Generically,LP is a linear space inP8of codimensionk.
Definition 3.6. The varietyR2XLP inP8is theepipolar varietyofP.
Under sufficient genericity ofP,dimpR2XLPq “7´kand degreepR2XLPq “3. Hence, the epipolar variety is empty whenkě8, consists of three points whenk“7, and infinitely many points whenkă7.
Forpui,viq PP, consider the following five-dimensional linear spaces ofP8that are in fact inR2: ĂWui :“ tX PP8 : Xui“0u and ĂWvi :“ tX PP8 : viJX“0u.
Their intersections with the epipolar variety are the linear spaces:Wui :“LPXĂWui andWvi :“LP XWĂvi, each of which generically has dimension6´ksinceWĂui,WĂvi ĂLpui,viq.
Definition 3.7. The linear spaceWui(resp.Wvj) will be called theui(resp.vj)walland the intersectionWuiXWvj will be called thepui,vjqcorner.
Wheni ‰ j, apui,vjqcorner has dimension5´kgenerically since there are at most5independent equations amongvjJX “0“Xui. Thus a wall has codimension one and a corner has codimension two in the epipolar variety, generically. For non-generic dataP, all the dimensions computed above may be larger.
The second condition in Theorem 3.4 can now be rephrased as the existence of a real rank two matrixX in the epipolar variety ofPsuch that for eachi,Xis either in thepui,viqcorner or in the complement ofWuiYWvi. Note that a rank twoXcan lie in at most onepui,viqcorner because the pointsui(andvj) are pairwise distinct.
Going forward, we will work both inP8, the space of fundamental matrices, and inP2ˆP2, the space of epipoles.
These spaces are related by theadjoint map,
adj :P899KP8, X ÞÑadjpXq
whereadjpXq “cofpXqJandcofpXqis the cofactor matrix ofX. IfX PR2thenX¨adjpXq “adjpXq ¨X “0and thus, ifrankpXq “2, then all non-zero rows (resp. columns) ofadjpXqare multiples of each other and generate the left (resp. right) kernel ofX. Since generators of the right and left kernels of a fundamental matrix represent epipoles, the adjoint map provides a convenient connection between epipole space and fundamental matrix space.
3.2. Chiral reconstructions. A physical constraint on a true reconstruction pA1, A2,Qqis that the reconstructed world points inQmust lie in front of the camerasA1andA2. Recall from the Introduction that this means we require Qto lie in the chiral domain ofpA1, A2qor equivalently,χpqi;pA1, A2qq “ 1 for allqi P Q. A full development of multiview chirality can be found in [1]. For this paper, we use the following inequality description of the chiral domain for two views from [1, Theorem 1] as a definition. The cited result shows that these inequalities cut out the Euclidean closure of the set in Definition 2.1 (under the mild assumption that it has non-empty interior).
Definition 3.8. Achiral reconstructionofPis a projective reconstructionpA1, A2,QqofP with finite non-coindent cameras such that for alli,
pnJ8qiqpnJ1qiq ě0, pnJ8qiqpnJ2qiq ě0, and pnJ1qiqpnJ2qiq ě0 wheren8“ p0,0,0,1qJandniis the principal ray ofAi.
Recall that two projective reconstructionspA1, A2,Qqand pA11, A12,Q1q are projectively equivalentif they are related by a homography ofP3, i.e., there is aH P GL4 such thatA1i “ AiH´1andQ1 “ HQ :“ tHqi, i “ 1, . . . , ku. A projective reconstruction which is not chiral can sometimes be transformed into a chiral reconstruction by a homography [1, 11, 23]. We recall the conditions under which this is possible.
Theorem 3.9. Consider a finite projective reconstruction ofP with non-coincident camerasA1 “ “ I 0‰
,A2 “
“G t‰
, and world pointsQ“ tq1, . . . ,qku ĂP3. Then the following are equivalent.
(1) There exists a projectively equivalent chiral reconstructionpA1H´1, A2H´1, HQqofP.
(2) pnJ1qiqpnJ2qiqhas the same sign for alli.
(3) w1iw2ihas the same sign for alli.
Furthermore, if noqilies on the baseline ofpA1, A2q, then (1), (2), (3) are equivalent to (4) ptˆviqJptˆGuiqhas the same sign for alli.
5
Proof. The equivalence of (1) and (2) is Theorem 8 in [1]. The equivalence of (1) and (3) is Theorem 17 in [11]. The epipoles of the given cameras aree1„G´1tande2„t, and hence if no world point lies on the baseline,G´1tui
(equivalently,tGui) andt vi for anyi. Also, sinceA1, A2 are non-coincident andt‰ 0, Theorem 3.2 and Item 1 imply thatt,vi, Guiare collinear. Therefore,ptˆviq „ ptˆGuiqand soptˆviqJptˆGuiq ‰0for alli.
The equivalence of (3) and (4) can then be derived from the same arguments as in Lemma 7 in [1].
If a world point lies on the baseline, then its imagespui,viqinA1, A2are the epipoles of the cameras, and the expression in (4) becomes zero. However, since any point on the baseline can serve as the world pointqiimaging to pui,viq, we can control the sign ofpnJ1qiqpnJ2qiqas shown next.
Lemma 3.10. For a pair of non-coincident cameraspA1, A2q whose baseline is not contained in either principal planeLA1andLA2, there existq`andq´on the baseline such thatpnJ1q`qpnJ2q`q ą0andpnJ1q´qpnJ2q´q ă0.
Proof. Since c1 P LA1, nJ1c1 “ 0. On the other hand, since the baseline is not contained inLA2 andc1 ‰ c2, c1 R LA2, and sopnJ2c1q ‰ 0. By continuity, there exist perturbationsq` andq´ ofc1on the baseline such that pnJ1q`qpnJ2q`q ą0andpnJ1q´qpnJ2q´q ă0.
Remarks 3.11. For reconstructions where both epipoles are finite, the hypothesis of Lemma 3.10 is satisfied. Indeed, if for instance the baseline was contained in the principal planeLA1 thenc2 PLA1and soe1 „A1c2would be an infinite point.
We now have a necessary and sufficient condition for the existence of a chiral reconstruction.
Lemma 3.12. There exists a chiral reconstruction ofP if and only if there existtPR3zt0uandGPGL3such that rtsˆGis aP-regular fundamental matrix and
`ptˆviqJptˆGuiq˘ `
ptˆvjqJptˆGujq˘
ě0 for all 1ďiăjďk.
(3)
Proof. SupposepA1, A2,Qqis a chiral reconstruction ofP with non-coincident finite cameras whereA1 ““ I 0‰
. ThenA2 “ “
G t‰
for somet P R3zt0uandG P GL3, and by Theorem 3.4, rtsˆGis a P-regular fundamental matrix associated toA1andA2. For allisuch thatqiis not on the baseline,ptˆviqJptˆGuiqhas the same sign by Theorem 3.9. If some world pointqiis on the baseline, then its imagepui,viqis the pair of epipolespG´1t,tq, and henceptˆviqJptˆGuiq “0. Therefore, the inequalities in (3) hold.
Conversely, suppose there existtPR3zt0uandGPGL3such thatrtsˆGis aP-regular fundamental matrix and the inequalities (3) hold. By Theorem 3.4, there exist world pointsQsuch thatpA1““
I 0‰
, A2 ““ G t‰
,Qqis a projective reconstruction ofP. LetQp ĎQbe the set of world points not on the baseline ofA1andA2. Since the inequalities (3) hold, the quadruple productsptˆviqJptˆGuiqhave the same sign for allqiPQ. By Theorem 3.9,p there exists a constantσP t´1,1usuch thatσ“sgnpnJ1qiqpnJ2qiqfor allqi PQ.p
If some pointqj PQlies on the baseline, thenqj images to the pair of epipolesG´1tandtin the two cameras and henceptˆvjqJptˆGujq “0. By Lemma 3.10, we may replaceqjby some world pointq1jon the baseline such thatsgnpnJ1q1jqpnJ2q1jq “ σ. LetQ1be the modification ofQobtained by replacing all world points on the baseline as above, but keeping all other world points intact. By construction,sgnpnJ1q1iqpnJ2q1iq “ σfor allq1i P Q1. The transformed reconstructionpA1, A2,Q1qis projectively equivalent to a chiral reconstruction by Theorem 3.9.
Lemma 3.12 implies that for a chiral reconstruction to exist, there must beP-regular fundamental matrices that satisfy the inequalities (3). In the next section, we examine these inequalities to understand the regions of the epipolar variety in which fundamental matrices that lead to chiral reconstructions live.
4. CHIRAL TOOLS
In this section we develop tools to prove the existence of chiral reconstructions. In Section 4.1, we describe the semialgebraicchiral epipolar regionof fundamental matrices associated to chiral reconstructions ofP. In Section 4.2, we show how inequalities defining the chiral epipolar region can be checked in epipole space. Even if not stated explicitly, we are working overPRwhen dealing with inequalities. We combine these tools in Section 4.3 to prove that a set of three point pairs has a chiral reconstruction. In Section 4.4, we show how the walls and corners of the epipolar variety can be used to decide if a set of more than three point pairs has a chiral reconstruction.
6
4.1. The chiral epipolar region. By Lemma 3.12, a fundamental matrix must satisfy (3) to yield a chiral reconstruc- tion. In this section, we describe the strict subset of the epipolar variety satisfying these constraints.
Definition 4.1. LetX PPpR3ˆ3qdenote a real3ˆ3-matrix up to scaling. For each point pairpui,viq, thei-th chiral polynomialis
gipXq:“vJi r´tsˆXui“ ptˆviqJXui
wheretJX“0. The set of allgigjpXq “gipXqgjpXq ě0are called thechiral epipolar inequalities ofP. Here, the same representativetfor the left-kernel ofXmust be used ingiandgj.
Thei-th chiral polynomial is, strictly speaking, not a polynomial because there is no way to write a generator of the left kernel of a matrixXas a polynomial expression that works for every3ˆ3matrix of rank2. To be technically precise, it is a section of a line bundle on the quasi-projective variety of3ˆ3 matrices of rank exactly two. We avoid these technicalities and argue that the chiral epipolar inequalities are well-defined in an elementary way using the adjoint: WritingtwithtJX “0in terms ofXis the composition of the adjoint mapadj :R2zR1ÑP8, whose image isP2ˆP2in its Segre embedding, with the projectionP2ˆP2 ÑP2. So locally,tis given as a row of the adjoint matrixadjpXq(but only on the open set where that row is non-zero). The entries of the matrixr´tsˆX are polynomials of degree three in the entries ofX. This shows that the inequalitiesgigjpXq ě0are locally of degree six in the entries ofX. Also, if two rowstandt1ofadjpXqare non-zero and differ by a negative multipleλPRă0, i.e.t“ λt1, the sign ofgigjpXqdoes not change because it essentially differs byλ2. Therefore the sign ofgigj is well-defined for every real3ˆ3matrix of rank two up to scaling, i.e. for every fundamental matrix. The set of real matricesX inR2for whichgigjpXq ě0is a semi-algebraic subset ofPpR3ˆ3qin the following sense: There is an open affine cover ofR2zR1(by sets on which we can writetas a polynomial function ofX), such that the inequalities gigjpXq ě0become polynomial and hence define a semi-algebraic set in each open subset of (the real points in) this cover. On the intersection of any two open sets in the cover, the regions cut out by these inequalities agree.
We now show that the chiral polynomialgipXqrecords the quadruple productptˆviqJptˆGuiqfrom Lemma 3.12.
Lemma 4.2. IfX “ rtsˆG, thengipXq “ ptˆviqJptˆGuiqfor eachi.
Proof. gipXq “vJir´tsˆXui“ pr´tsJˆviqJrtsˆGui“ prtsˆviqJrtsˆGui“ ptˆviqJptˆGuiq.
The next theorem, which is analogous to Theorem 3 in [23], now follows from Lemma 3.12 and Lemma 4.2.
Theorem 4.3. There exists a chiral reconstruction ofPif and only if there exists aP-regular fundamental matrixX such thatgigjpXq ě0for all1ďiăjďk.
Definition 4.4. Thechiral epipolar region ofPis the set ofP-regular fundamental matricesXsuch thatgigjpXq ě0 for all1ďiăjďk.
The chiral epipolar region ofP is contained in the semialgebraic subset of the real part of the epipolar variety R2XLP that is cut out by the chiral epipolar inequalities. It is not necessarily equal to this set because the chiral epipolar region additionally requires the fundamental matrices to beP-regular. However, sinceP-regularity only fails on a proper algebraic subset, if the chiral epipolar region has non-empty interior, the boundary of the interior is determined by the points where the chiral epipolar inequalities change sign, which we study next.
Lemma 4.5. LetX be a fundamental matrix ofP. ThengipXq “0if and only ifX PWui orXPWvi.
Proof. Clearly,gipXq “ vJi r´tsˆXui “0 ifXui “ 0. IfvJiX “ 0, thenvi andtare collinear and therefore vir´tsˆ“0, which impliesgipXq “0. For the other implication, we know thatvJiXui “0andvJir´tsˆXui“0, where the three vectorsvi,ui, andtare real and non-zero. We assume thatXui ‰0and show thatvJiX “0. We knowXuiis orthogonal tovi and´tˆvi. Therefore, it must be collinear withviˆ pviˆtq, which is the same aspvJitqvi´ pvJi viqt. We also know thattJX “0, which implies thattis also orthogonal toXui, hence also to viˆ pviˆtq. The dot producttJ`
pvJitqvi´ pvJi viqt˘
“ 0, i.e.pvJitq2 “ pvJi viqptJtq. The Cauchy-Schwarz inequality implies thattandviare collinear, which implies the claimvJi X“0.
The goal of the paper is to understand when the chiral epipolar region ofPis non-empty, or equivalently, whenP has a chiral reconstruction. Whenk“7, genericallyR2XLP consists of three points and it is easy to check if the real points lie in the chiral epipolar region ofP. Therefore, our focus will be on values ofkă7.
7
4.2. Translating to epipole space. In this section, we show how we can check the validity of chiral epipolar inequal- ities inP2RˆP2R, the space of epipoles. Consider thep1,1q-homogeneous quadratic polynomial
Dijpu,vq:“det“
ui uj u‰ det“
vi vj v‰ (4)
wherepu,vq PP2RˆP2R. Note thatDijpu,vq “0if and only if either factor is zero which is if and only ifui,uj,u are collinear orvi,vj,vare collinear. Werner uses the quantitiesDijpu,vqto impose an orientation constraint on the epipolar line homography described in our Theorem 3.2, see [22, Section 5] and [21, Section 6.5]. We will show that Dijpu,vqis closely related to the products of chiral polynomialsgigjpXqwhereuandvgenerate the right and left kernels of a fundamental matrixX. We rely on the following well known identity [5, 21].
Lemma 4.6. Suppose q1,q2,q3 P R4. Let A “ “ G t‰
be a finite camera with Cramer’s rule center cA “ detpGqp´G´1t,1q. Thendet“
Aq1 Aq2 Aq3‰
“det“
q1 q2 q3 cA‰ .
Lemma 4.7. Consider a projective reconstructionpA1, A2,QqofP, i.e.,A1qi “w1iui andA2qi “w2iviwhere wij‰0. SupposeDijp´A1c2, A2c1q ‰0whereciis the Cramer’s rule center ofAi. Then
sgnDijp´A1c2, A2c1q “sgnpw1iw2iqpw1jw2jq.
Proof. ExpandDijp´A1c2, A2c1qas follows.
Dijp´A1c2, A2c1q “ ´det“
ui uj A1c2‰ det“
vi vj A2c1‰ (5)
“ ´det
” 1
w1iA1qi w1
1jA1qj A1c2ı det
” 1
w2iA2qi w1
2jA2qj A2c1ı (6)
“ ´det
” 1
w1iqi w1
1jqj c2 c1ı det
” 1
w2iqi w1
2jqj c1 c2ı (7)
“ 1
w1iw1jw2iw2j det“
qi qj c1 c2
‰det“
qi qj c1 c2
(8) ‰
“ 1
w1iw1jw2iw2j
pdet“
qi qj c1 c2‰ q2. (9)
Equation (7) follows from Equation (6) by applying Lemma 4.6 to both determinants in the product. Since
Dijp´A1c2, A2c1q ‰0by assumption, we conclude thatsgnDijp´A1c2, A2c1q “sgnpw1iw2iqpw1jw2jq.
Lemma 4.8. LetX be a fundamental matrix ofP. SupposeDijpadjpXqt,tq ‰0wheretJX “0. Then sgnDijpadjpXqt,tq “sgngigjpXq.
Proof. Write X “ rtsˆGfor t P R3zt0uand some G P GL3. We know thattandadjpXqtgenerate the one- dimensional left and right kernels ofX, respectively. SinceDijpadjpXqt,tq ‰0, we know thatt‰0,adjpXqt‰0, adjpXqtis not collinear withuianduj, andtis not collinear withvi andvj. In particular, this means that neither uinorujis the right kernel ofX and neithervi norvj is the left kernel ofX. It follows thatX ispui,viqregular andpuj,vjqregular and neither is the epipole pair. By Theorem 3.4, there exists a finite projective reconstruction pA1 “ “
I 0‰
, A2 “ “ G t‰
,tqi,qjuq of tpui,viq,puj,vjqu such that the world points qi,qj are not on the baseline.
Lemma 4.7 implies thatsgnDijp´A1c2, A2c1q “sgnpw1iw1jqpw2iw2jq. By Theorem 3.9,pw1iw1jqpw2iw2jq ą 0if and only if“
ptˆviqJptˆGuiq‰ “
ptˆvjqJptˆGujq‰
ą0. Combining this fact with Lemma 4.2, it follows thatsgnDijp´A1c2, A2c1q “ sgngigjpXq. Finally note thatA2c1 “tand´A1c2 “ detpGqpG´1tqwhich is a positive multiple ofadjpXqt. Indeed,
adjpXqt“adjprtsˆGqt“adjpGqadjprtsˆqt“detpGqG´1pttJqt“ }t}2detpGqpG´1tq.
(10)
SubstitutingadjpXqtfor´A1c2andtforA2c1, the result follows.
The computation in the previous proof, in particular (10), shows that iftis a non-zero generator of the left kernel of a fundamental matrixXthenadjpXqtis a non-zero generator of the right kernel ofX.
Note thatdet“
ui uj A1c2
‰can be zero withoutuiorujbeing the epipoleA1c2. Indeed, by Lemma 4.6 this happens wheneverqi,qj,c1,c2are coplanar. On the other hand, Lemma 4.5 implies thatgivanishes atX if and only ifuiorviis an epipole ofX. Therefore,DijpadjpXqt,tqmay vanish even whengigjpXq ‰0.
Lemma 4.8 shows that knowing the specific generators of the kernels ofX, i.e.,tandadjpXqt, respectively, is enough to compute the sign of the chiral epipolar inequalities. Note that a choice of generatortfor the left kernel
8
ofX determines a signed generatoradjpXqtfor the right kernel. WhenDijpadjpXqt,tqdoes not vanish, we can use it to infer the validity of chiral epipolar inequalities via Lemma 4.8, and hence argue for the existence of a chiral reconstruction ofP. We now identify a situation where we can useanygenerators of the kernels ofXinDij. Definition 4.9. SupposeX is a fundamental matrix ofP. DefineIpXqto be the set of indicesisuch thatgipXq ‰0, i.e., the index set of inactive chiral polynomials atX. LetPIpXqbe the subset of point pairs inP indexed byIpXq.
Theorem 4.10. Let X be a fundamental matrix of P where e1 and e2 generate the right and left kernels of X, respectively. Suppose|IpXq| ě3, andDijpe1,e2q ‰0for alli, j PIpXq. Then there exists a chiral reconstruction ofPIpXqassociated toX if and only ifDijpe1,e2qhas the same sign for alli, jPIpXq.
Proof. Suppose there exists a chiral reconstruction ofPIpXqassociated toX. Then by Theorem 4.3,gigjpXq ě0for alli, jPIpXq. In fact,gigjpXq ą0for alli, jPIpXqsince ifgigjpXq “0for somei, jwhileDijpadjpXqt,tq ‰0, we would contradict Lemma 4.8. Indeed, ifDijpe1,e2q ‰0for some kernel generatorse1,e2, it remains non-zero for any other pair of kernel generators. By Theorem 4.3,DijpadjpXqt,tqhas the same sign for alli, j, and sinceadjpXqt andtare (non-zero) generators of the right and left kernels ofX, the result follows.
Conversely, supposeDijpe1,e2qhas the same non-zero sign for alli, jPIpXqwheree1ande2generate the right and left kernels ofX, respectively. Thene1“λadjpXqe2for some non-zeroλby (10). By Lemma 4.8, we know
sgnDijpe1,e2q “λsgnDijpadjpXqe2,e2q “λsgngigjpXq
for alli, j. This shows thatgigjpXqhas the same sign for alli, jPIpXq. Since|IpXq| ě3, this common sign cannot be negative and hencegigjpXq ą 0for alli, j PIpXq. These strict inequalities also imply thatX isPIpXq-regular.
Then by Theorem 4.3 there is a chiral reconstruction ofPIpXqassociated toX.
We remark thatDijpe1,e2qdoes not have a well-defined sign onP2RˆP2Rbecause it is linear ine1ande2. To get an inequality description of chirality in epipole space, we can take pairwise productsDijpe1,e2qDikpe1,e2qwhich are quadratic in eachP2Rfactor. If Dijpe1,e2qDikpe1,e2q ą 0, thengjgkpXq ą 0 for any fundamental matrix X with epipolese1ande2. However, sinceDijpe1,e2qmay vanish even whengigjpXqdoes not, we observe that Dijpe1,e2qDikpe1,e2q ě0for all triplesi, j, k is not equivalent togigjpXq ě0, gigkpXq ě0andgigkpXq ě 0.
Due to this subtlety, we primarily study chirality usinggigjě0inP8Ras opposed toDijDikě0inP2RˆP2R. 4.3. Three point pairs always have a chiral reconstruction. In this section, we apply the tools developed so far to show that there is always a chiral reconstruction when|P| “3, and hence also when|P| ď3sincePcan have a chiral reconstruction only if all its subsets have one. We begin with two technical lemmas.
Lemma 4.11. Supposea1,a2,a3 are three non-collinear points inR3. Then for each of the eight elements inσ P t`,´u3, there is anePR3such thata1,a2,a3,eare in general position (no three in a line) and
σ“ psgnpdetra1a2esq,sgnpdetra1a3esq,sgnpdetra2a3esqq.
Further, for eachσP t`,´u3, the corresponding choices ofecome from an open polyhedral cone inR3.
Proof. The expressiondetraiajes “lijpeqis the linear form whose kernel is the span ofaiandaj. Sincea1,a2,a3
are non-collinear, the hyperplanes cut out byl12peq, l13peq, l23peqdivideR3 into eight regions, each of which is a polyhedral cone. The interiors of these cones correspond to the eight sign patternsσ.
Forv1,v2PR3, letconepv1,v2q:“ tλ1v1`λ2v2:λ1, λ2ě0ube the convex cone spanned byv1andv2. Lemma 4.12. Supposevl,vr,tare points in R3 on an affine lineL. If t R conepvl,vrq, then for allw1,w2 P conepvl,vrq,ptˆw1qJptˆw2q ą0.
Proof. If t R conepvl,vrq, then either vl P conept,vrq or vr P conepvl,tq. Suppose vl P conept,vrq. Since w1,w2 Pconepvl,vrqandconepvl,vrq Ďconept,vrq, we knoww1,w2 Pconept,vrq. Writew1 “λ1t`λ2vr andw2“µ1t`µ2vrwhereλi, µjě0. Sincewi‰t,λ2, µ2ą0. The result follows from direct computation using thatt‰vr:
ptˆw1qJptˆw2q “ ptˆ pλ1t`λ2vrqqJptˆ pµ1t`µ2vrqq “λ2µ2ptˆvrqJptˆvrq ą0.
(11)
Similar reasoning applies ifvrPconepvl,tq.
Theorem 4.13. If|P| “3thenP has a chiral reconstruction.
9
Proof. We break the proof into two parts:
(1) SupposeU “ tu1,u2,u3uorV “ tv1,v2,v3uis in general position, sayU is non-collinear. Choosee2not on the line spanned byvi andvj for anyi, j, so thatdet“
vi vj e2‰
‰0 for alli, j. By Lemma 4.11, there exists ane1such thatDijpe1,e2qhas the same non-zero sign for alli, j. Sincek“3ande1ande2are chosen from open regions,We1XWe2contains at least one rank two matrixX. By construction, thisX is a P-regular fundamental matrix with epipolese1,e2andIpXq “ t1,2,3u. By Theorem 4.10,Xyields a chiral reconstruction ofP.
(2) Suppose bothU andVare collinear and consider the affine linesLU andLVinR3spanned by these points, which all have last coordinate1. Letul,urbe the furthest left and right points on theLUline, so that the third point lies strictly betweenul,ur. Similarly letvl,vrbe the furthest left and right points on theLV line. Let tPLVzconepvl,vrqand chooseGPGL3such thatGul “vlandGur“vr. DefineX “ rtsˆG. Since t,vi, Guiare collinear for alli, theith epipolar equation is satisfied. Since the chosen epipoles forXdo not coincide with any data points,Xis aP-regular fundamental matrix. By constructionGuiPconepvl,vrqfor eachi. Combining Lemma 4.2 and Lemma 4.12, it follows thatgipXq ą 0for eachi, and there is a chiral reconstruction ofPassociated toXby Theorem 4.3.
4.4. Walls and Corners. To understand the existence of chiral reconstructions when|P| ě 4, we need one more tool that we now develop. Recall that the chiral epipolar region ofP is the set ofP-regular fundamental matrices that live in the semialgebraic subset of the real epipolar variety cut out by the chiral epipolar inequalities. Lemma 4.5 implies that the chiral epipolar region is bounded by theWui,Wvj walls. The fundamental matrices on walls are generallyP-irregular and do not correspond to a reconstruction. However, we show thatP-irregular fundamental matrices that are smooth points of the epipolar variety and yield partial chiral reconstructions, can be perturbed to P-regular fundamental matrices that yield chiral reconstructions ofP.
Lemma 4.14. SupposeR2XLP is irreducible. IfX is a smooth fundamental matrix that ispui,viq-irregular, then there is a tangent directiondPTXpR2XLPqsuch that the directional derivativeDdgipXq ‰0.
Proof. SupposeX is a smooth fundamental matrix ofP. Smoothness implies that the tangent space at X to the epipolar variety has the same dimension as the variety. If X ispui,viq-irregular for somei, thenX is in exactly one ofWui or Wvi. SinceR2XLP is irreducible, each wall must be an embedded component of strictly smaller dimension. This means that the wall’s tangent space is strictly contained in the tangent space of the epipolar variety atX. Therefore, we can choose a directiondtangent to the epipolar variety atX which is not tangent to the wall which containsX. Lemma 4.5 implies thatgivanishes on the real part of the epipolar variety only on the walls. By
constructionDdgipXq ‰0.
The following lemma is needed for Theorem 4.16 below, but its proof might be best understood after Section 7.
Lemma 4.15. Suppose|P| ď5andR2XLP is irreducible. If a wallWui(orWvi) contains a matrix of rank two, then a generic pointY on the wall is a smooth point ofR2XLP.
Proof. We reduce to the case of dimpLPq “ 3 as follows. If the wall contains a smooth point, then so will its intersection with generic data planesLpuj,vjq. Therefore, cutting with sufficiently many of these, using Bertini’s Theorem, we can assume thatLP has dimension three,R2XLP is an irreducible cubic surface inP3, andWuiis a line on it. Suppose for contradiction thatR2XLP is singular at every point inWui.
The cubic surfaces which are singular along a line have been classified, see e.g. [3, in particular Case E]. We show thatR2XLP cannot be any of these types, essentially because it contains too many intersecting lines. Indeed,Wul intersectsWvm as long asl ‰ mbecause the equationsXul “ 0andvJmX “0impose at most three additional conditions on the three dimensionalLP. Additionally, the assumption that the wallWuicontains a matrix of rank two implies that this wall does not coincide withWuj forj‰i.
The first examples of cubic surfaces singular along a line are the cones over a singular plane cubic curve. Our epipolar variety cannot be such a surface because it contains intersecting lines with distinct intersection points, which these cones do not. There are only two other types of cubic surfaces (up to change of coordinates) that are singular along a line.
The next type that is singular along a line, contains a one-dimensional family of lines, and one more line. A representative is given by the equationw2y`x2z, which contains the linesVpw, xq,Vpy, zqand a family of lines that
10
