Related Work

Scaled Eucl.

registration problems

point-to-point

with scaling

[*],[+]

fixed scaling

point-to-line

central case (PnP)

[1],[2]

[1],[2],[3],[*],[+]

non-central case (Generalized

PnP)

with scaling [3]

[1],[2],[3],[*],[+]

fixed scaling [2]

[2],[+]

point-to-plane

central case (PnL)

[1],[2],[3],[*],[+]

non-central case

with scaling

[1],[2],[3],[*],[+]

fixed scaling

[2],[+]

Figure 2.1.: Applicability of existing rotation solvers [1] [HR11],[2] [KLS14],[3] [SFHT14] to various registra-tion problems as originally proposed (marked in red) and when using our orthogonal complement formulation (marked in blue). We also propose an additional rotation solver for the central cases with the lines and planes having a common intersection point and for the registration with variable scale (marked with an asterisk [*] [WK17]) as well as a rotation solver which is applicable to almost all cases [+].)

2.2. Related Work

Due to the high relevance for computer vision, most of the relevant work addresses camera pose estimation from 2D image information. In rare cases, other modalities are also considered, such as the measurements of a tactile robotic arm or line scans of a laser projector system [Che90,OKO09]. We will therefore focus on the area of absolute pose estimation, which commonly refers to the determination of Euclidean or similarity transformation parameters (pose) using image measurements with known associations to 3D reference data in a given coordinate system. By contrast,relative pose estimationis purely based on 2D measurements, notably between the views of two or more cameras [Nis04,LHhK08,KL14,Har97b,EE11]. As for the latter, the choice of the coordinate system is subject to some arbitrariness (gauge freedom) [TMHF00], so one deliberately selects the coordinate system of one of the cameras as reference, therefore the label’relative’. In this chapter, we are concerned with the former, and absolute pose algorithms can be categorized according to the following criteria:

• Type of correspondence:Camera pose estimation from 2D image point measurements and corresponding 3D points is termedperspective-n-point problem(PnP), whereas the pose estimation from 2D line segments in the image and corresponding 3D model lines is referred to as theperspective-n-line problem (PnL).

Within the generalized perspective of this work, we also consider pose estimation between two sets of 3D points known as theabsolute orientation problem[AHB87,Hor87,HHN88,Ume91,Lou16].

• Central versus non-central: Pose estimation from a single image can often be modeled with the pinhole camera model with a single center of projection (central PnP or PnL). Application examples of non-central or generalized camera models [SRT^∗11,Pon09] include pose estimation for calibrated multi-camera sys-tems [Ple03,PFC^∗17], registration of SLAM or SfM trajectories with regard to pre-reconstructed maps [CSP16,SFHT14], wide-angle catadioptric cameras [ATR10], or cameras observing the scene through a planar refractive medium as in underwater imagery, resulting in infinitely many projection centers along a line (axial cameras) [ARTC12]. Pose estimation from an arbitrary configuration of measurement rays is referred to as the generalized PnP / PnL (GPnP / GPnL) problem.

• Minimal versus overconstrained: For solving the six degrees-of-freedom of the pose at least three 2D image measurements are necessary - each of them imposing two constraints (P3P and P3L). Efficient

minimal solvers play an important role inside a RANSAC-loop [FB81] for handling outlier measurements.

However, it is desirable to have algorithms that can also work on a larger set of correspondences in order to suppress noise by optimally exploiting the redundancy.

• Iterative versus closed-form: Iterative approaches aim at successively reducing the error in each step, whereas closed-form solvers compute the solutions in a single pass - albeit possibly using higher level operations (e.g. SVD, Eigenvalues, polynomial root-finding) that internally may require iterative tech-niques [MP13].

Minimal solvershave been proposed for the central P3P [FB81,LHD88,HLON91,GHTC03], the generalized (non-central) P3P [CC04,NS07,HLLPF16,MA15], the central (perspective) P3L [DRLR89] and the non-central P3L problem [Che90,NF93]. In all of these cases, the problem is reduced to an eighth-order univariate polyno-mial from which up to eight solutions are obtained. In the P3P / GP3P case the derivation is typically based on the requirement that the distances between any two points must be preserved after rigid transformation, resulting in a polynomial parameterized with the unknown depth of one of the image points. The fastest and most stable P3P algorithm to date is proposed by Kneip et al. [KSS11], where solutions for the camera pose are calculated directly without the need of the intermediate point depths. In another work by Kneip et al. [KFS13], a minimal solver for the GP3P problem can be found, which uses the Cayley parameterization for rotations and the Gröbner basis technique. In the P3L / GP3L case, usually the coplanarity constraint between the transformed model line and a plane containing the image line segment is imposed. Using an intermediate coordinate frame the poly-nomial is formulated as a function of one of the Euler angles of the rotation. Ramalingam et al. [RBS11] also proposed a minimal solver for a mix of three point or line correspondences (hybrid P3P / P3L).

For generalized cameras the scale becomes important. For example, for pose computation with stereo rigs, the displacement of the camera centers must be defined in the same unit system as the 3D reference points.

Otherwise, incorrect results are obtained or there even may not exist a feasible solution. Conversely, it is also possible to estimate a scaling parameter (7 DoF) if another fourth image measurement is added. Ventura et al. [VARS14b] proposed a minimal solver for the generalized pose and scale problem (GP3P with scale). They set up a linear system of equations and determine the optimal weights for the null space singular vectors by reimposing orthogonality on the rotation matrix by means of ten quadratic equations in 21 monomials. Again, up to eight solutions are obtained with a Gröbner basis solver. Recently, Kukelova et al. [KHF16] presented a general solution for the intersection of three quadrics in three variables. Its application to the generalized pose and scale problem resulted in a very efficient minimal algorithm that also covered the failure cases of Ventura’s approach (planar scene configuration). Camposeco et al. [CSP16] proposed a minimal solver from two image rays and one 3D point (obtained by triangulation).

For theoverconstrained casethere exist many closed-form algorithms that determine a solution by using al-most exclusively tools from linear algebra, in particular the singular value decomposition (SVD). The earliest variant - the DLT-algorithm [Sut74,Gan84,HZ04] - initially computes a linear solution (null space) to a set of equations, assuming that the nine entries of the rotation matrix are independent. This is followed by a subsequent re-orthogonalization of the rotation. Several variants of DLT for the central PnL problem have been recently published by Pˇrilbyl et al. [PZv17]. Although these algorithms can determine a solution in linear time if properly implemented, they have limitations concerning the accuracy. More importantly, for less than six points, five lines or planar scene configurations the null space is spanned by more than one singular vector, rendering these algo-rithms inapplicable to such settings. The ’lifting technique’ formalized by Ansar and Daniilidis [AD03] consists of reimposing nonlinear constraints to a general set of null-space vectors. The extended vector of monomial terms is again solved via SVD, which results in an algorithm having octic complexity with regard to the number of correspondences. Other approaches require a cubic [QL99] or quadratic [Fio01] effort. The idea of the EPnP approach [LMNF09] consists of expressing the equations in terms of four control points, for which reimposing nonlinear constraints is a constant operation and the overall effort becomes linear. The EPnP algorithm requires

2.2. Related Work

special treatment for planar scene geometries, is known to be inaccurate for small numbers of correspondences and quasi-planar situations, and minimizes an artificial, non-intuitive algebraic error. The approach by Kneip et al. [KFS13] for generalized cameras (GPnP) is a straight-forward extension of EPnP and thus also inherits most of its drawbacks. Common to all these approaches is, that they use the SVD to estimate the solution space of nonlinear parameters (in particular the rotation). They generally fail whenever the solution is not unique as for minimal or other ambiguous configurations, because the nonlinearity is not modeled appropriately. The GPnP algorithm by Ess et al. [ENvG07] also falls into that category.

A number of iterative approaches have also been proposed that search for the global optimum. Some of them rely on bootstrapping the solution with a scaled orthographic projection approximation [DD95,LHM00].

This may be too coarse whenever the scene depth is very heterogeneous or when using wide-angle cameras, so there is a high risk that a false local minimum will be returned. Others seek for the global minimum by means of semi definite programming and convex relaxations [SP08,KH07,HK10,OKO06,OKO09], but despite linear complexity the overall computational burden remains fairly high. In general, all iterative solvers have in common that they only search for asingle solution. If multiple global optima are present, such as for minimal configurations (three correspondences), or if a local minimum exists with a similar error level, the returned global minimum may nonetheless be the wrong one due to presence of measurement noise. Algorithms that are capable of determining all global and local minima at once offer the possibility of resolving this deception on a higher level within the application.

A closed-form solver (called RPnP), which is applicable in a general central PnP setting (including minimal, ambiguous and overconstrained cases), was proposed by Li et al. [LXX12]. From allK available correspon-dences they select a pair and form triplets with all remainingK−2 correspondences. From each triplet an eighth-order polynomial is derived (as in the minimal P3P algorithms), and they minimize the squared sum of all these polynomials (all expressed in one variable). First order optimality results in a 15-th order polynomial whose roots yield up to eight minima. Since the formulation is centered around the initially chosen correspondence pair, their solution is also particularly dependent on the noise of these two measurements. Xu et al. [XZCK17] is an variant of RPnP for lines (PnL). In a similar way, they consider triplets of line correspondences. And again, in order to obtain a mono-variate polynomial, each triplet contains the same two base lines, which again introduces a noise bias with regard to these measurements.

Recently, several linear complexity closed-from algorithms have been proposed, which can determine all rele-vant minima with regard to an unbiased, least-squares optimality criterion (object space error) for both, minimal or ambiguous and overconstrained cases. With this, they overcome all of the limitations of the earlier approaches.

Among them are in particular to mention: the DLS algorithm for central PnP [HR11], UPnP for generalized PnP (GPnP) with fixed scale [KLS14], gDLS/gDLS+++ [SFHT14,SFHT16] for generalized PnP (GPnP) with vari-able scale and the algorithm of Mirzaei and Roumeliotis [MR11] for PnL. However, although they represent a generalization in their domain, they are nonetheless limited to their specific problem instance and can only be applied to their appropriate correspondence type.

In this chapter, we further unify these recent approaches. To this end, we take a generalized view and re-interpret them as 3D point-to-line, point-to-plane and point-to-point registration problems. We follow their general strategy by eliminating the linear parameters first, which results in a least squares error function that only depends on the nonlinear rotation parameters and a small symmetric coefficient matrix of fixed size. Then, in a second step, the rotation is solved with algorithms that are derived using methods from algebraic geometry such as the Gröbner basis method [CLO07]. Our unified formulation is based on a representation with orthog-onal complements, which allows to combine different types of constraints elegantly in one single framework.

We show that with our unified formulation the existing polynomial solvers of the former approaches can be interchangeably applied to problem instances other than those they were originally proposed for. It becomes possible to compare them on various registrations problems with respect to accuracy, numerical stability, and

computational speed. Our variable elimination procedure not only preserves linear complexity, it is even faster than previous formulations. For the second step we also derive an own algebraic equation solver, which can additionally handle the registration from 3D point-to-point correspondences, where other rotation solvers fail.