F P
1
F2 2
P1
Figure 1.9: The Pareto-optimal values (right) and the corresponding parameters (left) of the vector-valued function (F1(P1, P2), F2(P1, P2)) are shown in red.
Multi-objective optimization algorithms are modeled with the help of two distinct modules: the analyst and the decision maker. The analyst, typically an algorithm, is responsible for the mathematical part of the optimization, and should find the set of Pareto equivalent optimal solutions. The decision maker, an algorithm or a human, is responsible for choosing one solution out of the Pareto optimal set. Multi-objective optimization requires cooperation between the two entities.
In ideal conditions (no noise, non-degenerate signals, well behaved objective functions), the Pareto set of global optimal solutions of the registration problem should contain a unique solution. In reality, due to a multitude of facts, such as digitization errors (geometry and color) or simplification in the camera model, the objective functions are not optimized simultaneous, leading to a set of Pareto optimal solutions.
1.4 Previous Work
In this section we review the existing work regarding global registration of signals and texture registration, and briefly mention the shortcomings of the texture registration methods in use.
1.4 Previous Work
1.4.1 Global Registration
Although not explicitly stated in literature, global registration algorithms dif-ferentiate themselves from usual registration algorithms by using optimization techniques specific to multi-objective optimization. The global registration algo-rithms, whether applied to scan registration, image registration, or other fields, recurse to a graph representation of the problem, as shown in Figure 1.8.
A global algorithm for registration of 3D scans is presented in [33]. The algo-rithm performs pairwise registration of all pairs, rejects some pair-matches which have large error, and then searches a consistent graph, using a mixed combinato-rial and continuous optimization strategy. A closed-form solution for spreading the error optimally over a cycle in the graph is developed in [64] for scan regis-tration. The technique is extended to a general graph by an iterative heuristic.
A closed-form initialization and optimization procedure for global scan registra-tion, based on Newton’s method and exploiting the Lie group structure of the 3D rotations, is shown in [37]. A global algorithm for automatically reassembling fractured objects is presented in [32]. Each piece of the object is scanned, and the object is reconstructed by searching a consistent graph consisting of all pieces.
The article brings several contributions regarding robust pairwise optimization using feature clusters, and graph optimization for multi-piece global matching.
Global image registration of multi-modal retinal images is modeled as a graph problem in [10]. Initially, pairwise registration is run for all pairs (if there is an overlap) and the registration error is saved. The optimal image to be used as a reference frame, together with the spanning tree which minimizes the global error, is found by solving the all-pairs shortest path problem.
1.4.2 Texture Registration
The most common registration criterion is based on point correspondences, usu-ally chosen interactively, as in [59]. Point correspondences may be used together with other features [52], or in a pre-registration step, to initialize another al-gorithm [35]. The other criteria used in [52] are outlines and image attributes (color), and are optimized independently and jointly, in a rather complex proce-dure.
1.4 Previous Work
Another feature used in registration is thesilhouette. In [40], the silhouette is used both for initialization, by sparsely sampling the parameter space and choose the parameters with smallest error, and for the actual optimization, based on the downhill simplex method, slightly modified towards a simulated annealing behavior. The color information from overlapping images is used in a global optimization step, for final tuning. Silhouettes were also used in 3D reconstruction from images [31].
Provided with a rough initialization, featureless (intensity-based) registration criteria are used in [11; 35; 42; 53; 78; 81]. Some of them [11; 42; 78], although defined for image-to-surface registration, were not applied to acquisition of high quality 3D models, but were very successful in medical applications.
Mutual information between the surface normals and the intensity image was proposed in [78] for image-surface alignment, and extended for a calibrated multi-camera system in [42]. Its drawback is that it does not consider the surface characteristics (BRDF). The solution proposed in [53] is to use the mutual in-formation between the image and the surface reflectance obtained from the 3D scanner, already aligned to the surface. Reflectance images were also used in [81], with the chi-square statistics as the registration criterion, and in [38], with a feature-based method (edge information).
The photo-consistency criterion is based on the fact that, if the camera pro-jections for two cameras are known, the images of a 3D point visible in both views are similar. In ideal conditions, if the surface is Lambertian, the color of the projected points is the same. In [11], this criterion is used to define an optimization problem for 2D-3D registration for a system of calibrated cameras (known relative poses of the cameras). In [35], photo-consistency is used with a pair of uncalibrated cameras.
A rather unusual texture registration method for outdoor scenes, relying on shadows, is proposed in [73]. The time information for each photo allows estimat-ing the position of the sun and, correspondestimat-ingly, the shadows can be modeled, and then matched with the shadows present in the image.
The registration methods presented above used the pinhole camera model for 2D-3D parametrization. The unknowns usually consist in the six extrinsic parameters of the camera, and, perhaps, the focal length. In [11;42], the problem
1.4 Previous Work
is simplified by using a calibrated camera system, and only the pose of the object is unknown (six parameters), while [35] considers the full pinhole camera model (eleven parameters). A whole range of optimization methods have been used, such as: Levenberg-Marquardt [52], downhill simplex [40], simulated annealing [73], genetic algorithms [35], stochastic gradient descent [78], and Powell’s method [81].
Following the registration phase, the images are sampled on a texture map.
Smooth transition between images and elimination of outliers (such as specular highlights) is achieved in [52] with a weighting approach. In [59], several images are recorded keeping the camera fixed, with different illumination, allowing de-tection of shadows and specular highlights; the diffuse reflectance coefficients of the surface are found as the result of a linear problem. Misregistration artifacts are solved by a local registration step for boundary triangles. The seams are eliminated with a smooth multi-band weighting scheme in [4]. The multi-band blending technique was introduced by Burt and Adelson, already in the eighties, for image registration [9]. The images containing lower resolution bands have larger transition regions, to eliminate the seams, and those containing high reso-lution bands have sharper transition regions, to preserve the sharp features. The color discrepancies of the overlapping images were reduced, in [3], by a set of linear transformations in color space. For image stitching, cost functions defined in the gradient domain were proposed in [43], and extended to texturing mesh models in [39], with a cost function optimized using a Markov Random Fields formulation. Other issues that can be considered at this stage refer to texture interpolation in regions with no texture [52].
1.4.3 Shortcomings
The techniques applied so far to texture registration are based on the features [40; 52; 59], which implies a feature extraction algorithm, as well as existence of features (i.e., the whole silhouette to be visible in any image). When point-correspondences were interactively chosen, we experienced many cases when the number of features visible in the color image and on the model were insufficient.