5.4.1 Rigid Models for the Object Parts
The first step to create the hierarchical model is to generate a rigid model for each object part. Although already during the training models have been created, the orientation range for which the models have been built does not necessarily coincide with the desired orientation range during the online phase. For example, the user may intro-duce prior knowledge about the possible orientation of the compound object in the search image. This is similar to the model creation for rigid object recognition (cf. Section 4.2.6) and may help to keep the memory require-ment of the model small. However, when dealing with compound objects the question arises how the orientation of a compound object is defined. In this dissertation the orientation of the compound object is equated with the orientation of a user-specified reference object part. For the reference part the user may specify the orientation range for which the associated rigid model should be created. The orientation ranges of the remaining object parts are then automatically defined by the relations between the parts. Because the search order is unknown `a priori, a worst case estimation is used to get the orientation ranges of the remaining object parts.
1m 2m 3m 4m 5m
6m 7m 8m 9m 10m
Figure 5.21: The relations of all pairs of object parts are visualized. Each picture represents an object parti1 for which the relative movements of the remaining partsi2are displayed.
5.4.2 Optimum Search Trees
Based on the relations, an optimum hierarchical search strategy can be found by minimizing the computational search effort in the online phase. One major part of the optimum hierarchical search strategy is represented by the optimum search trees. In this section, the definition and computation of the search trees is explained.
The relation between parti1andi2is used to compute the relative search effortΩri1,i2that must be spent in order to search part i2 relative to parti1. The computation of the search effort depends on the kind of applied rigid object recognition method. If either the MGHT or the SBM is used the search effort is approximately given by:
Ωri1,i2 = 2ari1,i2·2bri1,i2·(ϕr,maxi
1,i2 −ϕr,mini
1,i2 )· nm,topi2
4nli2−1·∆ϕtopi2 . (5.27) The search effortΩri1,i2 is proportional to the area of the rectangle multiplied by the size of the orientation angle interval, both given by the relations. Hence, this product represents the size of the continuous 3D parameter space to be scanned. Because of the quantization of the parameter space and the use of image pyramids, the search effort of part i2 is reduced by a factor that depends on the number of pyramid levelsnli
2 and the quantization of the orientation on the top pyramid level given by the orientation step ∆ϕtopi
2 . Finally, the search effort increases linearly with the number of model edge pointsnm,topi
2 that remain at the top pyramid level. It should be noted that the search effort is not symmetric, i.e., Ωri1,i2 6= Ωri2,i1 (for details see Figures 5.18 and Figure 5.21 of Sections 5.3.4 and 5.3.5, respectively).
Assume that part jserves as root part, i.e., partj is the only part that will be searched within the entire search space during the online phase. Then, the task for finding the optimum hierarchical search tree for the preselected root partjcan be equated with minimizing the overall relative search effort ¯Ωrj:
¯Ωrj = n Xp
i=1,i6=j
Ωrp(i),i−→min , (5.28)
whereΩrp(i),idenotes the effort to search object partirelative to its predecessor partp(i). By definition, the root part does not have a predecessor part, and hence is excluded from the calculation. Informally speaking, this opti-mization problem means to find a predecessor part for each object part such that ¯Ωrj is minimized. Furthermore, it must be ensured that in the online phase the pose ofp(i)has already been determined before searching for parti.
1 2
3 4
5 6
7 8
10
9
1
3
5
1940
9
2855 10801
4152 7756831779
739
3728
4093 57661 6610 11252
1
3
5
9
2855
4152 3728
(a) Directed graph (b) Subgraph of (a) (c) Optimum search tree Figure 5.22: The object parts and the relations between them are represented in a directed graph (a). For illustration purposes a small subgraph is selected showing four nodes and the corresponding weights (b). The minimum spanning arborescence of the graph represents the optimum search tree (c). Here, part 9 was selected as the root part.
1 3
5
9 1
3 5
9 1
3 5
9 1
3 5
9 1
3 5 1 9
3 5
9 1
3 5
9 1
3 5
9
1 3
5
9 1
3 5
9 1
3 5
9 1
3 5
9 1
3 5
9 1
3 5
9 1
3 5
9 1
3 5
9
Figure 5.23: All possible spanning arborescences of the subgraph shown in Figure 5.22(b) with part 9 as root part
Consequently, the search in the online phase can be represented by a tree, where the root node represents the root part, which is searched within the entire search space, and the other nodes represent the parts that are searched relative to their associated predecessor part.
To solve this optimization problem, one may think of the object parts and the relations between them as a complete directed graphG(V, E), whereV denotes the set of nodes with|V| = np and E is the set of arcs with |E| = np(np−1). A complete directed graph is a directed graph where each two nodesi1andi2are connected by the two arcs (i1, i2)and (i2, i1). The nodes in the graph represent the object parts, the arcs represent the relations.
The arc (i1, i2) is weighted by the search effortΩri1,i2 and the arc (i2, i1) is weighted by Ωri2,i1. Figure 5.22(a) shows the corresponding graph of the example.
The optimum search tree can now be obtained by computing the minimum spanning arborescence of G with respect to a certain root nodej. In the relevant literature, the term arborescence is used synonymously with the term tree, however, it implies that the tree has directed arcs. The minimum spanning arborescence in a directed graph is defined as a directed spanning tree H(V, E0), where E0 is a subset ofE such that the sum of Ωri1,i2 for all(i1, i2)inE0 is minimized. The directed spanning tree is defined as a graph that connects all nodes with np −1 arcs, i.e., each node, except the root node, has exactly one incoming arc. To illustrate this definition, in Figure 5.22(b) a detailed view of a small subgraph with four nodes is shown. After selecting part 9 as the root part, the minimum spanning arborescence is calculated. The result is shown in Figure 5.22(c). It is easy to
convince oneself that in the result three arcs are contained, where each non-root node has exactly one incoming arc and that the overall sum of weights associated with the result is 10735. This is the minimum weight among all possible directed spanning arborescences rooted at part 9. For illustrative purposes Figure 5.23 shows all possible spanning arborescences of the respective subgraph. As can be seen, there are 16 different ways to connect the three nodes to the root node. The result of Figure 5.22(c) suggests that after the upper body (part 9) is found in the image, it is most efficient to search the left arm (part 1) and the left leg (part 5) relative to the upper body and to search the left hand (part 3) relative to the left arm.
The minimum spanning arborescence can be seen as the equivalent to the well-known minimum spanning tree in an undirected graph (ifΩri2,i1would be symmetric one would obtain a undirected graph). The two most prominent algorithms to efficiently compute the minimum spanning tree are the Kruskal and the Prim algorithm (Graham and Hell 1985, Clark and Holton 1994). Unfortunately, these algorithms cannot be used or even extended to cope with directed graphs. Solving the problem of finding the minimum spanning arborescence in a directed graph is much more complicated in comparison to solving the equivalent undirected problem. A polynomial algorithm for the minimum spanning arborescence was independently proposed in (Chu and Tseng-Hong 1965), (Edmonds 1967), and (Bock 1971). In (Tarjan 1977) and (Gabow et al. 1986), efficient implementations of the algorithm are presented. The implementation used in this dissertation is presented in (Fischetti and Toth 1993). It makes use of simple data structures leading to a run time complexity of onlyO(n2), wherenis the number of nodes in the graph. For a detailed description of the algorithm or of the implementation the interested reader should refer to the cited literature.
Root Root
Root Root
Root
¯Ωr1= 27·103 ¯Ωr2= 28·103 ¯Ωr3= 25·103 ¯Ωr4= 26·103 ¯Ωr5= 29·103
Root
Root Root
Root
Root
¯Ωr6= 39·103 ¯Ωr7= 29·103 ¯Ωr8= 39·103 ¯Ωr9= 29·103 ¯Ωr10= 30·103 Figure 5.24: The minimum spanning arborescences for each of the ten object parts serving as root part and the associated overall search efforts are shown. Additionally, the relations between two adjacent nodes are superimposed.
It is obvious that for different root partsjdifferent minimum spanning arborescences are obtained with different overall weights ¯Ωrj. Therefore, for each root part the associated minimum spanning arborescence is computed.
Figure 5.24 shows the result for each of the ten object parts serving as root part. It should be noted that although in this example the same two object parts are directly connected in all minimum spanning arborescences it is not necessarily the case in general.
5.4.3 Root Part Ranking
To complete the optimum hierarchical search strategy, the question remains, which part to choose as the root part. One criterion for a suitable root part is a small overall search effort ¯Ωrj of the associated minimum spanning arborescences.
However, ¯Ωrj only describes the search effort that must be spent during the relative search. Therefore, the effort Ωrootj that must be spent to search the root partjitself must be considered as a second criterion:
Ωrootj =R·C·(ϕmaxj −ϕminj )· nm,topj 4nlj−1·∆ϕtopj
, (5.29)
whereRandCare the number of rows and columns in the search image, and hence describe the position search range for the root object part, while(ϕmaxj −ϕminj )describes the orientation angle search range.
As a third criterion the uniqueness of the root part must be considered. The root part should exhibit as few symmetries and as few similarities to other object parts as possible. Assume that the left leg (part 5) serves as root part in the online phase and is searched within the full orientation search range. Because of its rotation symmetry and its similarity to the right leg, it would be found at four different poses (ignoring possible clutter in the search image). When looking at the associated search tree the upper body and the left foot must be searched relative to each of the four poses of the left leg. Consequently, the search effort in the online phase increases with the number of symmetries and similarities. Therefore, the symmetries and similarities of all object parts are determined using the analysis described in Section 5.3.3.2, which matches object partjto itself and to all other object parts. Assume that object partjhas been foundnsymj times on itself andnsimj times on other object parts during the analysis. Then, the search effort of the relative search ¯Ωrj must be multiplied by(nsymj +nsimj ) in order to approximately estimate the influence of the non-uniqueness of the root part on the search effort. One could argue that it is sufficient to only multiply the relative search effort of the parts that are adjacent to the root part since the search can be aborted if the adjacent parts are not found. However, the multiplication of the overall relative search effort is legitimated since the object recognition, which will be described in Section 5.5, should be able to cope with occlusions. Thus, the search cannot be stopped if one object part is missing.
Finally, the search effortΩjthat is associated with the root partjis obtained:
Ωj = Ωrootj + (nsymj +nsimj )¯Ωrj . (5.30) By sorting the possible root parts with respect to Ωj in ascending order one obtains a root part ranking that expresses the suitability of all object parts to serve as the root part. In Table 5.1 the respective ranking of the example is presented. It can be seen that the head (part 10) and the upper body (part 9) are best suited to serve as the root part. This is because they both do not show any rotation symmetry or similarity to other object parts and because five pyramid levels can be used during the search. In contrast, taking one of the two hands (part 3 or part 4) as root part would result in the highest search effort: they both exhibit symmetries and mutual similarities.
Furthermore, only three pyramid levels can be used because of their small size.
After this step, the creation of the hierarchical model is completed. However, a manual selection of the root part by the user is still reasonable. This is because the selection of a suitable root part also depends on the application (cf. Section 5.5.1). Therefore, the root part ranking is returned in order to help the user to select the appropriate root part for his specific application. Consequently, the search trees for all root parts are stored in the hierarchical model. In the online phase, the search tree that is associated with the user-specified root part is selected from the hierarchical model and used to search the object. Summing up, the hierarchical model consists of the rigid models of the object parts, the relations between the parts, and the hierarchical search strategy. The hierarchical search strategy is represented by the optimum search trees, which are given by the minimum spanning arborescences, and the root part ranking.
Rank 1 2 3 4 5 6 7 8 9 10
Root part j 10 9 7 8 5 6 1 2 3 4
Ωj·10−5 4 5 21 21 26 27 32 32 140 140
Table 5.1: The root part ranking expresses the suitability of the object parts to serve as the root part. Parts 10 and 9 are best suited, whereas parts 3 and 4 would result in the highest search effort.
As a last point, a special case should be discussed that, however, is rather rare in practice: assuming a compound object that consists of identical object parts, then no distinct root part can be determined. For example, a chain consists of several identical links. In this case, the presented approach cannot be used because solving the ambiguities during the training would fail. This is because there is no preferable configuration of the links since the overall configuration itself is ambiguous. However, even if a hierarchical model is available its use would not be advisable because the root part would be found as many times as links are contained in the chain. Then, for each found instance of the root part the hierarchical search would be started, which leads to a high computational effort. Fortunately, in this case the use of the hierarchical model is not necessary. In contrast, it is sufficient to search only one single link in the image. The search already returns all instances of links, and thus the matches of all object parts. Hence, no further (relative) searches need to be performed. However, the correspondence problem still needs to be solved. It should be noted that in most cases when dealing with objects that consist of identical parts, it is possible to determine an auxiliary root part. For example, if several identical modules on a circuit board must be recognized, it is desirable to use the hierarchical model, and hence profit from the inherent determination of the correspondence. This can be achieved by including an additional object part in the compound object, e.g., a different module, a corner of the circuit board, a fiducial mark, or any other print on the board. The additional object part can then serve as the root part.