4.4 Performance Evaluation of the MGHT and the SBM
4.4.4 Computation Time
0 1 2 3 4 5
−0.2
−0.15
−0.1
−0.05 0 0.05 0.1 0.15 0.2
Rotation [deg]
Error in orientation [deg]
MGHT SAD
0 1 2 3 4 5
−0.2
−0.15
−0.1
−0.05 0 0.05 0.1 0.15 0.2
Rotation [deg]
Error in orientation [deg]
NCC PQ
0 1 2 3 4 5
−0.06
−0.04
−0.02 0 0.02 0.04 0.06
Rotation [deg]
Error in orientation [deg]
SBM LSPR PM
0 1 2 3 4 5
−0.06
−0.04
−0.02 0 0.02 0.04 0.06
Rotation [deg]
Error in orientation [deg]
GMF−medium GMF−high
Figure 4.33: Orientation accuracy plotted as the difference between the actual object orientation of the IC and the returned angle by the recognition approach while rotating the IC successively by approximately 1/9◦counterclockwise. The different scalings of the plots should be noted.
all approaches in the test, reaching a maximum error of 1/5 pixel. Since the errors iny approximately have the same magnitude as inxthey are not presented.
Figure 4.33 shows the corresponding errors of the returned object orientation. Here, the SBM complemented by the LSPR, the GMF, and PM are superior to all other candidates. They reach maximum errors between 1/50◦and 1/100◦ in this example. Furthermore, the improvement of both the LSPR in comparison to the SBM and the high accuracy level of the GMF in comparison to the medium accuracy level becomes visible. The error of the SBM is reduced to 42% when using the LSPR. Similarly, the high accuracy level of the GMF results in orientation errors that are about 84% in magnitude of the errors when using the medium accuracy level. The remaining approaches return a less accurate object orientation. The corresponding maximum errors of the MGHT, the SAD, the NCC, and PQ are about 1/6◦ (10’) in this example. However, it should be remarked that the accuracy of the MGHT can be easily improved by the LSPR in a similar way as the SBM because the same features (edge position and orientation) are used in both methods. Thus, a similar accuracy level can be expected when applying the LSPR to the MGHT.
Experimental Set-Up. In order to test the third criterion, the configurations that were used for testing the robustness against occlusions (cf. Section 4.4.2) and for testing the accuracy (cf. Section 4.4.3) are used. The computation time of the recognition processes was measured on a 400 MHz Pentium II for each image of the sequences and for each recognition method. Therefore, one has to keep in mind that the measured recognition times would reduce to a fraction on current computers (e.g., a 2.8 GHz Pentium 4). However, it can be assumed that the relation of the recognition times between different approaches approximately remains unchanged.
The computation time of the SBM was measured for two greediness values of 0 and 1 with using the sequence for testing the robustness in order to be able to estimate the increase in computational cost for a gain in robustness.
Furthermore, in order to assess the correlation between the size of parameter space and computation time, the two sequences for testing the accuracy (horizontal shift and rotation) are used a second time without restricting the angle interval to[−30◦,30◦], but searching the object in the full range of orientations ([0◦,360◦[).
In this context it should be noted that the MGHT and the GMF are the only candidates whose implementations are able to recognize the object even if it partially lies outside the search image. The other approaches automatically restrict the range of possible object positions to those at which the object completely lies within the search image. Therefore, particularly in the case of large objects both methods are disadvantaged when comparing their computation times to those of the remaining approaches. This should be kept in mind when analyzing the results.
Results. At first, the computation time of the respective approaches during the robustness test is analyzed. In Figure 4.34 the mean computation timeT over the 500 images of the sequence is plotted for three different values ofemax (SAD) andsmin (others), respectively. This facilitates an easy assignment of the computation times to the recognition rates displayed in Figure 4.25. Because for real-time applications the maximum time to find the object is often of interest, additionally, the maximum computation time is plotted. One can see that the approaches can be divided into two groups, where the approaches within the same group exhibit similar recognition times. In the first group (MGHT, SBM, NCC, PM, PQ) the mean recognition time varies in the range from 18 to 59 ms. In general, the lowersmin is chosen, the higher the robustness against occlusions (see Figure 4.25), but the higher the computation time. It also becomes evident that PQ is faster than PM. Furthermore, a greediness value of 1 speeds up the SBM in comparison to a greediness value of 0, especially ifsmin is chosen small, i.e., a high number of match candidates must be tracked through the pyramid. In the second group (SAD, GMF) the mean recognition time is much higher. Even ifemax of the SAD is set to the relatively small value of 30, where the associated recognition rate is only 34%, the maximum computation time already exceeds one second. If higher recognition rates should be obtained (at the expense of higher false alarm rates) the computation time may even reach several seconds. The high recognition rates of the GMF have already been depreciated by the associated high false alarm rates. Additionally, its computation time reaches several seconds. Hence, the performance of the GMF suffers from a further depreciation. However, in contrast to all other approaches the computation time increases if smin is set to a higher value. During evaluation it became obvious that the GMF is slowed down considerably if the object cannot be found or if the scoresis nearsmin. From this, it can be deduced that the implementation of the SAD and the GMF are not suited for real-time object recognition.
Figure 4.35 shows the respective mean and maximum computation times when applying the accuracy test. Again, the MGHT, the SBM, PM, and PQ show fast computations. Furthermore, the mean computation time of the SBM extended by the LSPR only marginally increases in comparison to the SBM. The computation time of the LSPR, which is represented as the difference between the computation time of “SBM+LSPR” and “SBM” in the plots, does not depend on the size of the parameter space: it is constant for a given object, and is 8 ms in the case of the translation sequence and 10 ms using the rotation sequence. Therefore, the larger the parameter space the smaller the influence of this constant part becomes. In contrast, again the SAD and the GMF are substantially slower.
However, because the object does not show partial occlusions in the two sequences, the GMF is not slowed down to a high degree as in the occlusion case. This is becausesmin was set to 0.8 and the score hardly differs from 1.
Becauseemaxwas set to 25, the robustness of the SAD would be very poor. Thus, also the SAD would show much longer recognition times if higher robustness was desired. When looking at the result of the NCC an increase of computation time is noticeable when searching the rotated object. During evaluation it became evident that the more the IC is rotated relative to the reference orientation in the model image the longer the computation time
MGHT SBM SBM NCC PM PQ 0
10 20 30 40 50 60 70 80 90
(g=0) (g=1)
T [ms]
smin=20 smin=50 smin=80 Maximum
SAD GMF
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
T [ms]
emax=50 / smin=20 emax=30 / smin=50 emax=20 / smin=80 Maximum
Figure 4.34: Mean and maximum recognition times of the respective approaches when applying the sequence for testing the robustness against occlusions. The computation time of each approach is represented by three bars, each associated with a certain value foremax(SAD) andsmin(others), respectively. The different scalings of the two plots should be noted.
MGHT SBM SBM+ SAD NCC GMF GMF PM PQ 0
100 200 300 400 500 600
LSPR medium high
T [ms]
∆T= 56% ∆T=121%
∆T=105%
∆T=143%
∆T=113%
∆T= 4% ∆T= 3%
∆T= 60%
∆T=100%
[−30°;+30°]
[0°;360°[
Maximum
(a) Sequence of horizontally shifted IC
MGHT SBM SBM+ SAD NCC GMF GMF PM PQ 0
100 200 300 400 500 600
LSPR medium high
T [ms]
∆T= 29% ∆T=100% ∆T= 83%
∆T= 97%
∆T= 27%
∆T= 4% ∆T= 4%
∆T=141%
∆T=200%
[−30°;+30°]
[0°;360°[
Maximum
(b) Sequence of rotated IC
Figure 4.35: Mean and maximum recognition times of the respective approaches when applying the sequence of the shifted IC (a) and that of the rotated IC (b). The computation time of each approach is represented by two bars, where the left bar and the right bar correspond to the restricted and the unrestricted orientation search range, respectively. The associated time increase∆T is printed in percent.
of the NCC became. Obviously, the implementation of (Matrox 2001) does not scan the whole orientation range at the highest pyramid level before the matches are tracked through the pyramid, but starts with a narrow angle range close to the reference orientation. Hence, the computation time of the NCC is not directly comparable to the other approaches, because the orientation range of[−30◦; +30◦]and[0◦;360◦[is not really scanned. Hence, a comparable computation time would be still higher.
From the time increase ∆T when extending the angle search range from[−30◦; +30◦]to[0◦;360◦[conclusions can be drawn about the ability of the approaches to deal with more general transformation classes. The percent-age increase of the mean computation time is printed in Figure 4.35. As can be seen from Figure 4.35(a) the computation time of the GMF merely increases by 4% and 3%, respectively. Obviously, the GMF ignores the restriction of the orientation search range and always recognizes the object within the full range of orientations.
This makes the use of prior knowledge about the object orientation more difficult. With this discovery the com-putation time of the GMF that is plotted in Figure 4.34 is overestimated because the GMF cannot profit from the parameter space restriction as the remaining approaches. Nevertheless, the computation is much too slow for real-time applications. Because of the implementation characteristics of the GMF and the NCC, only the MGHT, the SBM, the LSPR, the SAD, PM, and PQ can be compared objectively when using∆T as criterion.
Here, the MGHT shows the smallest time increase (∆T = 56%), which indicates an advantage of the MGHT over the remaining approaches if the parameter space increases. The time increase of the SBM extended by the LSPR is lower than that of the SBM because of the constant part of the LSPR. Also, the computation time of PM only increases moderately. For most methods, a similar behavior is obtained when searching for the rotated IC (see Figure 4.35(b)). Also here, the MGHT seems to be the method that is most suited when dealing with large parameter spaces. The corresponding time increase is only 29% in this case. The computation times of PM and PQ and the associated values for ∆T are significantly higher than in the case of the shifted IC. The reason for the totally different computation times when using the two sequences is the automatic computation of the coarse grain limits (cf. Section 4.4.1.5). During the first sequence that uses the shifted IC the grain limit of both methods was automatically set to 3.72, while during the second sequence that uses the rotated IC the grain limit was automatically set to only 2.92. This results in an increased complexity. There is no obvious reason for this difference, because the object was the same in both cases. Experiments have shown that the automatic computation of the grain limit may result in a completely different value if the ROI of the model image is shifted by just one pixel without changing the number of edge points within the region.