4.4 Performance Evaluation of the MGHT and the SBM
4.4.3 Accuracy
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0
20 40 60 80 100
smin
Recognition rate [%] MGHTSBM
NCC HD
GMF PM PQ
10 15 20 25 30
0 20 40 60 80 100
emax
Recognition rate [%]
SAD
Figure 4.29: The recognition rate indicates the robustness against arbitrary changes in brightness. The recognition rate depends on the chosen value forsminandemax, respectively.
a maximum error of 20, which is also 11%, it is obvious that even by further increasing the maximum allowable error no meaningful improvement can be reached. In contrast, the recognition rate of the NCC is substantially better. Obviously, the robustness of the NCC against changes in brightness is higher than against occlusions.
This can be attributed to its normalization, which compensates at least global changes in brightness. The result obtained by the HD is superior to that of the NCC. Especially, in the case of large values for the minimum score it shows good results. However, for lower values it cannot reach the performance of the remaining approaches.
If the minimum score is set low enough, the recognition rate of the MGHT even surpasses that of the SBM, PQ, and PM, reaching a result comparable to the GMF. For higher values its recognition rate decreases rapidly. PM and PQ show approximately equivalent results, both of which are inferior to the SBM for almost all values of smin. Also here, the GMF achieves a very high and approximately constant recognition rate even for large values ofsmin.
In the case of the MGHT and the GMF the recognition rate additionally depends on the chosen threshold for the edge extraction in the search image. As in the case of occlusions the recognition rate of the SBM addition-ally is influenced by the greediness parameter. Therefore, Figure 4.30 shows the recognition rates of the three approaches for different parameter settings.
The MGHT (see Figure 4.30(a)) allows to specify the minimum edge magnitude γmin. The recognition rate of the MGHT strongly depends on the chosen threshold for edge extraction in the search image. As expected, higher recognition rates are obtained for lower values of the minimum edge magnitude, because fewer edge pixels fall below the threshold γmin. The higher the minimum edge magnitude, the more edge pixels are missed, because dimming the light as well as stronger ambient illumination reduces the contrast. Thus, this effect is comparable to the effect of higher occlusion. Therefore, a high recognition rate can be obtained by setting the minimum score to a lower value or by choosing a lower threshold for the edge magnitude. For example, a minimum score of 0.5 and an edge threshold of 10 leads to a recognition rate of 84%. Nevertheless, the true invariance of the SBM against changes in brightness could not be reached by the MGHT. In the case of the GMF (see Figure 4.30(b)) the influence of the edge extraction is less distinct. Nevertheless, if the medium detail level is chosen, the recognition rate decreases by more than 15%. The recognition rate of the SBM (see Figure 4.30(c)) in the case of changing brightness is less sensitive to the chosen greediness parameter g than in the case of occlusions. Only, when setting gto 1 and choosing high values forsmin a significant deterioration in the recognition rate is observable.
Disregarding the result obtained with greediness set to 1, the discrepancy is smaller than 10%.
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0
20 40 60 80 100
Recognition rate [%]
smin
γmin = 5 γmin = 10 γmin = 15 γmin = 20 γmin = 25 γmin = 30 γmin = 5 γmin = 10 γmin = 15 γmin = 20 γmin = 25 γmin = 30
(a) MGHT
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0
20 40 60 80 100
Recognition rate [%]
smin detail level = medium detail level = high detail level = very high
(b) GMF
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0
20 40 60 80 100
Recognition rate [%]
smin g = 0.0
g = 0.2 g = 0.4 g = 0.6 g = 0.8 g = 1.0
(c) SBM
Figure 4.30: The recognition rate in case of changes in brightness additionally depends on the threshold for edge extraction in the search image. The threshold can be set byγmin for the MGHT (a) and by the detail level for the GMF (b). For the SBM (c) the greedinessginfluences the recognition rate.
remaining candidates. Additionally, the least-squares pose refinement (LSPR), which has been presented in Section 4.3.3, is taken into account in order to evaluate the gain in accuracy that is achieved by this method.
Therefore, in the following the results denoted by SBM refer to the pose refinement that is obtained by the polynomial fitting, whereas LSPR implies the additional improvement of the pose parameters that is obtained by the least-squares adjustment in the SBM.
Experimental Set-Up. To evaluate the accuracy, the IC was mounted onto a table that can be shifted with an accuracy of 1µm and can be rotated with an accuracy of 0.7’ (0.011667◦). Figure 4.31 illustrates the experimental set-up. Three image sequences were acquired: In the first sequence, the IC was shifted in 10µm increments to the left in the horizontal direction, which resulted in shifts of about 1/7 pixel in the image. A total of 40 shifts were performed, while 10 images were taken for each position of the object. The IC was not occluded in this experiment and the illumination was approximately constant. In the second sequence, the IC was shifted in the vertical direction with upward movement in the same way. In this case, a total of 50 shifts were performed. The intention of the third sequence was to test the accuracy of the returned object orientation. For this purpose, the IC was rotated 50 times for a total of 5.83◦. Again, 10 images were taken at each orientation.
The search angle for the object orientation was restricted to the range of[−30◦; +30◦]for all approaches, whereas the object position again was not restricted. Since no occlusions were present smincould be uniformly set to 0.8 for all approaches. For the SAD emax was specified to be 25. The greediness parameter of the SBM was set to 0.5, which represents a good compromise between recognition rate and computation time. For the GMF, the accuracy level was varied from medium to high.
∆ =10 m µ
Figure 4.31: To test the accuracy, the IC was mounted on a precisexyϕ-stage, which can be shifted with an accuracy of 1µm and can be rotated with an accuracy of 0.7’ (0.011667◦).
Results. To assess the accuracy of the extracted object position, a straight line was fitted to the mean extracted coordinates of position. This is legitimated by the linear variation of the position and orientation of the IC in the world, which can be assumed when using the precise xyϕ-stage. Because the IC is shifted in world units (µm) while the recognition approaches return the position of the IC in pixel coordinates, the exact position of the IC is only known in world units but not in pixel coordinates. Because this scaling is unknown, the slope of the straight line cannot be set to 1 but must be estimated during the line fit. In contrast, to assess the accuracy of the extracted object orientation the straight line does not need to be estimated but can be computed directly. This is because the unit in which the IC is rotated on the stage and the unit in which the recognition approaches return the object orientation are identical, and hence no scaling needs to be considered. The residual errors, i.e., the differences of the extracted position and orientation to the straight lines, shown in the Figures 4.32 and 4.33, are a well suited indication of the achievable accuracies.
−5
−4
−3
−2
−1 0
−0.2
−0.1 0 0.1 0.2
Horizontal translation [pixel]
Error in x [pixel]
MGHT SBM LSPR SAD
−5
−4
−3
−2
−1 0
−0.2
−0.1 0 0.1 0.2
Horizontal translation [pixel]
Error in x [pixel]
NCC GMF−medium GMF−high PM PQ
Figure 4.32: Position accuracy plotted as the difference between the actualx coordinate of the IC and thexcoordinate returned by the recognition approach while shifting the IC successively by 1/7 pixel to the left
As can be seen from Figure 4.32, the position accuracy of the MGHT, the SBM, the LSPR, the NCC, the GMF (both accuracy levels), and PM are very similar. The corresponding errors are in most cases smaller than 1/20 pixel. The conspicuous peaks in both error plots of Figure 4.32 occur for all these approaches with similar magnitude. Therefore, and because of the nearly identical lines, it is probable that the IC was not shifted accurately enough, and hence the error must be attributed to a deficient acquisition. Nevertheless, it can be con-cluded that the error in position in most cases must at least be smaller than 1/20 pixel, which is sufficient for most applications. However, the high and oscillating error of about 1/10 pixel when using the SAD cannot be attributed to this deficient acquisition. This error occurs because of subpixel translations that influence the gray values especially in high contrast areas of the image. Finally, PQ shows the highest errors in thexcoordinate of
0 1 2 3 4 5
−0.2
−0.15
−0.1
−0.05 0 0.05 0.1 0.15 0.2
Rotation [deg]
Error in orientation [deg]
MGHT SAD
0 1 2 3 4 5
−0.2
−0.15
−0.1
−0.05 0 0.05 0.1 0.15 0.2
Rotation [deg]
Error in orientation [deg]
NCC PQ
0 1 2 3 4 5
−0.06
−0.04
−0.02 0 0.02 0.04 0.06
Rotation [deg]
Error in orientation [deg]
SBM LSPR PM
0 1 2 3 4 5
−0.06
−0.04
−0.02 0 0.02 0.04 0.06
Rotation [deg]
Error in orientation [deg]
GMF−medium GMF−high
Figure 4.33: Orientation accuracy plotted as the difference between the actual object orientation of the IC and the returned angle by the recognition approach while rotating the IC successively by approximately 1/9◦counterclockwise. The different scalings of the plots should be noted.
all approaches in the test, reaching a maximum error of 1/5 pixel. Since the errors iny approximately have the same magnitude as inxthey are not presented.
Figure 4.33 shows the corresponding errors of the returned object orientation. Here, the SBM complemented by the LSPR, the GMF, and PM are superior to all other candidates. They reach maximum errors between 1/50◦and 1/100◦ in this example. Furthermore, the improvement of both the LSPR in comparison to the SBM and the high accuracy level of the GMF in comparison to the medium accuracy level becomes visible. The error of the SBM is reduced to 42% when using the LSPR. Similarly, the high accuracy level of the GMF results in orientation errors that are about 84% in magnitude of the errors when using the medium accuracy level. The remaining approaches return a less accurate object orientation. The corresponding maximum errors of the MGHT, the SAD, the NCC, and PQ are about 1/6◦ (10’) in this example. However, it should be remarked that the accuracy of the MGHT can be easily improved by the LSPR in a similar way as the SBM because the same features (edge position and orientation) are used in both methods. Thus, a similar accuracy level can be expected when applying the LSPR to the MGHT.