6. Experimental Evaluation 81
6.4. Localization-based Error Detection
6.4.3. Estimation Improvement
6.4. Localization-based Error Detection
Conclusions from the evaluation Our automatic initialization method saves the time and effort for training. It results in an location estimation inaccuracy which is about 20% higher than the respective inaccuracy with the manual method. In the next two experiments we will consider what is the effect of this inaccuracy on the estimation improvement, model calibration and error detection.
In this experiment, we made another important observation. When a mobile evaluation profile is used and the antenna of the mobile station is directional, the location estimation accuracy has a significantly higher inaccuracy as compared to the case of static evaluation profile. This is important since the purpose of localization in the context of this thesis is to collect information from mobile stations during their normal operation. In the general case, this means that they will move and will have a situation similar to the mobile evaluation profile. For this reason, we will take a closer look at the mobile evaluation profile in the next two experiments. First, we will evaluate the ability of Kalman smoothing to improve the location estimation of the mobile profiles. Secondly, we will evaluate the ability of our localization method to give new information about the environment and the ability of our localization-based error detection method to detect the environmental dynamics.
6. Experimental Evaluation
Scenario 1 (University of
Magdeburg)
Scenario 2 (Galileo logistics
lab) Initialization method Manual training Automatic
Antenna profile Omnidirectional Directional Movement profile Constant speed,
almost no curves
Constant speed, few curves
AssumedSpeedmax,[m/s] 1 1.5
AssumedLocationInnaccuracystd,[m] 4 6
Process noisevar(wx), var(wy),[m/s]2 0.33 0.75 Measurement noisevar(zx), var(zy),[m2] 22.63 50.91
Table 6.8.: Evaluation scenarios and parameters for the estimation improvement (Speedmax = 1m/s) along the corridors of the building which had only one curve.
During the movement we recorded the position of the mobile station every T = 1sec for evaluation purposes.
In scenario 2, we have used the same experimental setup as for the evaluation of the localization initialization in the previous section: automatic training data, directional antenna profile, slightly higher speed (Speedmax = 1,5m/s) and several curves. Because of the directional antenna and the automatic training, this scenario is a worst case.
The Kalman filter noise parameter values are significant for the estimation improvement. Therefore, we determined these values in a way, based on information which will be available in a real application scenario in automation: the maximum speed and the standard deviation of the location estimation inaccuracy. Then we used equations 4.34 and 4.39 for this purpose. The used parameter values are shown in table 6.8.
Evaluation results Figure 6.11 shows the results of the experiments in scenario 1. It shows the movement profile of the mobile station, the location estimation, the estimation improvement with filtering and with smoothing and a table with statistics. The statistics clearly show that the estimation improvement with smoothing significantly decreases the localization inaccuracy. The mean is decreased by 36%, the standard deviation by 59%
and the maximum is decreased by 79%. Our estimation improvement approach with smoothing is also better than the traditional approaches that use filtering. The traditional approach also significantly decreases the inaccuracy (20%, 28% and 70% for the mean, standard deviation and the maximum respectively). However, the smoothing approach decreases the inaccuracy more than the filtering approach. This can also be observed on the plotted movement profiles on figure 6.11. The estimation improvement with smoothing fits the real movement profile better than the estimation improvement with filtering.
6.4. Localization-based Error Detection
A) Locations of the base stations
B) Movement profile for the evaluation
Figure 6.10.: Evaluation scenario “University of Magdeburg” for estimation improvement. The source of the images is [121].
40 45 50 55 60 65 70 75 80
0 5 10 15 20 25
X coordinate [m]
Y coordinate [m]
True locations Location estimation Estimation improvement (filtering)
40 45 50 55 60 65 70 75 80
0 5 10 15 20 25
X coordinate [m]
Y coordinate [m]
True locations Location estimation
Estimation improvement (smoothing)
Localization inaccuracy [meter]
Mean Standard deviation Minimum Maximum
Location estimation 3.9 3.2 0.2 29.6
Estimation improvement (Kalman filter)
3.1 2.3 0.05 8.5
Estimation improvement (Kalman smoother)
2.5 1.3 0.03 6.2
Figure 6.11.: Estimation improvement results for scenario 1
6. Experimental Evaluation
0 5 10 15 20 25 30 35 40
0 5 10 15 20 25
X coordinate [m]
Y coordinate [m]
True locations Location estimation Estimation improvement (filtering)
0 5 10 15 20 25 30 35 40
0 5 10 15 20 25
X coordinate [m]
Y coordinate [m]
True locations Location estimation
Estimation improvement (smoothing)
Localization inaccuracy [meter]
Mean Standard deviation Minimum Maximum
Location estimation 10.9 6.2 2.8 40.2
Estimation improvement (Kalman filter)
9.9 2.4 6.2 16.9
Estimation improvement (Kalman smoother)
8.3 2.7 5.3 16.9
Figure 6.12.: Estimation improvement results for scenario 2
Figure 6.12 shows the results for scenario 2 in a way that is analogous to the results for scenario 1. A similar trend is observed in these results. The estimation improvement with smoothing decreases the inaccuracy of the location estimation. It decreases it more than the traditional approaches with filtering. Similar to scenario 1, in this scenario the most significant decrease in inaccuracy is in the standard deviation (56%) and in the maximum value (58%). The decrease in the mean inaccuracy is also significant (by 24%).
The plots of the movement profiles show the effects of different conditions on localization accuracy. In “good” conditions the movement profile is almost perfectly tracked (figure 6.11). In “worse” conditions, the smoothing has a significant improvement over the location estimation, but the movement profile is less accurately tracked.
Conclusions from the evaluation Both evaluation scenarios have shown that our approach for estimation improvement with Kalman smoothing significantly decreases the inaccuracy of the location estimation. Our approach is better than traditional improvement approaches which use Kalman filtering.
In addition, the experiments confirm our method for setting the Kalman filter noise parameters. The results show that with simple assumptions about the system (the stations speed and the location estimation accuracy) and with an adequate way of determining the parameters, one can get a significant improvement.
6.4. Localization-based Error Detection