6.6 Evaluation
6.6.2 Curve Ordering
Figure 6.10: Wins/Losses/Ties of the algorithm incorporating Fréchet distance with respect to the IGD+and IGDX indicators over different instance size intervals [WM22a].
(a) IGDX results
Results by size interval (IGDX indicator)
[15,20] [20,24] [24,26] [26,28] [28,30]
Size Interval 0
5 10 15 20 25
Number of Instances
FD-MED FD-MIN Ties
(b) IGD+results
Results by size interval (IGD+ indicator)
[15,20] [20,24] [24,26] [26,28] [28,30]
Size Interval 0
5 10 15 20 25
Number of Instances
FD-MED FD-MIN Ties
Figure 6.11: Indicator values of the instance CH P1 K3 BT for different sizes, comparing FD-MEDand FD-MIN[WM22a].
(a) IGDX results
15 20 25 30
Instance Size
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5
IGDX Value
102 103
# of Pareto-optimal solutions
CH P1 BT K3 FD-MED
FD-MIN
# of POS
(b) IGD+results
15 20 25 30
Instance Size
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5
IGD+ Value
102 103
# of Pareto-optimal solutions
CH P1 BT K3 FD-MED
FD-MIN
# of POS
FD-MINobtained better or similar values in terms of the IGDX values but was outperformed byFD-MEDwith respect to IGD+.
When analysing the results, we noted that the high number of ties indicates room for improvement. The many ties are a result of the path similarity sorting.
Sorting with two subsets of distinct paths that have relatively short distance values inside the subset will result in a short distance value for each path, although the two sets may be apart from each other [WM22a].
node n∈V has various properties, including their respective coordinates, a height value and constraints concerning possible movements when this node is traversed (see Chapter 4). The problem has five objectives, i.e., path length, the time needed to traverse the path, expected delay, total positive ascent of a path, and curvature (smoothness).
To compare our results, we used the proposed algorithm in [WM21b] as our baseline. We used the same implementation but changed the Fréchet sorting method to our proposed ordering and neighbourhood relation identification method. The two contraction metrics are used in dedicated algorithms. The reason for using NSGA-II for a many-objective problem is that in the original study, it outperformed many-objective algorithms such as NSGA-III [WM22b].
We ran the algorithms 31 times with a population size of µ =212 for 500 generations, totalling 106 000 evaluations. The crossover and mutation rates were set to 0.8 and 0.2, respectively. For performance comparison, we used the IGD+indicator [IMN15] and the IGDX indicator [AQY09] to measure the distance in the decision space. The distance from the Pareto-set to the candidate solutions are, again, measured using the Fréchet distance. To compute a reference front for the real-world example, we combined the results from all runs of all algorithms and computed the non-dominated set. The statistical evaluation was performed using the Kruskal-Wallis test and applied Bonferroni correction for multiple comparisons, as suggested in [KTZ06], and statistical significance was set at p<0.01. In pairwise comparisons, we assumed the null hypothesis held, namely that the distributions of the two samples had equal medians [WM21a].
The algorithms were implemented in Java using thejMetalframework [NDV15]
and the jGraphT library [MKNS19].
Results
Figure 6.12 shows the results of the algorithms. Depicted are the IGD+values and their corresponding standard errors for different instance sizes. It is evident that the proposed method performed better for larger sizes. Mainly from the size of 10 and above, the new method outperformed the other in most instances.
This trend indicates that the proposed niching methods should not be used with small instance sizes. The reason for the poor performance on smaller instances is that the contraction points are comparatively close to each other or may even exist at the same coordinates, which makes it hard for the algorithm to find less dense areas; this holds for both variations employing different contraction metrics. However, the geometric median approach performed slightly better.
The approach in [WM21b], by contrast, was better on smaller instances, as it determined crowded areas using a distance matrix, which gives a better estimation of densities [WM21a].
The results also showed that the proposed approach outperformed the origi-nal method when the Pareto-optimal solutions were comparatively dense in decision space. The former approach has various drawbacks in such a situa-tion, since the values in the distance matrix are mostly the same. This makes it hard for the algorithm to determine the solution with the lowest degree of isolation [WM21a].
In Figure 6.13, different size intervals are grouped, and the respective wins of each algorithm are shown. It is evident that with increasing size, the proposed method won most benchmark instances. In the diagram, each interval contains 48 benchmark instances. For example, the interval[5,6]contains instances with
Figure 6.12: Results of the algorithms (IGD+indicator) by size of the instance typeLA P1 K2 BF. [WM21a]
5 6 7 8 9 10 11 12 13 14
Instance Size
0 0.2 0.4 0.6 0.8 1 1.2
IGD+ Value
0 20 40 60 80 100 120 140 160
# of Pareto-optimal solutions
LA P1 BF K2
NSGA-II-CRFD NSGA-II-DEFDCT
NSGA-II-DEFDGM
# of POS
all different obstacle, neighbourhood and elevation configurations, but only those of size 5 and 6 [WM21a].
In Table 6.3, the wins, losses and ties concerning the two used indicators are shown. The table numbers indicate the respective number of benchmark in-stances. With respect to IGD+, our proposed methods won most of the instances.
Moreover, using the geometric median as the contraction metric can yield bene-fits over using the centroid approach. The reason is the characteristics of the geometric median, which by nature is more robust to outliers (cf.mean and median). If a path is mutated so that a few nodes are shifted, the centroid’s position is impacted more than the geometric median’s coordinates. In this approach, the geometric median can still provide a good estimate of the path’s relations to its neighbours, compared to the centroid. In other words, using the centroid approach, a neighbourhood relation can change more quickly if only a few nodes change their positions [WM21a].
However, there are still several losses and also ties, indicating room for improve-ment. Figure 6.12 illustrates a direct comparison of the obtained IGD+indicator values concerning the instances’ sizes. Our proposed methods achieved better results with increasing map size [WM21a].
A different outcome was obtained by analysing the results using the IGDX indicator, which measures the distance to the true Pareto-set in the decision space. When the method from [WM21b] was compared to our approach, our methodology was outperformed more often than it won an instance. Further-more, there were many ties between the methods. The reason is that the older method uses a dissimilarity matrix to determine crowded areas and is therefore more sensitive to differences in paths. The path comparison in [WM21b] uses the complete path information, whereas our technique uses a single point of representation. We also compared the centroid-based methodology to the geo-metric median-based approach. They won similar cases against each other, and the comparison also resulted in numerous ties [WM21a].
The results of the real-world example indicated that the proposed method was superior to the original approach. This conclusion is supported by the IGD+ indicator, shown in Figure 6.14, depicting the indicator value over the number of evaluations. Here, a faster convergence can be observed. Finally, the proposed approach outperformed the reference methodology regardless of the chosen
Figure 6.13: Wins and Ties of the proposed approach by different instance size ranges (using the centroid method) concerning the IGD+ indicator. [WM21a]
[5,6] [6,7]
[7,8] [8,9]
[9,10] [10,11]
[11,12] [12,13]
[13,14] 0
5 10 15 20 25 30 35 40 45 50
Size Interval
NumberofInstances
Results by size interval
Method [WM21b] wins Centroid wins Ties
Figure 6.14: IGD+indicator over function evaluation for the real-world problem. The run, with the median value at the end
of the experiments, is depicted. [WM21a]
2 4 6 8 10
Evaluation Numbers 104 0
0.5 1 1.5 2
Indicator Value
IGD+
NSGA-II-CRFD NSGA-II-DEFDCT NSGA-II-DEFDGM
contraction metric. The two metrics had no statistically significant difference regarding the values of the IGD+indicator [WM21a].
A comparison of the computational costs indicated the benefits of using our proposed method. In [WM21b] a distance matrix was used, resulting inµ−1 distance computations for each solution, whereµ is the population size. In our approach, after ordering the set and determining neighbourhood relations, we needed to compute only two distances for each solution. The algorithms’ run time was lower than in the baseline approach. However, the computation of the contraction points must also be considered. Computing the centroid of a set of points needs less computational effort, since the points’ coordinates are averaged. There is no analytic solution for the geometric median, because this is a computationally challenging task and an iterative algorithmic approach has to be used. There are newer approaches to solve the problem of finding the geometric median which achieve lower computational complexity than Weiszfeld’s algorithm, i.e., [CLM+16], which achieved a nearly linear time complexity ofO ndlog3nε
for a (1−ε)-approximate geometric median. In contrast to our study,n denotes the number of points and d the number of dimensions.
Table 6.3: Wins, losses and ties of each algorithm pair (rows vs.
column) with statistical significance atp<0.01, Bonferroni correction applied, IGDX and IGD+ indicator [WM21a].
Centroid Geom. Med.
[WM21b] IGDX 66/54/120 76/67/97 IGD+ 51/93/96 47/109/84
Centroid IGDX 54/56/130
IGD+ 20/50/170