Results and Analysis: Large Budget

In this section the results of the large-budget experiments are analysed. These experiments used 10,000,000 function evaluations and are conducted with SMPSO, WOF-Randomised, LMEA, MOEA/DVA and S³-CMA-ES. The respective settings of the algorithms are the same as in Section 6.1.1. The results are summarised in Tables 6.12 and 6.13 and details are found in Appendices B and C. As described above, the amount of used benchmarks varies in these experiments due to the large computation time when applying all algorithms to a large set of different benchmarks. Therefore, only large-scale instances were used in these experiments, and further details regarding the exclusion of other benchmarks for algorithmic reasons in case of S³-CMA-ES are given below.

Comparison with MOEA/DVA and LMEA

At first, we pay attention to the results in Table 6.12, where we see the winning rates of WOF-SMPSO and WOF-Randomised (the two best performing WOF versions in the previous experiments) compared to the original LMEA and MOEA/DVA on 42 different large-scale instances from the 2- and 3-objective LSMOP, UF and DTLZ families. In these experiments we obtain mixed results and, more importantly, different results than above when the same algorithms were used with random groups and small computational budgets.

Comparing MOEA/DVA and LMEA reveals that LMEA can only win in around 16%

of problem instances while it is outperformed by MOEA/DVA in over 76% of cases.

This picture opposes that of Table 6.9 and indicates that MOEA/DVA uses its large computational budget more efficiently than LMEA. More precisely, MOEA/DVA has a mechanisms of updating the population during the interaction-based grouping. If a solution produced throughout the interaction analysis can dominate a population member, the population is updated accordingly. Therefore, a large share of function evaluations (see Table 4.4) is used to actually contribute to the search process in contrast to LMEA, where the interaction-based analysis is carried out independently of the subsequent optimisation steps. This implementation detail may explain why MOEA/DVA can outperform LMEA in the experiments that include the interaction-based grouping mechanisms. This theory is explored further below in Section 6.7, where exactly this property is deactivated to examine the influence of variable groups on the search in more detail.

When taking the WOF algorithms into account, we observe that LMEA is not able to compare with the rest of the algorithms in their unity. Out of all 42 large-scale problems, LMEA performs significantly worse than one of the other three method on 40 benchmark instances, ties with the other algorithms on the 2-objective DTLZ7 and performs best only one time, on the 3-objective DTLZ7. The performance of MOEA/DVA compared to WOF reveals more mixed results, with WOF-SMPSO winning against MOEA/DVA in 20 instances, and also loosing in 20 instances. The same holds for the randomised WOF version. A closer look into the tables in the appendix (Tables B.37 to B.39) reveals that

results on the LSMOP are generally mixed, while MOEA/DVA seems to be the better option in almost all UF problems and WOF seems to be the better choice in almost all DTLZ problems.

Comparison with S³-CMA-ES

The last comparison with the state-of-the-art on a large computational budget involves the S³-CMA-ES algorithm. This method is based on the concept of covariance matrix adaptation, and also utilises interaction-based variable groups. The results based on 28 large-scale problems are shown in Table 6.13 and Tables B.44, B.45 and C.12 in the appendices of this thesis.

The results, in comparison with MOEA/DVA, LMEA and two WOF versions, do reveal a rather poor performance of S³-CMA-ES, which is significantly worse than the state-of-the-art in 82% and 75% of cases compared to the WOF-SMPSO and WOF-Randomised respectively. It is further outperformed by MOEA/DVA and LMEA in 60% and 53%

of benchmarks respectively. While these numbers suggest that S³-CMA-ES is at least comparable to LMEA in its performance (winning 42% against LMEA), it shows a worse performance especially in comparison with WOF. And while S³-CMA-ES can win in 9 out of 28 instances (32%) against MOEA/DVA, it must be noted that these experiments did not include the UF benchmarks (reasons are explained below). It is therefore expected that if more benchmarks like UF were used, the percentage of wins of MOEA/DVA compared to S³-CMA-ES would lie even higher, since we have seen in the above experiment that MOEA/DVA performs mostly superior to other algorithms on the UF benchmarks.

Next, we focus on a major weakness of the S³-CMA-ES algorithm. While this method is built upon the (otherwise successful) concept of covariance matrix adaptation, a big problem actually lies in the size of the covariance matrices in the algorithm. S³ -CMA-ES uses interaction-based groups, and optimises each group separately using its own covariance matrix adaption instance to optimise only the variables in a specific group.

Therefore, each of the matrices’ dimensions are determined by the sizes of the variable groups. Since the dimensions of the matrices rise quadratically with the number of variables, a quadratic amount of memory is needed to store them. The issue, however, lies in the computation time, since inside of the algorithm matrix multiplications and inversions are calculated frequently. For an optimisation problem containing equal-sized groups, which do not contain too many variables each, this results in several moderate-sized matrices. However, when problems are non-separable, the interaction analysis produces (in the worst case) one large group of variables which includes the entirety of variables. In our experiments, this has been the case for all of the UF problems except UF3 and the LSMOP1 and LSMOP3 problems. In these cases, the algorithm operates with one large matrix of size 1000×1000 and as such uses a large computation time for multiplication and especially inversion of these. In our experiments, while S³-CMA-ES

6.6. COMPARISON WITH RELATED LARGE-SCALE APPROACHES 165

solved other problems with 1000 variables like LSMOP8 or DTLZ1 in around 14 minutes on a single core, for some of the UF functions a single run of the algorithm took more than 22 hours of calculation time. In case of the LSMOP benchmarks, not a single run had finished after more than 46 hours of computation time. Since this kind of runtime would make it impossible to test this algorithm for over 30 independent runs on these problems, the UF and LSMOP1 and LSMOP3 problems were therefore omitted in this experiment.

In conclusion, S³-CMA-ES certainly has interesting properties, since it is easily adapted into a parallel algorithm (see Table 4.2). On the other hand, it is outperformed in most instances by WOF or MOEA/DVA, and it heavily depends on suitable variable groups of small to moderate sizes. If this algorithm was applied to real-world problems with non-separable variables, its long computation time, apart from the usual required time for the 10,000,000 function evaluations, might render S³-CMA-ES unsuitable in practise.

0. 0 0. 2 0. 4 0. 6 0. 8 1. 0 Evaluations

10

10

10

10

10

10

10

IGD Value

MOEA/DVA WOFRandomised S3-CMA-ES LMEAWOF-SMPSO

(a) 2-objective DTLZ2

0. 0 0. 2 0. 4 0. 6 0. 8 1. 0 Evaluations

10

10

10

10

10

10

10

10

IGD Value

MOEA/DVA WOFRandomised S3-CMA-ES LMEAWOF-SMPSO

(b) 2-objective LSMOP7

Figure 6.11: Convergence behaviour of LMEA, MOEA/DVA, S³-CMA-ES and different WOF versions. All problems have 1000 decision variables.

Finally, we look into the convergence behaviour of LMEA, MOEA/DVA and S³-CMA-ES in Fig. 6.11. Multiple interesting features are visible in these two plots of the 2-objective DTLZ2 and LSMOP7 problems. First of all, due to the contribution-based and interaction-based grouping phases, the MOEA/DVA, LMEA and S³-CMA-ES start their optimisation procedures later than the two WOF versions. This is visible immediately in the IGD values of LMEA and S³-CMA-ES since they do not produce IGD values before their groups are finished. Here we also see immediately the difference in the grouping mechanisms.

S³-CMA-ES uses a version of DG2, and therefore finishes the grouping phases after

around 500,000 evaluations. In contrast, LMEA and MOEA/DVA need multiple millions of evaluations in both benchmarks before they obtain the interaction-based groups. In MOEA/DVA, however, this is not immediately visible, since it saves the solutions created during the interaction analysis and constantly updates its initial population during the process. Especially in Fig. 6.11a it is visible that the IGD values gradually improve until the optimisation starts after around 9 million evaluations, and the IGD improves rapidly afterwards.

Another interesting observation is the sudden increase in IGD of the S³-CMA-ES. This algorithm has two different stages, which are the independent optimisation of the popula-tions until convergence and the diversity optimisation. It is visible that the IGD converges in the beginning and once no more improvement is possible, the algorithm detects that all populations have converged, starts the optimisation of the diversity-related variables, and creates new independent populations for the next iteration. However, this mechanism seems to increase the IGD values in both benchmark functions, and leads to an overall worse performance. This insight can be valuable to reconsider the way the diversity optimisation is carried out in S³-CMA-ES, or how the preservation of good solutions in an archive is implemented in future versions of the S³-CMA-ES.