With the implementation of the local search framework and a machine learning-based oracle with high accuracy, we can now present the first results of our framework. There are, however, several things to consider in this evaluation. The starting instance has the most influence on the local search. Parameters like the neighborhood size, and how much additional travel time is acceptable also significantly impact the process. Because the microscopic rerouting during the generation of an instance is expensive this is also only done once in a set interval of optimization steps.
For the first set of experiments, we selected these parameters as follows: neighborhood size of 4N = 80, a rerouting interval of 10, and a possible travel time increase of 10%
9.5 Results of Optimization Framework 133
0 5 10 15 20 25 30 35 40 iteration
82.5 85.0 87.5 90.0 92.5 95.0 97.5 100.0
realrobustness(%ofinitial)
100 102 104 106 108 110
averageperceivedtraveltime(%ofinitial)
Real Robustness Travel Time
Fig. 9.12: Example instance of thegoevb net-work [MH+22]
0 10 20 30 40
iteration 82.5
85.0 87.5 90.0 92.5 95.0 97.5 100.0
realrobustness(%ofinitial)
100 102 104 106 108 110
averageperceivedtraveltime(%ofinitial)
Real Robustness Travel Time
Fig. 9.13: Example instance of the lowersaxonynetwork [MH+22]
Typical instances from optimization have good costs and connections for the passengers. These instances often suffer from bad robustness. We used an example for these instances in our experiments. Figure 9.12 shows the behavior of such a starting solution for the local search on the datasetgoevb. After 40 iterations, the robustness of the instance is improved by 13%, while at the same time the average perceived travel time is increased by 10%. We already mentioned that we are only willing to sacrifice a defined portion of the travel time to gain robustness. This is why the local search finished at this point.
A similar instance from the lowersaxony dataset is shown in figure Figure 9.13 with an improvement in robustness of 17.5% while increasing the average perceived travel time again by 10%. Note that we are showing the real robustness here, i.e., the robustness given by simulations of the corresponding instances instead of the predicted value. Furthermore, we see that while the local search only chooses a new solution if the predicted robustness is better than before this must not be the case for the real robustness, i.e., the real robustness observed is not monotone.
In both cases, it can be observed the perceived traveling jumps every ten steps. This is because of the rerouting intervals already mentioned. In between, the passenger paths are assumed to be constant even when the public transport schedule changes. This small error, which is corrected every ten steps, is gaining a significant amount of running time.
The local search performs differently when the starting instance already has good robustness.
Figures 9.14 (for datasetgrid) and 9.15 (for datasetring) show such instances. Both instances show, that further improvement is possible. An intriguing behavior can bees in instance ring where it was also possible to improve the average travel time in addition to the robustness. The machine learning predictor shows problems once instances get further away from the initial clusters. Since the oracle does not have any real knowledge about the importance to evaluate the quality of estimation. To investigate this effect, we revisit the example from Figure 9.12 and observe not only the reevaluated real robustness but the predicted robustness as well in Figure 9.16. While the real robustness is improved by 13%, the estimated robustness improvement
134 Chapter 9 Improving the Robustness of Schedules Using Machine Learning
0 25 50 75 100 125 150 175 iteration
70 75 80 85 90 95 100
realrobustness(%ofinitial)
100 101 102 103 104 105
averageperceivedtraveltime(%ofinitial)
Real Robustness Travel Time
Fig. 9.14: Example instance of the grid net-work [MH+22]
0 10 20 30 40
iteration 90
92 94 96 98 100
realrobustness(%ofinitial)
98.0 98.5 99.0 99.5 100.0 100.5
averageperceivedtraveltime(%ofinitial)
Real Robustness Travel Time
Fig. 9.15: Example instance of the ring net-work [MH+22]
0 5 10 15 20 25 30 35 40
iteration 20
40 60 80 100
realrobustness(%ofinitial)
100 102 104 106 108 110
averageperceivedtraveltime(%ofinitial)
Estimated Robustness Real Robustness
Travel Time
Fig. 9.16: Measuring gap between estimated and real robustness. Exam-ple instance of the goevb net-work [MH+22]
0 10 20 30 40
iteration 80.0
82.5 85.0 87.5 90.0 92.5 95.0 97.5 100.0
realrobustness(%ofinitial)
100 102 104 106 108 110
averageperceivedtraveltime(%ofinitial)
Estimated Robustness Real Robustness
Travel Time
Fig. 9.17: Measuring gap between estimated and real robustness. Example in-stance of the lowersaxony net-work [MH+22]
is much larger with about 80%. The local search procedure is overestimating the robustness improvement immensely. After the local search is finished it is always important to reevaluate the result using the original robustness test. However this was a rather extreme case, and an average tracking is seen in Figure 9.17, a revisit of Figure 9.13.
Because the re-training of the oracle is relatively fast we can mitigate this effect. In the future, instances containing new structures should be added to the training as soon as their real robustness is known.
Since we are interested in instances that have a good perceived travel time and/or good robustness, there are many instances that are of interest. The final evaluation of the local search shows all starting and finishing solutions in Figure 9.18 and 9.19 where the improvement in the Pareto-front can be seen. Especially forgrid (Figure 9.18) we are able to produce solutions that
9.5 Results of Optimization Framework 135
0 20 40 60 80 100 realized robustness
8 10 12 14 16 18 20 22
averageperceivedtraveltime
Original Instances Local Search Solutions
Fig. 9.18: Pareto front of gridinstances
0 20 40 60 80 100
realized robustness 20
40 60 80 100 120
averageperceivedtraveltime
Original Instances Local Search Solutions
Fig. 9.19: Pareto front ofring instances
are far better w.r.t. the travel time of the passengers and the robustness. For ring(Figure 9.19) the new Pareto-front has a smaller benefit, but a better range of solutions.
In addition to the methods and experiments described in this chapter, we published another set of optimization experiments [Mül+21b]. In this work, we used the ANN one net oracle, in contrast, to support vector regression when optimizing our instances. Instead of a deterministic local search approach to improving on starting instances we used a genetic algorithm to get more diverse solutions trying to have more improvements along the whole Pareto-front. Figure 9.20 shows the results of this approach.
