Centralized Configuration in Weakly Heterogeneous ScenariosScenarios

Initially, we focused on weakly heterogeneous environments, consisting of exactly one powerful device and several weak devices. In these scenarios, we first compare the new centralized Direct Backtracking algorithm to its closest competitor to show the efficiency of our algorithm. Then, we evaluate Direct Backtracking in comparison to the decentralized algorithm which was previously used in our system.

Experimental Setup

We first evaluate the centralized DBTalgorithm in the PCOM Simulator and com-pare our algorithm to Synchronous Backtracking (SBT, [YDIK98]), which is a depth-first search algorithm that does neither perform backtracking avoidance, nor does it provide intelligent backtracking mechanisms. As mentioned in Section3.1, advanta-geous algorithms like Synchronous Backjumping, Dependency-Directed Backtrack-ing or Dynamic BacktrakBacktrack-ing either rely on a stack with monotonously growBacktrack-ing space consumption, or change the order of the variables, which is not an option here.

After having shown thatDBTsignificantly outperformsSBTin typical Pervasive Computing scenarios, we compareDBTto the distributed assembler that is based on Asynchronous Backtracking (ABT, [Bak05]) and was developed in an earlier work by Handte et al. [HBR05].

For our evaluations, we use the following variable system input parameters to distinguish between different scenarios:

• Application size: For these evaluations, we use applications with abstract components which form binary trees. By selecting different depths for the application trees, we adapt the size of the applications to analyze the behavior of the algorithms in various realistic scenarios.

• Number of multi-optional contracts: Multi-optional contracts have a large impact on the efficiency of specific configuration algorithms, as they rep-resent points where the algorithm can choose among several options. Thus, when we compare DBT to SBT, we evaluate different numbers of multi-optional contracts to compare the performance of the algorithms in scenarios with difference degrees of resource availability.

• Number of involved devices: While the efficiency of the actual centralized configuration calculation is rather depending on the computation power of the single configuration device, the number of totally available devices has a strong impact on decentralized configuration approaches, as all devices have to communicate to each other there. Furthermore, the number of available devices also influences the time it takes to gather the resource information of the remote devices as well as distributing the results after the configuration to the other devices.

Furthermore, we evaluate the algorithms according to the following metrics:

• Configuration latencies: The most important metric in our evaluatiosn to identify the efficiency of an algorithm is the configuration latency, which represents the time span between the start of the configuration process and the provision of the calculated application composition. Thus, we present results of extensive measurements where we compare our centralized approach with related centralized and decentralized approaches in weakly heterogeneous environments.

• Space and memory overhead of Direct Backtracking: We compare DBT and SBT according to the space consumption of their Java classes on the device’s hard disk, and according to the consumed Random Access Memory (RAM) while the algorithms are running.

• Class loading latencies of the clustering framework: As the developed pre-configuration process requires a proactive loading of the needed configura-tion classes, we measure the latencies that arise by this process. Please notice that these times do not affect the configuration calculation latencies, since these tasks are supposed to be performed prior to the configuration.

Moreover, we determine the break-even points when we compare DBT to SBT.

There, we show that there exist only few scenarios where SBT outperforms DBT.

However, by determining the break-even points in scenarios with different appli-cation sizes and fractions of multi-optional contracts, we will discover that DBT performs better thanSBT in the vast majority of the evaluated scenarios.

Latency Comparison of Direct Backtracking and Synchronous Backtracking The main goal in the development of an advanced centralized configuration algo-rithm was the reduction of the configuration latency that is noticeable for the user.

Figure 4.34 shows DBT’s latency relative to the respective SBT performance with three different application sizes (15, 31, and – to evaluate scalability of the ap-proaches – 63 components). Values above 1.0 indicate a DBT performance that is worse than that ofSBT, while values below 1.0 indicate better performance.

On average, there is an immense improvement when using DBT, which becomes particularly large when many contracts are multi-optional: SBT needs to perform many backtracking processes there, while DBT avoids most backtracking processes due to its proactive backtracking avoidance. Furthermore, it performs the remaining adaptations much smoother due to its intelligent backtracking, and avoids thrash-ing effects by considerthrash-ing the cause of a backtrack. DBT’s performance gain even increases with increasing application size, as adaptation processes cover more com-ponents then. For instance,DBT induces just about 4.2 % of the latency of SBTif an application consists of 63 contracts and 14 of these contracts are multi-optional.

Nevertheless, it must be mentioned that for small fractions of multi-optional con-tracts, SBT performs better than DBT by up to 20 %. This is because of DBT’s additional checks for avoiding conflict situations and the process of storing the

back-4.5. Evaluation 129

Figure 4.34.: Latency of centralized configuration with DBT, relative to SBT refer-ence (k ∈ {15,31,63})

tracking causes. Since absolute latencies were very small in those cases (in the range of few milliseconds), we consider this overhead as neglectable.

Subsequently, we determine thebreak-even points, which represent the fractions of multi-optional contracts where DBTstarts to outperform SBTwhen an application of a specific size is used. Therefore, we perform measurements with various tree heights (i.e., application sizes) and numbers of multi-optional contracts. Figure4.35 shows the relative performance of DBT and SBT depending on these two param-eters. The figure shows that with an increasing tree height, the break-even points exponentially decrease in dependence of the fraction of multi-optional contracts.

This means that for huge applications, even if just a small amount of contracts is multi-optional, DBT is the better choice regarding configuration latency. Thus, DBT is particularly helpful in reducing the latencies when extensive calculations have to be performed.

Besides the configuration latencies, we evaluated the following issues:

• Communication overhead: Compared to the other centralized backtracking algorithms presented in Section 3.1, no additional communication overhead arises during runtime of our algorithm, as the calculations are solely performed on a single device.

• Success quota: The scenarios have been created in a way that at least one valid configuration exists for each scenario. In every single simulation run, both SBT and DBTterminated successfully with a valid application configuration.

• Memory overhead: Compared to SBT, DBT needs to store additional in-formation about arising conflicts, especially the contract chosen for adaptation and the contract where the algorithm has to continue after an adaptation has been performed. We measured the average random access memory

consump-Figure 4.35.: Break-even points, depicted by borderline

tion of the algorithm on a common desktop PC and compared it to SBT.

While the memory consumption ofSBTwas almost independent from the ap-plication size and the number of conflicts (the standard deviation was below 2 % in all runs), the overhead ofDBTincreased with the application size and the number of multi-optional contracts, but remained within acceptable limits.

The average memory overhead ofDBTvaries between 8.0 % for small applica-tions with tree height 2, and 27.6 % for very large applicaapplica-tions with tree height 10. The maximum overhead of DBTcompared to SBT in a single simulation run was 38.5 %, the absolute amount of required memory was 19.2 Megabytes.

The results of the memory evaluation are shown in Figure4.36.

• Source code size: While our Synchronous Backtracking implementation needs 10.8 kB of disk space, Direct Backtracking consumes about 96.9 kB, mainly because of the code overhead for proactive backtracking avoidance and intelligent backtracking.

Regarding that the algorithm is optimized for the use on resource-rich devices be-cause of its centralized nature, the additional code overhead as well as the memory overhead do not prevent the use of DBT.

Latency Comparison of Direct Backtracking and Asynchronous Backtracking In the previous section, we have shown that DBT significantly outperforms SBT due to its advanced backtracking mechanisms. Now, we focus on the efficient ex-ploitation of the computation power of an available resource-rich device in a weakly heterogeneous environment now.

Therefore, we perform measurements on our prototype and compare centralized configuration that uses DBT (cf. Section 4.2.1) and VCs (cf. Section 4.3.5) to the decentralized approach [HBR05] based on Asynchronous Backtracking (ABT)

4.5. Evaluation 131

Figure 4.36.: Memory overhead of Direct Backtracking

[YDIK98] in two environments: a homogeneous environment consisting only of resource-weak devices, and a weakly heterogeneous environment that consists of one resource-rich device and up to six resource-poor devices.

As the realized configuration process uses the presented Virtual Container con-cept, the required classes have to be loaded via mobile code, which causes additional latencies. This has to happen initially, and also after environmental changes, e.g.

the presence of new devices, have taken place. These latencies are presented in Fig-ure 4.37. The figure states that in the heterogeneous environment, the class loading is performed by 33 % faster in case of two devices, and up to 55 % if five devices are involved compared to the homogeneous environment (where the classes were loaded on one of the resource-weak devices). This is because the loading of remote classes is performed significantly faster on a resource-rich device.

When the cluster head changes, a handover mechanism (implemented within the so-called Handover Mobile Code Accessor, cf. Section 6.3.2) immediately transmits the previous cluster head’s state to the new cluster head. Because of this, the new cluster head does not need to request the configuration classes from the involved devices. Figure 4.37 displays that when the cluster head changes for a number of resource-weak devices, the use of this handover mechanism reduces class loading latencies up to 33 % in a pure Ad Hoc environment with five involved devices, and around 31 % in the corresponding heterogeneous scenario. Thus, the use of such a handover mechanism can significantly speed up class loading processes, particularly when the number of involved devices rises.

Figure4.38shows the measured overall configuration latencies in the same scenar-ios. In every single measurement, DBTcalculates a valid configuration much faster than ABT. On average, DBToutperforms ABT by around 38.6 %. This is because DBTcalculates configurations completely local on the cluster head, as the required resource information is proactively obtained when a cluster member is mapped and

Figure 4.37.: Class loading latencies (k = 15)

stored in the Virtual Containers. Hence, the resource-rich cluster head does not have to wait for I/O operations.

Thus, centralized DBTconfiguration should be preferred over decentralizedABT configuration whenever exactly one resource-rich device is available, as this leads to considerable performance gains.