Simulation studies

In the simulation studies the methods were evaluated in term of sensitivity and specificity. Within each study 17 parameter configurations (Table 2.4) were considered and for each configuration the median sensitivity and specificity of 1000 simulation runs were computed. The parameters and their settings are described in detail in 2.3Simulations section.

3.2.1.1 Study 1: with original pathways

Figure 3.2 depicts the results of simulation study with the original overlapping KEGG pathways.

With weak changes of the mean vector (mean = ±1) sensitivity of GS methods based on ranking was the best: 0.67 for Wilcoxon rank sum (WRS) and 0.58 for Kolmogorov-Smirnov (KS), followed by PathNet with a sensitivity of 0.5. All methods based on over-representation analysis (ORA) (both GS and PT-based) had 0 sensitivity; however, that was reflected in a specificity of 1. With a higher mean change, i.e. mean = {±2,±6}, all methods were comparably sensitive (between 0.92 and 1), whereas the best specificity scores (0.51 and 0.49) were reached by WRS. In the case of small pathways being deregulated, most of the methods were less sensitive than in detecting the bigger pathways. However, CePa GSA (self-contained) and WRS performed best with a sensitivity of 0.92 and 0.83, respectively. Specificity decreased for the medium and big pathways. Especially CePa GSA was very unspecific on the big pathways (0.22) in comparison to the other methods, whose specificity ranged from 0.46 to 0.56. When more than half of the database pathways were deregulated (N = 70) both sensitivity and specificity decreased compared to N ={12,23}. For N = 70 CePa GSA specificity was only 0.087 whereas the others ranged between 0.3 and 0.5. At low level of detection call (DC= 10%)

coupled with betweenness topology design the PT-based methods and Fisher’s exact (FE) test were more sensitive (0.58−0.92) than ranking GS methods (0.33−0.58). However, on 30% and higher detection call (DC) levels all methods performed comparably well in the term of sensitivity (0.83−1). At the same low level of detection call (DC = 10%) coupled with neighborhood topology design PathNet had the highest sensitivity of 0.58, followed by other PT-based methods and FE (0.42−0.5). In comparison to the corresponding simulation type with betweenness design, in this setting only CePa ORA and CePa GSE reached a sensitivity over 0.8 forDC = 30%.

The sensitivity was fairly high across the configurations and rather com-parable for both GS and PT-based methods. However, the specificity over all parameter configurations was very low for all methods. Thus, I performed the same five simulation types on the non-overlapping pathways in the Study 2.

3.2.1.2 Study 2: with non-overlapping pathways

Figure 3.3 depicts the results of simulation study with the non-overlapping KEGG pathways, whose topology was preserved but the nodes were assigned with unique synthetic IDs. This resulted in markedly better specificity, whereas the sensitivity results tell rather similar story as in the Study 1.

On the low level of mean parameter (mean=±1) again the GS methods based on ranking were the most sensitive (0.5). In detecting small pathways the best sensitivity (0.75) was reached by WRS and PathNet. Whereas for the big pathways sensitivity of WRS was 1 and of PathNet 0.92. The sensitivity of all methods except CePa GSA decreased when many pathways were deregulated (N = 70), even to a bigger extent than in Study 1. Interestingly, FE and KS specificity was only 0.17 for N = 70, while most of the other methods reached a specificity of 1. When pathways did not overlap the DC level of 50% in the betweenness design was needed to achieve sensitivity over 0.8, whereas for the original pathway study all methods reached 0.8 sensitivity at a DC = 30%

level. For the low DC (DC = 10%) PathNet’s sensitivity was the best (0.58).

Simulation type of DC coupled with neighborhood topology design had a similar behavior pattern as the betweenness setting; only for the both CePa methods sensitivity on DC= 10% level decreased to 0.

3.2 Comparison of enrichment methods 43

Figure 3.2. Simulation study 1 using original pathways with overlapping genes: Sensitivity and specificity scores of seven methods under 17 parameter configurations. Each cell summa-rizes a median value of 1000 runs. The same color code key applies for all simulation types.

Figure adapted from Bayerlov´a et al. (2015a).

Figure 3.3. Simulation study 2 using non-overlapping pathways with unique gene IDs:

Sensitivity and specificity scores of seven methods under 17 parameter configurations. Each cell summarizes a median value of 1000 runs. Figure adapted from Bayerlov´a et al. (2015a).

3.2 Comparison of enrichment methods 45

3.2.1.3 Overall performance in simulations

Average sensitivity, specificity, and accuracy were calculated across 17 configura-tions for each method in both studies (Figure 3.4). In the first simulation study with overlapping pathways the average specificity was very low for all methods, achieving maximum of 0.55 for WRS and SPIA. This was also reflected in the best accuracy of these two methods: 0.6 for WRS and 0.59 for SPIA. The rest of the methods achieved comparable accuracy, with levels between 0.53 and 0.57, with the exception of CePa GSA, which showed an average accuracy score of 0.46. In the second simulation study with non-overlapping pathways, both overall specificity and accuracy increased. For PT-based methods the average specificity was 1, whereas there were prominent differences within GS methods;

average specificity for both WRS and FE was 0.89, for KS it was only 0.65.

The sensitivity in Study 2 varied moderately among the methods from 0.63 up to 0.77. The best overall accuracy in the non-overlapping pathways setting was achieved by the four PT-based methods (0.95−0.96), followed by FE with 0.88 and WRS with 0.86 scores.

Figure 3.4. Overall sensitivity, specificity and accuracy in the two simulation studies: Mean of three measures for each method over 17 configurations.