Evaluating and Comparing Docking Hypotheses

For testing docking algorithms, the hypotheses predicted by ElMaR are compared to a known complex⁵with identical sequence. A well known measure to compare two structures is the so called root mean square deviation (RMSD):

RMSD= 1 N

∑

q

(a_i−bi)² (7.5)

The RMSD value gives the euclidian distance (in ˚A) between two structures a and b. A small value indicates a good similarity whereas a large value shows significant differences between them. In equation 7.5,Nis the number of C_αatoms. a_i andb_idenote the atoms of the two structures to be compared. In order to calculate the RMSD the two structures are superimposed.

Besides the RMSD, the ranking of docking hypotheses is of interest, too. Halperin and coworkers (Halperin et al., 2002) defined a set of different measures including rank and RMSD, called DRUF (Docking Results Unified Format). Here, the N10, N50 and N100 measures are chosen because these scores reflect the quality of predictions of a complex from two unbound proteins. The measures are defined as follows:

N10: Number of hypotheses within the first 10 ranks with an RMSD≤3 ˚A.

N50: Number of hypotheses within the first 50 ranks with an RMSD≤4 ˚A.

N100: Number of hypotheses within the first 100 ranks with an RMSD≤5 ˚A.

By the definition of these measures, changes within the result set a test case processed in the different experiments is very simple. Since all results are stored in the database, the calculation of the NX scores can be realised through SQL queries (see Fig. 7.10).

select Entry, count(*) from Hypothesis

where rmsd<[3|4|5] and rank<[10|50|100] group by Entry;

Figure 7.10: SQL query for the N10, N50, N100 measure of the DRUF Protocol.

Other measures defined within DRUF are:

5Known complex means a crystallographically refined complex structure, deposited in thePdb.

• RMSD of the hypothesis at Rank #1

• Rank of the first solution with RMSD <5 ˚A

• Rank of best RMSD hypothesis

A disadvantage of these scores is that the flexibility of a side chain is not taken into account.

The different side chain conformations of multiple hypotheses cannot be compared by the C_αRMSD. Furthermore, the DRUF protocol only focuses on the top ranked hypotheses. A measure that summarises the performance of a docking run over the whole set of hypotheses would be desirable. Neumann (Neumann, 2003) proposed the IPI (Integrated Performance Indicator). The IPI summarises the performance by a weighted sum of the scores of all hypotheses:

IPI=

∑

i

Rankmax−Ranki

Rankmax

| {z }

Rank weighting

·max(10˚A−RMSDi,0) 10˚A

| {z }

RMSD weighting

+

(p_a i f ^RMSD_scoreⁱ

i >^RMSD_score^max

max

pb else (7.6)

The score of a hypothesis iis the product of the normalised rank and the weighted RMSD.

Additionally, an error term is added whether the hypothesis has an RMSD above or below the diagonal. The IPI is the sum of all hypotheses of the test case. Figure 7.11 visualises the components of the IPI measure.

Cost

RMSD

(a) RMSD between 0-10 ˚A normalised to [0..1] (RMSD weighting).

Cost

RMSD

(b) Rank normalised to the interval [0..1] (Rank weight-ing).

Cost

RMSD

(c) Exclusion of false posi-tives by the error term (last term of equation 7.6).

Figure 7.11: The components of the integrated performance indicator. Hypotheses that fall into the green area contribute to a good score. Courtesy of Neumann (Neumann, 2003).

The IPI can give an overall score of the whole set of hypotheses. For a detailed analysis within parts of the result sets other methods have to be used. Besides the IPI, the minimal RMSD within the first 10, 50 and 100 ranks is considered. Here, changes within the best hypotheses can also be observed for test cases that are hard to predict (e.g. 1A2W).

Besides methods calculating scores that express the accuracy of the docking experiments, visualisation techniques can be used for qualitative analysis of docking experiments. There-fore, the rank or the costs are plotted against the RMSD. In this thesis several docking

experiments for one test case are conducted and have to be compared. Simply, the results of different experiments can be drawn into one plot. Since the number of hypotheses pre-dicted byElMaRis large (700 hypotheses per test case), these plots become rather complex.

In order to avoid this a different method for comparing and visualising the differences be-tween the docking runs is used. The whole plotting area is sampled into rectangles. Here, a rectangle of size of 10×1.5 is used. The size of the rectangle is abutted from the N10 measure. A width of “10 ranks” and a height of “1.5 ˚A” yields good results. On the one hand, the rectangle is not too small, e.g. several hypotheses are covered. On the other hand, the rectangle is not too large, so that a fine sampling is possible, visualising changes in detail. Within each rectangle the number of hypotheses of each experiment placed here are counted. By calculating the difference (∆Cx,y) between these numbers, changes can be easily observed:

∆Cx,y=C^B_x,y−C_x,y^A (7.7)

Here,C_x,y^B denotes the number of hypotheses of an experiment using flexibility information within the rectangle at positionx,y. C^A_x,y represents the number of hypotheses for the same docking experiment and the same rectangle but without using flexibility information. A posi-tive difference means that the number of docking hypotheses placed within the rectangle increased whereas a negative value shows a decrease, respectively.

Furthermore, the differences can be visualised by applying different colours for positive and negative differences and plotting the rectangles. The quantities of the changes can be expressed by the lightness of the colours. Light colours indicate few changes whereas dark shades represent many changes⁶.

0 50 100 150 200

0102030405060

Estimate

RMSD

Figure 7.12: Visualisation of changes between the docking of 1BELand1RAT.

6In case of no changes due to equal numbers of hypotheses or if no hypotheses are placed within that regions of the plot, the rectangles are coloured white.

Figure 7.12 visualises the application of this method to the results of the docking of 1BEL and 1RAT. Here, the docking with flexibility information is compared to a docking without flexibility. The differently coloured rectangles show the changes within the set of hypothe-ses. Green coloured parts show an increase, red coloured boxes a decrease in the number newly placed hypotheses.