Authentication Speed

0 500 1000 1500 2000 2500 3000 3500

active auth. times in ms

Mean 2130 2200 2510 2670

SD 160 220 180 320

Telephone Calculator Linear Random 0

500 1000 1500 2000 2500 3000 3500

preparation times in ms

Mean 1330 1580 1980 2600

SD 120 360 520 740

Telephone Calculator Linear Random

Figure 4.4: Left: The average times of the different layouts required for the active authenti-cation phase. Right: Average times for the preparation phase. The graphs show that prepa-ration time plays an important role when measuring the performance of an authentication system.

And finally, based on the follow-up study:

(H4) Using a consistent layout allows for a significant learning process, while using the random layout does not.

Having a look at the results of the preparation phase (figure 4.4, right), reveals a similar trend.

However, the negative effects of unfamiliar layouts are even graver in this phase. Again, the tele-phone layout requires the shortest time on average (M=1.33s, SD=0.12s) followed by calcula-tor (M=1.58s, SD=0.36s), linear (M=1.98s, SD=0.52s) and random layout (M=2.6s, SD=0.74s).

A one-way ANOVA revealed a significant main effect for keypad layout on preparation time (F2.15,126.66=5.63,p<.05, Greenhouse-Geisser corrected values). Tests of inner subject contrasts showed significant differences in the times required for the telephone layout compared to lin-ear (F_1,59=4.84, p<.05) and random layout (F_1,59=12.26, p<.05). The calculator layout shows no significant difference to the telephone and linear layout (allp>.05). Nevertheless, it shows a significant advantage compared to the random layout (F_1,59=4.84,p<.05). Again, no significant differences between the linear and the random layout were identified (p>.05).

Interestingly, performance issues for the linear and the random layout were almost identical. The analysis showed that performance problems of the linear layout were mostly due to longer active authentication times. This can be explained with the bigger distances between the keys on the keypad. On the other hand, the random layout mainly suffers from prolonged preparation times due to lack of consistency.

While for all layouts the preparation phase is shorter than the active authentication phase, the random layout requires almost identical amounts of time for both. That is, the overhead added by the preparation phase is around 100% for this layout. Considering the time sensitive nature of authentication, this can be considered a serious drawback of random layouts.

The results showed that the more the layout differs from the one the participants are used to (telephone layout), the more time is required as well for active authentication as for preparation.

This effect is even higher for the preparation phase. Based on these results, hypothesis (H3) can be accepted. The ratings of the layouts by the participants give additional confirmation for this hypothesis. The telephone layout was rated the fastest on a Likert scale from 1 (very slow) to five (very fast). 95% of the participants considered it very fast (79.7%) or at least fast (15.3%). The calculator layout is ranked second (28.8% very fast, 49.2% fast) followed by the linear layout (5.1% very fast, 18.6% fast). The random layout scored rather badly with 42.4%

of participants rating it as very slow. Therefore, the subjective opinion of the users correlates with the quantitative results of the study. The quantitative differences become even clearer when taking an in-depth look at the differences between the telephone and the random layout.

More Telephone vs Random Layout

The follow-up study was performed to allow for a more precise comparison between the tele-phone and the random layout. Therefore, this analysis is based on the data of these two groups (trained telephone and trained random) over a period of seven days during which authentication was performed once per day. The first day counted as training and was therefore not included in the analysis. Authentication times are reported without the time required to confirm. This was done to avoid unfair disadvantages of the systems. Only correct authentication attempts were included in the analysis.

0 500 1000 1500 2000 2500 3000

active auth. times in ms

Telephone 2241 1946 2136 2122 2158 2040

Random 2336 2618 2765 2561 2743 2323

day 1 day 2 day 3 day 4 day 5 day 6 0

500 1000 1500 2000 2500 3000 3500

preparation times in ms

Telephone 1621 1402 1790 1228 1038 989

Random 2168 1639 1671 3144 1800 1841

day 1 day 2 day 3 day 4 day 5 day 6

Figure 4.5: Left: Active authentication times for the telephone and the random layout.

Right: Preparation times show a significant learning effect for the telephone layout.

Figure 4.5 shows the comparison of the average times for the telephone and the random layout.

The left graph shows the speed of active authentication. For the telephone layout, all times are between 1.94 and 2.24 seconds. We conducted an analysis of variance which indicated no significant main effects of days (F_3.6,100.6=0.85, p>.05, Greenhouse-Geisser corrected values).

Authentication times of the random layout also show no effects of training (F3.3,32.7=1.25,p>.05, Greenhouse-Geisser corrected values) and vary slightly from 2.32 to 2.77 seconds. Within the single test days, the time differences between the two layouts are not significant too (allp>.05).

The black trend lines show how little effect exercise had on the times of both layouts. The linear trend lines are nearly parallel to the x-axis for both.

It should be noted again that in most studies, only active authentication time is measured and interpreted. Under the assumption that we had examined only this as well, we would have nec-essarily come to the conclusion that the additional work, which is associated with the use of a random layout, although measurable, is only marginal and stable. In addition, we would have to conclude that the lack of learning in the random layout is not a disadvantage, because the use of the telephone layout does not show any significant improvements as well.

However, taking a closer look at preparation times, which are shown in figure 4.5, right, it can be seen at first sight that the trend lines of both layouts are far apart. While there is no improvement for the random layout, the preparation times of the telephone layout constantly decrease with the exception of day 3. A test of the inner subject contrasts for the telephone layout showed that the times of the first (F1,28=6.51,p<.05) and second day (F1,28=9.31,p<.05) differ significantly from the time of the sixth test day⁴. Thus a significant learning effect was found using the telephone layout which was only revealed by evaluating the preparation phase.

This result is also reflected in the consideration of the overall time: preparation + authentication + confirm (no cleanup was measured since there was no additional interaction after confirm).

By including the preparation time, we see that the constant use of a fixed layout in conjunction with repeated use can increase the authentication performance. A t-test showed that the times

4 The “first day” refers to the second day of authentication since the real first day was considered as a learning phase.

0.00 1.00 2.00 3.00 4.00 5.00 6.00

error rates per layout in %

3rd grade errors 0.17 0.83 0.00 0.83

2nd grade errors 0.33 0.00 0.00 0.83

1st grade errors 2.33 2.50 3.33 3.33

Telephone Calculator Linear Random 0

1 2 3 4 5 6

absolute number of errors

Telephone 4 5 2 3 4 2 0 1 0 0

day 1 day 2 day

3 day 4 day

5 day 6 day

7 day 8 day

9 day 10

Figure 4.6: Left: Error rates for the different keypad layouts. Errors of 3rd grade indicate critical errors. Right: Error rates for the telephone layout over time. The graph contains critical as well as basic errors.

of the two layouts differ significantly beginning with the fifth day. The test results for the fifth, (t(38)=2.69, p<.05, r=.42) and sixth test day (t(38)=2.13, p<.05, r=.33) show significant differ-ences of medium effect size. These results confirm hypothesis (H4).

The importance of measuring preparation time is additionally highlighted by the fact that in our experiment, it was for many participants longer than the active authentication time. This effect was especially high for the random layout. That is, actual overall time was up to 100% more in these cases. Since the preparation time is significantly different for both layouts, neglecting it leads to completely wrong conclusions.