Discussion

caused by untrained users. Only the random layout is not easier for the trained group. Due to the overall low numbers of errors, no significant effects could be found. Therefore, hypothesis (H1) cannot be confirmed. Nevertheless, the trend indicates that the lack of conistency does have a bad influence on memorability.

This is supported by the results of the final questionnaire. Mostly trained users mentioned to have used visual or muscle memory strategies. The lack of the initial consistent training week of the untrained participants did not allow for developing such strategies. It is not surprising that on a Likert scale from 1 (no negative effect on memorability) to 5 (strong negative effect on memorability), the telephone layout was considered to have no (88.1%) or little (10.2%) influence on memorability. The linear layout (42.4% no effect, 16.9% little effect) was considered only slightly worse than the calculator layout (67.8% no effect, 22% little effect). This result partially contradicts the quantitative findings of the study. Even though the random layout was rated worst (16.9% strong effect, 15.3% high effect), a large group of 39% of participants attested it to have no negative effect on memorability. Interestingly, mainly trained users rated the negative effect as very high. This can be explained with the fact that their advanced learning strategies worked specifically bad using the random layout.

Based on this, we propose that whenever evaluating an authentication system, an approach similar to the one proposed in figure 4.2 should be used. This way, it is very unlikely to oversee important facts. Additionally, it resolves an issue that was discussed earlier in this work. It makes different authentication mechanisms more easily comparable to each other. Even if not analyzed, the different phases should at least be reported to enable others to adequately compare different authentication methods.

A big issue that has to be solved for each mechanism and setting is how to measure preparation time. Many systems do not have a dedicated starting point as the one we simulated in our ex-periment. Therefore, measuring this time is rather hard and requires a lot of creativity and extra work by the researcher. For instance, in another experiment, we designed and evaluated an au-thentication mechanism for a mobile device based on grasp recognition. To measure the starting point, we built a hardware device that enabled us to measure whenever the mobile device was lifted which allowed us to get a fix on the exact starting point of the authentication task and thus measure preparation time.

Additionally, some authentication mechanisms can require new preparation during the actual active authentication phase (e.g. if they are based on several consecutive challenges). Based on the results on the influence of preparation, this can be considered a drawback of the system and is very likely to make muscle memory strategies impossible. To some extent, randomization can thus be considered a usability problem of authentication mechanisms. This is more problematic since randomization is often used as a simple way to provide enhanced security.

4.4.2 Influences of Consistency

Related work already gave first indications that consistency is of high importance especially for a short and focused task such as authentication [107]. Till now, one of the main drivers for consistency has always been accessibility [58, 90, 105]. Within the long-term study, we found several results that support the assumption of the necessity of consistency, performance-wise as well as memorability-wise. Therefore, the results give further evidence that consistency is of high importance. Within this work, two types of consistency are distinguished. Firstly, outer consistencyrefers to the consistent use of the same authentication method. Strictly speaking, the different keypad layouts of this study could be considered different authentication mechanisms.

Inner consistencyon the other hand means that when using the same authentication mechanism for two different times, the user has to do the same input in the same way to authenticate. In the example of this experiment, inner consistency is therefore absent when using a random layout since the users have to press keys in a different order every time they want to authenticate. Talking about consistency within this thesis refers to inner consistency if not indicated otherwise.

The analysis of the different time measurements revealed that the results are influenced by ex-treme values. Especially preparation times are influenced by those values. Having a look at the 95th percentile of the results of the preparation times (thus ignoring the last 5% of times), shows that the maximum becomes the more extreme, the more the layouts differ from the participants’

standard layout, the telephone layout. For the telephone layout, the maximum of the 95th per-centile is 2.58 seconds. Using the calculator layout already increases this value by 45.3% (1.17 seconds). The linear layout increases it by 66.7% (1.72 seconds). The layout with the biggest difference from the telephone layout, the random layout, adds 214.3% (5.53 seconds).

This increase can be explained with human perception. Unfamiliar layouts have to be scanned sequentially to find the right digits. From study observations, we know that especially for PIN, participants tend to start the input after they successfully have identified all required digits. An-other influencing factor is the fact that many participants used strategies based on visual and mus-cle memory cues (simply said, remembering the positions instead of the digits). These strategies do not work when the layout changes and thus the PIN has to be remembered in a more complex way. Thus, the lack of consistency has an additional negative influence if consistency is expected by the participants due to the normal authentication approach.

Results of the follow-up study using the random layout are better when it comes to preparation times and error rate. For instance, participants in the follow-up study created hardly any critical errors using the random layout while this layout caused most of the critical errors during the main study. This gives additional support to the assumption that consistency is an important criterion.

Lacking inner consistency in the keypad layout (being random), the participants had to employ other (less effective) strategies to remember their PINs. Therefore, the preparation phase of the follow-up study only consists of the time required for the sequential search and advanced memory strategies could not lead to confusion due to their absence.

Based on these results only, one could assume that the lack of inner consistency in the random layout is acceptable since it forces the users to apply other strategies. However, in chapter 4.3.1, another advantage of the use of a consistent layout (outer consistency) was highlighted. As opposed to the random layout, using the telephone layout allowed for significant learning effects.

That is, participants employed advanced strategies that they improved over time which resulted in a significant improvement of authentication performance. Additionally, the overhead created by the preparation phase of the random layout is nearly 100% as shown in chapter 4.3.1. Also the error rates caused by memorability issues did not improve by training when using the random layout as opposed to the other layouts as shown in figure 4.8. This can be attributed to the lack of inner consistency as well and is a problem for a time-sensitive task such as authentication.

Due to the lack of inner consistency, randomized systems do not cope (or work together) with the diverse strategies that users employ (like using shapes and the like). This was confirmed by the results of the final questionnaire. Especially trained users noted that they used visual or muscle memory learning strategies which are superior to plain memorizing. Thus, it has to be considered an important criterion of authentication mechanisms for public spaces to provide consistency (of some kind). This helps to improve both, performance and memorability due to muscle memory effects that can improve memorability even after a long period of non-use [51]. For PIN-entry, it is really helpful that positions can be easier remembered than distances [64].

Another interesting finding on consistency issues was on the performance of trained versus un-trained users. Even though no significant differences between those two groups were found, the results of active authentication times show that especially for the random layout, trained users

have a strong tendency to be slower. For the random layout they performed on average 0.41 sec-onds worse then untrained users. This can be explained by a higher degree of confusion caused by alternative layouts in the trained group compared to the untrained users and different memory strategies. The preparation speed of trained users on the other hand was faster for the telephone layout.