Results and Discussion

One rather problematic issue with diary studies (although seldom reported in the HCI literature, for some reason) is the increased chance for participant drop-outs. In many cases, participants are not really aware of the burden of partici-pating in a study for several weeks or of keeping a diary every day or several times a day. Therefore, they underestimate the effort needed and after a few days they realize that they are unable to further participate in the study. In our case, three of the eight participants had dropped out by the beginning of the second week. There are several possibilities for decreasing the chance of such drop-outs. First, it is possible to split the financial reward into parts so that par-ticipants are encouraged to continue in order to get the full amount. Second, the relationship between researcher and participant plays a large role. If the re-searcher is helpful and supportive all the time and also shows that he or she cares about everything the participant records, this can help a great deal in in-creasing the intrinsic motivation “to be a good participant.” However, there is also a potential downside to this: Some participants may misinterpret “being a good participant” with “saying what the researcher wants to hear.” Therefore, it is essential that the researcher does not take sides and emphasizes that he or she does not have a preferred view on the study subject (e.g., which of the tools is “supposed” to perform better).

Our high-level results showed that both visualization techniques worked rea-sonably well; two participants even asked if they could continue to use them despite some technical bugs that limited the usability. Switching to the second system after one week was perceived as quite an easy undertaking, mainly due to the similar interaction concept applied in both systems when accessing an information object. We found that the HyperGrid was better suited to searching for one specific object, while the HyperScatter provided ways to compare infor-mation objects and to look for interesting clusters and correlations. In both cas-es, users took advantage of the browsing possibilities and especially liked the integration of external web services such as youtube.com and imdb.com. Be-cause of this function, they could focus on a single system and did not have to constantly switch between webpages with different interfaces. Interestingly, two

of our participants came to the conclusion that it would be beneficial to combine the two approaches (HyperGrid & HyperScatter) into one system; the other three were also positive about this possibility when asked.

Figure 25: Relationship between diary entries (left bars/participant), session numbers (right bars/participant) and usage duration (y-axis + dotted lines)

Examining the detailed results from a methodological perspective, we can see several things. First, Figure 25 neatly displays the relationship between diary entries and interaction logs. For example, participant 6 (VP6) completed a total of 23 diaries but only 12 logged sessions were recorded. Not all of them could be explained by multiple diary reports per session. While people marked “ex-plorative search” as their purpose only in nine diary reports, the possibility for browsing was reported as a major benefit in the interviews. The diary reports also revealed that people often used the integration of external web sources (in 17 of 39 logs for the HyperGrid and 10 of 24 for the HyperScatter), which could be confirmed by analyzing the interaction logs. During interviews, participants stated that they really appreciated the possibility to access external data sources such as Google or imdb.com directly from the application. People also

stated that they liked the possibility to freely assign metadata to a column or the axes. The log files confirmed that people not only liked this function but also used it heavily (in 25 out of 47 sessions total).

Overall, the triangulation paid off extremely well. It became quite obvious that the three data sources each revealed a different kind of information. The inter-views revealed overall impressions of the participants and allowed them to elaborate on certain problems and express suggestions for improvements in detail. The diaries, on the other hand, allowed us to get a better understanding of how and why the system was used, with detailed information about purposes of usage and specific problems and how these affected the overall rating of the system at that time. Furthermore, this information provided helpful during the interviews for elaboration. The interaction logs were mainly used to check how the qualitative expressions were reflected in real usage of the system and thereby proved to be very valuable to judge compliance, i.e., whether there was some “substance” to the talking (to see whether people really used a function they praised in an interview, for example). However, there was still room for im-provement. When evaluating software running on a PC, it seems preferable to directly integrate the diary functionality in digital form. In this way, participants could be reminded to fill out a diary whenever they closed the system. In addi-tion, an integrated on-demand diary function could be used for usability issues.

This could also trigger a screenshot, which would help during the analysis pro-cess. Unfortunately, integration of such an electronic diary directly into an appli-cation requires development resources that might not be available. A pen and paper diary proved to be a very cost-effective and informative solution.

The PocketBee approach presented in the next section could also be a very effective solution.

3.4 PocketBee – A Multimodal Diary and ESM Tool for