Demonstration and Evaluation

For demonstration and evaluation, we conducted a laboratory study with 12 participants. We implemented the usablilityWatch framework in the exemplary smartwatch application smartActivity, which provides collaborative support for employees in industrial workflows (Zenker / Hobert 2019, study 1). For that, an employee can receive, process, and return activities according to a defined workflow.

The application is composed of four screens: (1) a welcome page at the start of the application (welcome), (2) a list of assigned activities as illustrated in Figure 53 (activitylist), (3) a notification screen that informs a user about incoming activities that can be accepted or postponed (notification) and (4) a detail screen for a selected activity with a list of possible next steps according to the workflow as illustrated in Figure 55 (activity). During the study, the participants took the role of a technician who is responsible for several computer-operated milling and punching machines and traversed a scenario including various machine alerts and requests of a quality assurance department. After the scenario

176 Studies: Usability of Smartwatch-based Information Systems

the overall usability and usability problems that occurred during the operation of the smartwatch application. This enables us (1) to collect and analyze realistic data with usabilityWatch and (2) to have insights about the usability problems of real users. Matching both assessments allows us to evaluate the utility of the developed framework.

After conducting the laboratory study, we asked the participants to provide us feedback about usability.

On the one side, the participants highlighted several positive aspects regarding usability, like the clear arrangement of the application, intuitive use, a low number of touches to process activities, and fast loading times. On the other side, several problems were stated. Concerning lists, the participants mentioned, “the overview of activities automatically jumps up again very quickly, which makes the selection difficult” (participants 5 and 6, 8, 9, 10) and “the selection of the possible next work steps on the detail screen is very small” (participants 5 and 2). Both comments reveal serious problems since the list at the activitylist screen jumps to the top every five seconds whenever the list is updated due to a messy implementation, which disturbs the selection of the desired element and requires another swipe.

usabilityWatch detects both problems utilizing the swipe-to-touch metric shown in Figure 54.

Figure 54. Swipe-to-touch ratio for the different screens

The swipe-to-touch ratio outlines a very high value for activitylist, indicating that for each selection, many swipes are required. The list at the activity screen also triggers a high value that is more related to the small size, which can be proved by the high number of unsuccessful swipes in the vicinity of the list. The incomprehensible list and inappropriate swipe area smells are reported accordingly since the optimal sequence is to swipe to the element and touch it, resulting in a value of 1. Another issue is described as

“the back button was only half displayed and therefore hard to reach” (participants 4 and 5, 6, 7, 8). This can easily be seen in Figure 55 and is caused by an unintended shift of the whole layout of smartActivity to the bottom (small white area at the top). usabilityWatch reports the unresponsive element smell for touches close to the button. In combination with the heat map given in Figure 55, this issue can be detected.

0

8,04 0,61

5,81

welcome activitylist notification activity

swipe-to-touch ratio

Figure 55. Touch heat map of the activity screen

As the last commonly listed usability problem, we got “faulty touches quickly lead to unwanted entries”

(participants 4 and 2) and “I like to have more feedback that an action was executed after I touched a button” (participants 3 and 8). So far, there is neither clear feedback that an action succeeded nor a confirmation prompt if an action should be performed. This leads to user behavior in which the action is checked or restored subsequently. The framework reports the missing feedback and missing confirmation smell due to a looping index of 3.2 and 2.7, respectively.

Summarizing, usabilityWatch can identify the reported usability problems within the recorded data.

Some of the defects can be found completely automatically. For others, the usability smells are just an indication and have to be combined with other (visual) metrics to conclude the defect.

6.5 Discussion and Conclusion

In this paper, we presented a usability framework for smartwatches. Inspired by the design science research method (Peffers et al. 2007), we illustrated a problem-orientated research design. We first identified and described usability methods that are recently used for mobile devices (RQ1) since the usability analysis of smartwatches is a research gap. We formulated objectives and inferred requirements based on the conducted structured literature review and considered the unique characteristics of smartwatches. We presented the usabilityWatch framework composed of a smartwatch component and web backend (RQ2). It provides easy integration into a smartwatch Wear OS application, automated logging of user interactions, visualization of the collected data with, e.g., heat maps, and the analysis of usability defects. For that, we elaborated a list of usability smells suited for smartwatches. Finally, we proved in a demonstration and evaluation that the framework could find similar usability defects as the participants of a laboratory study for an exemplary smartwatch application.

178 Studies: Usability of Smartwatch-based Information Systems

There are some limitations to our research study. Since usability is a well-researched topic, the related literature is extensible, and we cannot claim our review to be complete. Second, we tested the framework with just one exemplary smartwatch application within an exemplary scenario. We are planning to do tests with more applications in order to improve the modularity and simplicity of integration of the framework. Furthermore, we want to extend the list of usability smells and like to optimize the thresholds for the existing smell metrics towards realistic values by expanding the practice. Though, the application of the framework requires a proper interpretation of the results in order to benefit from the generated insights and to identify false positives that may occur in the automated analysis. In addition, the user of the framework has to be aware of metrics like the swipe-to-touch ratio, which can be misleading whenever multiple scrollable elements appear on the same screen (unlikely due to small screen size), or the screen itself can be scrolled. Since hardware buttons or digital crowns are noted as very pleasant, these should also be included in the corresponding scrolling metrics, which remains a complicated problem due to heterogeneous hardware and software widgets.

Nevertheless, we verified the utility of usabilityWatch in a realistic scenario and contributed to practice and research. The developer of smartwatch applications can benefit from usability insights in order to reduce a user’s cognitive load and to improve their applications. This can easily be done by analyzing the user’s interactions, and no time-consuming and expensive qualitative studies like laboratory tests are required. For practice, we created an applicable software solution for targeting automatic usability analysis on smartwatch devices in order to support developers. Within the research domain, we reviewed recent approaches and methods, modified and complemented them according to the unique characteristics of smartwatches covering main aspects of the PACMAD model. This transfer of methods forms the foundation for future studies for usability analysis on smartwatches.

B

C Contributions

The presented studies aim at investigating the utilization of smartwatch-based information systems in the corporate context. This extensive objective has been subdivided into the three research complexes (I) smartwatch-based information systems supporting mobile employees executing manual work, (II) smartwatch-based information systems at the office workplace, and (III) usability of smartwatch-based information systems. This scope provides a multi-faceted and holistic view on the design principles of mobile information systems in the digital transformation of the workplace and, in particular, the utilization of smartwatch-based information systems in the corporate context.

As illustrated in Figure 56, in section C.1, the contributions of the individual research articles founding this cumulative thesis are briefly reiterated and related to the overarching meta-research questions formulated in section A.2. Subsequently, in section C.2, the practical and theoretical implications of these findings are discussed, and the contributions for research and practice are outlined. Finally, the methodical and contextual limitations are emphasized in section C.3 to motivate future research avenues based on the presented thesis in section C.4.

Figure 56. Structure of thesis’ part C presenting the research contributions A: FOUNDATIONS

B: STUDIES

C: CONTRIBUTIONS

Findings Implications Limitations Future Research

1 Summary of Results

This section surveys the findings regarding the meta-research questions outlined in section A.2. For that, the meta-research questions are reiterated, and the contributions are summarized. As illustrated in Figure 50, these results were developed based on six studies presented in part B of this cumulative thesis. The studies refine and address the three meta-research questions described and outlined in section A.2. They are arranged in the three overlying research complexes. Hence, the research complexes constitute and cover the overarching domain of this thesis about the utilization of smartwatch-based information systems in the corporate context from a design perspective.

Figure 57. Research progress from objective to findings

The central findings (F) of this theses are:

F1: The utilization of smartwatch-based IS can facilitate mobile employees executing manual work with an incidental and hands-free exchange of information, collaboration, workflow guidance, and context-awareness, decreasing the cognitive load and increasing the efficiency of work (Zenker / Hobert 2019, study 1; Zenker et al. 2020b, study 2; Zenker 2020, study 3; Zenker et al. 2020a, study 4).

F2: According to the unique characteristics of smartwatches, smartwatch-based IS should comply with the design principles formulated in the nascent design theory for successful utilization in the corporate context to fully profit from the potentials and overcome the device-related limitations (Zenker et al. 2020a, study 4).

UTILIZATION OF SMARTWATCH-BASED INFORMATION SYSTEMS IN THE CORPORATE CONTEXT RESEARCH COMPLEX I RESEARCH COMPLEX II RESEARCH COMPLEX III

STUDIES

1 2 3 4

STUDIES 5

STUDIES 6

F1 F2 F3 F4 F5

META-RESEARCH QUESTIONS META-RESEARCH QUESTIONS MRQ 2

META-RESEARCH QUESTIONS MRQ 3

MRQ 1 1.1 1.2 1.3

182 Contributions: Summary of Results

F3: Before utilizing smartwatch-based IS, technical, organizational, individual, and environmental prerequisites have to be considered formulated in the Smartwatch Applicability Framework (Zenker et al. 2020b, study 2).

F4: Smartwatches can be utilized for stationary work due they are directly connected to the employees’ body and thus can demand the employees’ attention as well as obtain sensor information, for example, for haptic notification of messages through vibrations or corporate health promotion programs (Wesseloh et al. 2020b, study 5).

F5: The usability is a crucial success factor of smartwatch-applications and can automatically be analyzed for improvement with the developed usabilityWatch framework (Zenker / Hobert 2020, study 6).

The meta-research questions for each of the research complexes covering the different aspects of this cumulative thesis can hence be answered as follows.