Summary

Chapter 10

154 10. General Discussion and Conclusion dominant because the hardware platform became standardized and input typically happens via mice and keyboards. For tabletops this clearly defined model has yet to emerge. One reason is that a standardization process on the hardware front has yet to happen. Many different approaches to user input sensing have been proposed, demonstrated and studied (cf. Section 6.1). One important aspect, is that tabletop hardware has become increasingly capable of sensing rich user input.

Our analysis of related work in Chapter2has identified two major interaction styles that have begun to emerge for tabletop computing. First, the use of pen- or finger-trace gestures to invoke commands and thus control the behavior of on-screen objects and the state of entire applications (Chapter2.2.2). Second, the combination of physical objects with direct-touch input (Chapter 2.2.3).

In PartIIwe have discussed and analyzed our own explorations into these interactions styles.

Chapter3details the motivations, design and study of BrainStorm. A system designed to support collaborative problem solving using several interactive surfaces embedded into walls and a table.

The chapter explores the suitability of a gesture-based interaction style as general model for tabletop computing.

Our analysis based on the criteria of physicalityandflexibility (Section3.5) has shown that the real world resemblance of virtual objects in the system helped users to understand and learn how to operate the system. However, interpreting Newtonian physics in an overly simple fashion proved to be problematic because users developed expectations that they could apply strategies learned in the real world to manipulate the virtual. However, when designingBrainstorm we only implemented a set number of gestures that could be recognized by the system. Many ma-nipulations possible in the real world were not translated into gestures. Albeit an extended ges-ture recognition algorithm would somewhat mitigate this problem and reduce the observed user frustration.

In some situations we believe there exists a more fundamental problem. When considering the rich and flexible ways we manipulate real world objects using various strategies as we see fit, it becomes apparent that this flexibility can not be matched by pre-programmed, scripted behavior. In the real world subtle changes in manipulation can have different outcomes (e.g., imagine the broad repertoire of tricks good ball players possess). Often these subtle changes are lost to a gesture recognition algorithm due to the need to reliably differentiate and classify various gestures. Finally, recognized gestures need to be mapped to commands within the application.

This mapping often happens a-priori and remains fixed. This again stands in contrast to the dynamic and flexible way humans manipulate objects in the real-world. Interaction breakdowns can occur whenever users try to interact with the system in ways not anticipated.

In Chapter 4 we explore how tangible objects on interactive surfaces might help to address the lack of flexibility observed with gesture-based approaches. Photohelix4.2is an application designed to support the co-located browsing and sharing of pictures on a digital tabletop. A physical handle is used to position and browse the entire picture collection while direct-touch interaction is used to interact with individual pictures. We had various assumptions about the benefits of using a physical handle for interaction (cf. Section 4.1). Our observations from a number of lab-based user-studies yielded interesting results but could not confirm all of them

10.1 Summary 155 (Section4.5). We could observe that the physical affordances and 3D nature of the input device allowed for richer and more flexible interactions than observed in the direct-touch condition.

These benefits however, are limited by problems with sensing and interpreting the manipulations (not unlike those in the gesture-based interface). Here the problem is that the virtual objects do not behave in a realistic manner and are not subject to the same laws as the physical interaction handle – therefore the manipulation of the virtual is again limited by what was anticipated by the designer in advance. Finally, we could not find particular strong evidence speaking in support of other assumed benefits such as eyes-free manipulations and natural support for bi-manual interaction.

Another problem that spanned across the two explorations became apparent during many hours of system usage and observations during user studies. Both prototypes,Brainstorm3and Photohelix 4, were built and deployed using the same hardware – DViT enabled Smartboards [SMA03]. This hardware platform is only capable of tracking two simultaneous contact points.

Due to the construction principle (four IR cameras – one in each corner – and a ring of infrared LEDs hidden in the bezel) objects on the surface (e.g., fingers) can easily occlude other objects.

In case two objects are aligned along one of the display diagonals they become difficult to disam-biguate. These sensing limitations caused user frustration because the system would often report wrong finger positions or not recognize fingers at all. More important the limitation to two con-tact points (which was perceived as arbitrarily chosen by many users) posed severe constraints on the interaction style. We could often observe that users – especially novice users – tried to use more than one finger, sometimes both hands at a time to interact with on-screen objects. We could also frequently observe how users tried to interact with other body parts than their fingers, for example whole hands, the edge of the hand and even whole forearms (e.g., in an attempt to stop sliding post-its inBrainstorm).

One of the main motivations for our later projects discussed in Part III was to overcome these hardware limitations to enable richer interactions with the digital more akin to the ways we manipulate objects in the real world. In Chapter6we discussed a variety of approaches to sense this richer interaction. We have discussed approaches that embed some sort of sensing electronics into the display surface itself (cf. Section6.1.1). For example, electronics that sense changes in capacitance or resistance but we have also seen approaches that embed infrared illumination and sensing optics into or behind thin form factor displays. Another popular approach are sensors based on a digital camera and computer vision algorithms. This approach is particularly popular because setups are relatively easy to construct – necessary parts, such as digital cameras and filters, are abundant and last but not least the possibility to leverage a wealth of existing computer vision techniques makes this approach flexible and powerful. For example many vision-based systems can track multiple fingertips and capture the shapes and outlines of arbitrary objects in contact with the surface such as whole hands. These capabilities promise interaction styles that mimic the ways we interact with the real world more closely.

In Section6.2we discuss our own approach to vision-based sensing of multiple simultaneous surface contacts. It works by liquid displacement inside a malleable projection surface. Our approach allows recognition of shapes and outlines of many different objects touching the surface with high precision. Our approach compliments the existing range of approaches 6.1.2. The

156 10. General Discussion and Conclusion signal is fairly unique, enabling advanced techniques such as pressure sensing or even some 3D shape reconstruction to be captured from imprints of hands and other objects. Section 6.3 summarizes our experiences from building different interactive prototypes and discusses various aspects of the systems described in the literature. It would stand to reason for tabletop computing¹ hardware matters especially; to a degree where hardware and software are interlinked and need to be considered in unison when designing and evaluating interaction styles and techniques.

Drawing both on hardware developments (cf. Chapter 6), that enable rich sensing of user interaction, and on our observations and learnings from Part II we debuted our new model for tabletop interaction in Chapter 7. This new model aims at providing more flexibility of inter-action, allowing users to appropriate an re-purpose user interface elements and apply strategies learned in the real world. Another goal was to maintain the positive aspects we could accredit tophysicality in our earlier prototypes. We demonstrated various ways to map rich input from vision-based interactive surfaces within virtual worlds where object behavior is controlled by a sophisticated physics simulation. The model enables a variety of open-ended, non-scripted in-teractions allowing users to leverage their fine-grained manipulation skills in ways similar to the real world. Allowing for and encouraging rich interactions with virtual objects, using not only multiple fingertips but novel, often in-situ designed interactions such as cupping, throwing and raking gestures. The technique allows users to interact with the virtual using multiple fingers, whole hands and physical objects as handles for interaction. We have also discussed the evolu-tion of our technique and shown how different incarnaevolu-tions could be used with other hardware platforms than the one used in our setup. Our user study and general observations have revealed that users are receptive to this new model of interaction. In general they had little problems in interacting with the virtual realm. However, kinematic more traditional interaction styles were still prevalent especially during initial contact with the system. We expect that users, as they become more familiar with the capabilities of the approach, will further draw on their real world knowledge and thus develop richer interaction strategies.

Our model makes digital tabletop interaction even more “natural”. However, because the interaction – the sensed input and the displayed output – is still bound to the surface, there is a fundamental limitation in manipulating objects using the third dimension. To address this issue, we started with a discussion of several emerging projection screen materials in Chapter 8. Materials that allow for simultaneous projection onto the surface and for imaging through the surface – thus enabling us to sense the users interactions at much greater depth than possible in standard tabletop setups.

Finally, in Chapter9we discussed our approach to creating a continuous interaction space – allowing for rich, natural interactiononthe surface andabovethe surface. We presented several ways to enable in-the-air interactions that allow users to interact with 3D data (rendered on a 2D display) in a as-direct-as possible fashion without the need for user instrumentation. The chapter introduces two novel, vision-based back projection tabletop setups that allow for sensing of on and above the surface interaction. Existing and new computer vision techniques have been proposed to estimate the position of hands in 3D. Our technique demonstrates how this can be achieved using an electronically switchable diffuser, a monocular camera, IR illumination and

1but possibly for all areas of non-desktop HCI