Vision-Based Multi-Touch Sensing

78 6. Technical Foundation display in the infrared spectrum. Several interpolation and smoothing techniques are discussed in the paper in order to attain an image suitable to reliably detect hands and other IR-reflective objects. Furthermore,Thinsightis capable of two-way communication with devices in front of the display via suitably modulated IR light. For example, data could be transmitted using the established IrDa protocol from the display to handheld devices. In the other direction a remote pointing device equipped with an IR LED can be used to send data (e.g., specific commands) to the display. This is even possible at the same time as sensing multi-touch input. The initial prototype was restricted in its size, so that only a small fraction of the display was active. In a later publication [IBH⁺08] this initial proof-of-concept was extended to a larger form factor where the entire visible area of a 21” LCD panel was able to sense touch input. Beyond scaling the sensing technique to a larger screen-size the authors present several interaction techniques making use of the system’s unique capabilities. Applications demonstrated included a painting application working with fingers and reflective brushes and palettes. Besides passive objects where presence detection is sufficient, also detection of object class based on shape heuristics is demonstrated (e.g., the color palette). Finally, object instance identification based on passive, reflective or active, IR-emitting markers (see Figure6.2c) has been shown.

Sidesight [BIH08] uses a similar sensing scheme enabling touch input around devices wit a small screen such as mobile phones. The idea is to avoid occluding information shown on the display while providing users with richer interaction techniques that make use of the space around the device. The system uses infrared proximity sensors embedded along the sides of the device to detect fingers and to track their position. A proof of concept image manipulation and menu selection application showcases the interactions enabled by the technology.

6.1 Multi-touch Input Technologies 79

(a) (b) (c)

Figure 6.3: (a) The VIDEOPLACE [KGH85] installation. (b) users could interact with virtual objects with their entire body. (c) Outlines of hands were segmented from the back-ground and used in the interaction.

screen and augmented with computer graphics or used to interact with virtual objects. This tech-nique and similar ones have been demonstrated in purely artistic contexts and applications such as remote communication between two VIDEOPLACES.

Wellner’s Digital Desk[Wel93] is an early example of a vision based tabletop system. The system uses frame differencing of thresholded images in order to segment fingers from a cluttered background (a regular work desk see Figure6.4a). A microphone is then used to determine when users tap the surface of the desk to indicate a “click”. Several interesting applications have been shown that merged the physical items on the desk with virtual objects projected onto the table from above, such as a virtual calculator displayed next to a printed spreadsheet. Figures from the spreadsheet may be used as input to the digital calculator by simply pointing with a pen. The Digital Deskstill serves as a source of inspiration for many recent tabletop projects.

(a) (b)

Figure 6.4: (a) Wellner’s DigitalDesk setup on the left. Example applications include a pro-jected calculator alongside a printed spreadsheet and copying printed information through simple gestures. (b) PlayAnywhere’s compact and mobile setup. Bi-Manual map manipu-lations enabled by optical flow motion estimation.

Wilson demonstrates several interesting advancements to the direct vision-based approach in thePlayAnywhere[Wil05] system, a compact portable front-projected and vision-based tabletop

80 6. Technical Foundation prototype. A commercially available short throw projector is augmented with an IR illuminant and an off-axis mounted camera equipped with a matching IR-pass filter. Several computer-vision techniques are demonstrated to detect touch/no-touch events, detect and track visual barcodes (position and orientation), track pages of regular paper as well as a novel algorithm to interact with on-screen objects through the calculation of motion-flow rather than tracking of individual fingertips (Figure6.4b). Touch detection is accomplished through IR shadow tracking. Changes in the shape of shadows casted by fingers are heuristically evaluated. As a finger approaches the surface the distance between finger and shadow decreases, ultimately the finger occludes its own shadow. This approach only works for one finger per hand. In order to enable more fluid interactions with virtual information using multiple fingers or even whole hands a novel algorithm to calculate optical flow from the captured image is detailed. Instead of tracking discrete contact points, simple statistics about changes in the image are calculated, and areas of motion at the location of virtual objects are integrated into a continuous position and orientation transform.

Thevisual touchpad [ML04] system follows a similar idea but features more robust finger tracking and an extensive set of gestures on and above the touchpad. The disparity of two, off-axis cameras is used to track fingers and calculate the height above a low-cost “touchpad” (basically a black cardboard rectangle). After background subtraction to isolate hand contours, the system uses heuristics to label fingers and calculate their 2D orientation allowing for multi-touch in-put as well as multi-finger gesture recognition. Several interaction techniques are demonstrated that enable fluid and bi-manual interaction in photo manipulating, finger painting and text input scenarios.

Indirect Multi-Touch Recognition

In the last few years a second approach to vision based sensing has enjoyed great popularity in the research community and in commercially available products: “indirect” vision systems. The key aspects here are an IR illumination scheme and a camera pointed at the projection screen from

(a) (b) (c)

Figure 6.5: (a) HoloWall illumination and sensing scheme on top. Bi-manual and multi-user interaction on bottom . (b) FTIR screen as seen by the camera. Two-handed multi-touch interaction enabled by the approach. (c) The approach scales to large wall-sized displays.

6.1 Multi-touch Input Technologies 81 behind or underneath. Touch events are not detected via directly segmenting hands and fingertips against the background and using second measurements (microphones, stereo disparity). Instead, these systems use a diffusive projection screen and the reflectiveness of skin in the IR spectrum to detect when the surface is being touched.

Matsushita and Rekimoto’s HoloWall [MR97] is an early and typical example for this ap-proach. The HoloWall is a wall size interactive display. It consists of a regular sheet of glass coated with a semi-opaque, diffuse projection screen. Behind the wall an IR light source and a video camera equipped with an IR cut-off filter are installed (only letting light pass above a wavelength of 840nm). A diffuse projection screen in front of the wall shields objects from the camera’s view. Unless an IR reflective object human skin is roughly 20% IR reflective -is brought into close vicinity of the glass. Depending on the selected threshold, objects close enough (0 to 30 cm) reflect some of the IR light back through the screen and therefore become visible to the camera (Figure6.5 a). Not only fingers but any IR reflective object can be sensed, for example objects tagged with a 2D barcode can be located and identified. The system can de-tect many contact points simultaneously and because of the camera’s location occlusion problems are mitigated.

Also relying on sensing in the IR domain, Han demonstrated a simply albeit robust scheme based onfrustrated total internal reflection[Han05] (FTIR). When light encounters an interface between media with different indices of refraction the light becomes refracted and remains within the medium with the higher index of refraction. This phenomenon is the basis for all optical light-guides. However, another material in contact with the interface between the two materials can change the index of refraction and frustrate the total internal reflection, causing light to escape the waveguide at this point.

(a) (b) (c)

Figure 6.6: (a) Multiple tradeshow visitors interacting with a custom made, multi-touch enabled tourist guide application. (b) The table in the show room at IFA 2007 in Berlin.

(c) A virtual tourist guide for the city of Berlin has been developed and deployed on the multi-touch table.

The approach uses an IR edge-lit sheet of acrylic as waveguide. Fingers touching the panel causeIRlight to escape and reflect it back through the screen where a camera equipped with an IR -pass filter detects this scattered light (Figure6.5 c + d). Simple computer vision techniques are used to detect fingertips and multiple contact points can be tracked at interactive rates across

82 6. Technical Foundation video frames. This system caught the attention of many researchers and DIY enthusiasts alike.

Due to its relative simplicity and ease of construction a multitude of devices in various form factors and for different application domains have been built based on this approach. For example, we have built our own multi-touch table based onFTIRtogether with a Berlin based company (Foresee²). The table was exhibited in a public showroom at the IFA tradeshow in September 2007 (see Figure 6.6). One limitation of the FTIR approach is the requirement for touching objects to be soft and slightly wet, such as skin. Therefore the system cannot detect objects made from hard material (e.g., a coffee mug) or detect visual markers such as barcodes. Furthermore the approach requires some form of compliant surface that allows to project virtual imagery onto the waveguide without getting in the way of theFTIReffect.