Tangible auditory interfaces : combining auditory displays and tangible interfaces

(1)

Tangible Auditory Interfaces

Combining Auditory Displays and Tangible Interfaces

Dissertation

zur Erlangung des akademischen Grades Doktor der Ingenieurwissenschaften der Technischen Fakultät der Universität Bielefeld vorgelegt von Till Bovermann am 15. Dezember 2009

(2)

(3)

Acknowledgements

I would like to thank my supervisor Thomas Hermann for all the freedom, support and lively discussions. It is a pleasure to work with him; without his active support this work would not have been possible. To my co-workers and friends in the Ambient Intelligence Group – Florian Grond, Tobias Großhauser, Ulf Großekatthöfer, Sebastian Hammerl Christian Mertes, Eckard Riedenklau and René Tünnermann – warm thanks for inspiring and stimulating collaboration and for making this thesis possible. Although some of you are not co-authors of the papers included here, your ideas and comments certainly have had a great influence. I would also like to thank all members of the Neuroinformatics Group, Bielefeld for providing a pleasant workplace and a friendly atmosphere. Due to the patience and kindness of Helge Ritter, head of group, I had much freedom to work on my projects. I thank him for his ongoing support. Thanks also to friends, colleagues, and personnel at the CITEC Center of Excellence and the Institute of Electronic Music and Acoustics for creating such vibrant and interesting environments to work in. I am especially grateful to Alberto de Campo for his active and thoughtful support in many circumstances. I learned a lot. I want to thank the Central Lab facility of the CITEC for their support in the production of the ChopStix system. I am also greatful to the people directly involved in the production process of ChopStix: Jan Anlauff, Holger Dierker, Florian Grond, Felix Hagemann, Simon Schulz, René Tünnermann, and Sebastian Zehe. Without you all, it would not have happened. I want to thank René Tünnermann, Florian Grond and Thomas Hermann for their valuable contributions to Reim. I would also like to thank Bodo Lensch and the Animax, Bonn for the opportunity to show Durcheinander in their performance space. For linguistic and language support during writing, I would like to thank Katrin Kaup.

I especially want to thank Claudia Muhl and Sven Nieder for their support on a professional, yet also private level. For both their professional support and general friendliness, I would like to thank Christof Elbrechter, Marianne Egger de Campo, Jonas Groten, Oliver Lieske, Lucy Lungley, Julian Rohrhuber, Katharina Vogt, Arne Wulf, and the inhabitants of the red house in Graz. Many thanks also to all of you I did not mention personally, who have provided feedback on this work by commenting and criticising, discussing and contributing ideas.

Finally, I am deeply grateful to my family and I dedicate this work to my parents and grandparents for their devoted support given to me throughout my life. To Ulrike, thank you for your support, hope, trust, joy and everything else.

During the work on this thesis, I have been employed by the Neuroinformatics Group and the Ambient Intelligence Group of Bielefeld University as well as by the Institute of Electronic Music and Acoustics, Graz. This work has been partially supported by the DFG through the SFB 673, and by the Center of Excellence, CITEC, Bielefeld University. I would like to thank the European Environment Agency and wunderground.com for their kind provision of near realtime ozone, respectively weather data.

(4)

(5)

List of Figures

1.1. Reim, a Tangible Auditory Interface for auditory augmentation. . . 2

1.2. AudioDB, a Tangible Auditory Interface. . . 3

2.1. Similarities between the data mining workflow and the typical handcrafting workflow. . . 13

3.1. Flowchart of Exploratory Data Analysis. . . 16

3.2. Fields related to Exploratory Data Analysis. . . 19

3.3. Data transcribed from a digital storage to human perception has to go through several layers. . . 19

3.4. Schematics for data representations. . . 22

4.1. Chatting people at the Grand Opening of the CITEC Graduate School in July, 2009. An example for Human-Human Interaction.. . . 26

4.2. Finger naming. . . 26

4.3. Video stills in which Andy Goldsworthy explores leaves while crafting an art-piece [riv]. He sits under the tree from which the leafs are originating, assembles the artwork, and places it back to the tree. The second row shows stills from the sequence that is analysed in the main text. . . 27

4.4. Difference of continuous and discrete variables as they appear in analogue and digital systems. . . 29

4.5. A slide rule. In its current configuration it can be used to read off all results for f (x) = πx. The hairline on its sliding window indicates that it is used for x = 1.16. . . 30

4.6. A mechanical planimeter by the Gebrüder HAFF GmbH. See Section 1.1 for c information. . . 31

4.7. Venn diagrams for Reality-based Interaction. . . 35

4.8. The translated text of the email survey on the evaluation of Human-Computer Interfaces. . . 37

5.1. As a side effect of physical constraints, the rotation of four audio-loaded cubes results in Shepard-Risset glissandi. . . 52

5.2. Examples for Tangible Interface Objects. . . 53

5.3. Object reactions in Tangible Interfaces.. . . 54

5.4. Cuboro example setup. . . 56

5.5. Cuboro cubes as an example for haptic symbols. . . 57

5.6. The second iteration of the tDesk system. Images courtesy of Eckard Rieden-klau. . . 61

5.7. Lentil-shaped objects on a surface. Prototypical objects for the introduced object-class Grains and their manipulation. . . 61

(10)

List of Figures

6.1. Design study of a dynamic information stream visualisation. . . 68

7.1. Information flow in a Tangible Auditory Interface. . . 78

7.2. Controller-based Object Use (left) vs. Data-Object Identification (right): The captured states of the objects are either used for real-time control of program parameters, or the users identify them directly with the referenced digital representation. . . 80

9.1. The MoveSound Logo. . . 87

9.2. Hardware surround panning interfaces. . . 88

9.3. Software surround panning interfaces. . . 88

9.4. Design study of the MoveSound environment. The tangible controller is located in the centre of the loudspeaker ring. . . 89

9.5. Tangible input devices for MoveSound. . . 90

9.6. Graphical User Interface of MoveSound in full/reduced mode. . . 91

9.7. MoveSound’s modules. . . 91

9.8. Source Selection: Ligeti and Dota are set active. . . 92

9.9. UML diagram of the MoveSound Model and its connection to the Sound Rendering. . . 95

9.10. UML diagram of the Human Interface Control and its relation to the model. 95 9.11. UML diagram of Status Graphics. . . 97

9.12. Video stills of the MoveSound interface from the video demonstration on the DVD. . . 98

9.13. Image of the screenplay as it was part of the MoveSound survey. . . 99

9.14. MoveSound manipulation of Participant 4 during the 4th challenge. The blue line represents the “Playground” source, purple “Footsteps”, yellow “Airplane”, cyan “Table Soccer”, and green “Radio”. Playback of recorded material is indicated by a red overlay. For further explanation, see main text. . . 101

9.15. Design concept of ChopStix. . . 104

9.16. ChopStix Tangible Controller is a plate with three sinks that are made to okace glasses. Each of the sinks are identified with one soundscape. Placing a glass activates the soundscape’s playback with a spatial emphasis determined by Stix. . . 105

9.17. The resulting design study of a ChopStix Interface mock-up session. . . 106

9.18. UML diagram of ChopStix-relevant classes and their dependancies. . . 107

9.19. A rendering of the location of the ChopStix Interface in a room. The spatial sound display is realised by the ring of loudspeakers on the ceiling. The long-term aspect in the control – near real time data streams change their values on an hourly basis – requires the interface to be constantly available, but not to be disturbing. Therefore it is placed near the edge of the used multi-speaker setup. . . 107

9.20. Computer vision-based design of CTI. . . 108

9.21. First prototype of the Hall-effect-based design of CTI. . . 109

9.22. Circuit diagram (left) and board layout (right) of the Hall-effect sensor based implementation of CTI. . . 110

9.23. Setup for Hall-effect sensor data acquisition. The setup was used to calibrate the Hall-effect sensors. . . 111

(11)

List of Figures

9.24. Equal power panning as it is used in ChopStix. The x-axis represents the sound position in the ring. The green to blue curves are the normalised amplitudes of the loudspeakers, whereas the red to yellow curves represent

the gain for each channel in dB. . . 112

9.25. Schüttelreim: One of several possible Reim setups. . . 116

9.26. General model of Reim-based auditory augmentations. . . 117

9.27. UML diagram of Reim-related classes. . . 121

9.28. Usage scenarios for Reim Stethoscope. . . 123

9.29. Video stills from a Paarreim exploration session. It can also be found on the DVD. . . 125

9.30. The hardware setup used by Participant 1. The transducer was attached to the external video adapter of her laptop. This made it easy for (dis-)assembly, since she only used WetterReim at her workplace, but carried her laptop with her. . . 129

9.31. The weather conditions for each participant during the WetterReim study. . 130

9.32. AudioDB, a system for sound sample representation. . . 132

9.33. Technology overview of AudioDB . . . 134

9.34. Overview of the AudioDB software and its interdependencies. . . 135

9.35. Example layout for the transition from Node Mode to Cluster Mode. . . 136

9.36. The transition from Node Mode to Cluster Mode viewed from the object that flips. Depending on the state, it either collects all sounds from objects nearby, or distributes its sounds to the surrounding node-mode objects.. . . 137

9.37. Conceptional and real TDS layout. . . 145

9.38. TDS schemata. . . 148

9.39. Examples for data exploration with TDS. The green data objects are excited by moving the plane T . . . 150

9.40. Jonas Groten practising with JugglingSounds. . . 152

9.41. Video stills of a JugglingSounds performance 2007 in Graz. The correspond-ing video is part of the accompanycorrespond-ing DVD. . . 155

9.42. Sonification and feature extraction strategy. . . 157

9.43. Components of JugglingSounds. . . 160

9.44. Video stills from the presentation of a prototype of Durcheinander at Animax, Bonn in late 2007. The corresponding video is part of the accompanying DVD.162 9.45. A 2D-plot of a two-dimensional artificial data set and its corresponding dendrogram. The red line indicates a specific clustering that defines the shape of the data items in the scatterplot. . . 163

10.1. Design and implementation of TUImod elements. (a) TUImod modular design. (b) TUImod object with all PF elements covered by example elements for UI and CI. . . 170

10.2. TUImod objects with different PF elements. (a) magnetic, (b) clip-in, (c) saw, (d) cube, (h) two-sided. Images (e)–(g) show examples of the use in the tDesk environment with front-projection.. . . 171

(12)

(13)

1. Introduction

The world around us is full of artificially gathered data. Upon that data we draw conclusions and make decisions, which possibly influence the future of our society. The difficulty hereby is not the data acquisition – we already have plenty – but our ability to process it [Gol97]. Arising from this circumstance, at least two demands for data preparation can be identified: gaining appropriate attention depending on the data domains’ nature and the

users’ needs [Gol06], and finding representations that truly integrate data and algorithmic

functionality into the human life-world. I argue that a thoughtful data representation, designed in a way that it benefits from the various aspects of the human’s

being-in-the-world [Hei27], i.e. the complex interplay between the human and its environment, can fulfil

these requirements.

Our awareness of being-in-the-world is often caused by the intensiveness of multi-sensory stimuli. The experience of walking through a cavern, feeling a fresh breeze that contrasts with the pure solid rock under the feet, hearing echoes of footsteps and water drops serves as a good example for this: All the simultaneously sensed impressions make us aware of our body and its integration into the cavern. The lack of a single sense, or a misleading impression would change the overall impression. In traditional computer-related work, many senses such as Hearing, Taste or Smell are underused. Historically developed paradigms such as the prominent Graphical User Interface (GUI) are not able to fully embed the user into the information to be mediated. Possible explanations for their nevertheless widespread

use should be searched more in their (historically developed) technical feasibility [Sut63],

rather than in usability and user-oriented simplicity. For about the last ten years, though, there has been a shift towards better representations of computer-based processes and abstract data, which try to close the gap between the users’ reality and the abstract environment of data and algorithms. These fields take advantage of both display and controlling strategies by primarily incorporating other modalities than vision. Currently, these systems take advantage of either alternative display technologies such as auditory or haptic displays, or advanced controlling approaches like multi-touch or tangibility. I argue that the already promising achievements will be even better, if auditory and tactile displays are complemented by direct controlling approaches. Furthermore, I believe that

their combination will unfold the true potential of interfacing technologies [Roh08]. In this

thesis, I present such an approach with Tangible Auditory Interfaces, a combination of Auditory Displays and Tangible Interfaces.

I herein argue that haptic feedback as well as rich controlling and display possibilities are essential for a sufficient interface. While Tangible User Interfaces provide rich and, at the same time, direct control over digital data, sound and therefore Auditory Displays are widely recognised as very direct and flexible in their dynamic allocation of user attention. A combination of both is considered promising, featuring an informational-rich interface where users can select, interpret and manipulate presented data based on their excellent structure recognition abilities. The aim of this thesis is therefore to provide insights into

(14)

1. Introduction

the fundamental research on a combination of Tangible and Auditory Interfaces as integral systems.

Dealing with data means to operate on it and to experience it. The best computing

Interfacing

algorithm is useless without an appropriate system to mediate and control its behaviour; data has to be fed into it, and the results have to be shown. Interfacing between data and algorithms on the one hand – abstract notions without a fixed physical representation – and the human reality – where physicality plays an important role – is a difficult venture. Many aspects like transparency and comprehensibility as well as perceptual and technical considerations have to be taken into account. Two areas, Human Computer Interaction (HCI) and Interaction Design (IxD) focus their research on this field. Although focused on the same – the interface between human and computer – the research fields are inherently different: While the IxD community takes care of the design and production of interfaces, HCI research is mainly about its analysis. In other words, while HCI primarily deals with the analysis of existing systems regarding their performance in various contexts, IxD is accounted for their design and production.

Figure 1.1.: Reim, a Tangible Audi-tory Interface for audi-tory augmentation.

For about the past ten years, both fields have

ex-perienced a change of their focus [LS04]. Originally

oriented almost exclusively towards screen, mouse and keyboard as the central human computer inter-faces, recent investigations cover also other interfac-ing techniques as they were developed for

augmented-or virtual-reality systems [HHH+08] [KHS89].

Ex-amples are tangible and multi-touch technologies: due to their interfacing capabilities, they provide a deep integration of the control of algorithmic systems into the user’s environment. Multi-touch technology hereby frees users from manipulating complex digital systems with only a single pointing device (e.g. a mouse) by means of fingertip tracking. Tangible In-terfaces go one step further by lifting actual parts of the digital data representation into the users’ reality, making them graspable and

manipu-latable just like other physical objects [Ish08]. Also, display technologies other than vision

were developed, leading to active discussions on multi-modality and its influence on display

technologies [KACS03] [LCS03] [OCL04].

The young field of Tangible Interfaces (TI) picks up the concept of a more direct interfacing

Tangible Interfaces

between users and computers that was not present in traditional GUI-based designs [UI00].

To achieve this, the community around TI introduced physical objects to the virtual world of the digital, fully aware of all their interaction qualities, but also of their ubiquitous limitations evolving out of their embedding in the real world. Tangible Interfaces exploit real world objects for the manipulation of digitally stored data, or – from a different point of view – enhance physical objects with data representations (either measured or rendered from artificial algorithms). This, on first sight very simple idea, turns out to be a powerful approach to the conscious development of complex, yet natural interfaces.

The user experience of a Tangible Interface is dominated by the incorporated physical objects. Their inherent natural features of which users already have a prototypical concept are valuable for the designer and make it easy to develop interfaces that are naturally

(15)

capable of collaborative and multi-handed usage. Even further, the usage of tangible objects implicitly incorporates a non-exclusive application, so the system designer does not have to explicitly implement it.

Among other reasons, this shift was possible due to the availability of cheap sensor hardware. Former arguments against custom-built physical interfaces for software systems such as expensiveness, and lack of hardware liability were weakened and countered by the fact that the presence of dedicated physical properties such as position or extent of physical artefacts, together with the natural human knowledge about these properties even support the design

of user-centred interfaces [PI07].

Not only research and perception of input technologies have changed over the last century, Auditory Displays

also the research in display technology has been taken a step further, discovering also non-visual modalities. The former focus on primarily visual displays has broadened to cover

auditory [Kra94] and haptic cues [MS94] [BD97]. Particularly Auditory Displays (AD) have

seen a strong uplift, since they support the human’s excellent ability to perceive structures in a very different way than it is possible with the predominant visual display techniques. It turned out that sound rendering processes provide a way to display a reasonable amount of complexity. Therefore they are suited to display high-dimensional data. The benefit of sound, contrasting to other modalities besides vision, is that it can be technically rendered in a reasonable quality and spatial resolution.

The human perception of sound differs from visual perception. The human developed other structure detection and analysis techniques regarding the auditory sense, making it sensible for different structural information than they are recognised in the visual domain. These are among others timing aspects like rhythm, and the native support of time-based structures. The combination of Visual and Auditory Displays makes it possible to get a more complete interpretation of the represented data. Thus, the provision of the same data by more than one modality makes it possible to extend the usage of human capabilities in order to reveal

the data’s structure. Auditory Displays also natively support collaborative work [HBRR07],

and allow for subconscious and ambient data representations [KL02].

Figure 1.2.: AudioDB, a Tangible Auditory Interface.

While both Auditory Display as well as Tangible Tangible Auditory

Interfaces

Interface research are highly promising as individual fields of research, a combination of their techniques and experiences introduces valuable cross-links and synergies beneficial for both. In Tangible Auditory In-terfaces, the tangible controlling component – focused on input capabilities of a system – is complemented by a display technology that supplements the existing haptic and visual cues. This combination forms an integral system for interactive representation of ab-stract objects like data or algorithms as physical and graspable artefacts. The primary modality of Tan-gible Auditory Interfaces for information and data mediation, therefore, is sound. During this thesis, I will point out that key features of Tangible Au-ditory Interfaces are interfacing richness, directness and flow, multi-person– as well as ambience and augmentation capabilities, and interface ergonomics. Furthermore, I state that specific requirements of Tangible Interfaces induce a

(16)

1. Introduction

certain gestalt or characteristic of the TAIs Auditory Display design and vice versa. Audio is a natural affiliate to physical objects; most of them already make sound, e.g. when touched or knocked against each other. Coming from the other direction, Auditory Displays in

general and Sonification in particular profit from a direct control interface [HH05]; especially

a highly interactive tangible input system allows a very close interaction loop between user and data representation.

1.1. Remarks

The following abbreviations will be used in this thesis:

Abbreviations

AD Auditory Display AR Augmented Reality

EDA Exploratory Data Analysis GUI Graphical User Interface HCI Human Computer Interaction

HID Human Interface Device protocol standard IxD Interaction Design

L1 – L5, R1 – R5 Finger indices (see Figure 4.2for details)

LOA Level of Abstraction MBS Model-Based Sonification RBI Reality-Based Interaction RFID Radio-frequency identification

SETO SuperCollider Environment for Tangible Objects TAI Tangible Auditory Interface

TDS Tangible Data Scanning TI Tangible Interface

TUIO Tangible User Interface Object Protocol Ubicomp Ubiquitous Computing

VR Virtual Reality

All photos and images in this thesis are copyright by Till Bovermann. Exceptions are

Figures

Figure3.4(a) reprinted for exemplification from Auditory Display [Kra94], Figure3.4(b)

reprinted for exemplification from Science By Ear: An Interdisciplinary Approach to

Sonifying Scientifc Data [dC09a], Figure4.3that is a reprint of video stills taken from the

DVD Rivers and Tides: Andy Goldsworthy working with time [riv], Figure 4.6, printed

by permission of the GNU Free Documentation License1, Figure5.6 courtesy by Eckard

Riedenklau, Figure9.2 and Figure9.3 product images from various companies of digital

audio workstations, and Figure 9.44 that is a collection of stills from video footage by

Henrik Niemann.

1

(17)

1.2. Document Structure

Code listings are intended to exemplify specific aspects, and are therefore optimised Code listings

for readability rather than computing time efficiency. They are written in

SuperCol-lider [McC02] [WCC09], if not otherwise noted.

The accompanying DVD2 contains a file hierarchy according to the chapters in this thesis. DVD

Videos and additional material are sorted into these folders. All videos are encoded using

the H.264 standard, so that they should play back with any recent software video player.3

1.2. Document Structure

This work is divided into two parts. While the first part deals with theoretical deliberations on Tangible Auditory Interfaces and related fields, the second utilises the gained knowledge for practical investigations.

PartI is structured as follows: In Chapter2, I provide an introduction to data on which

all the other covered fields will be based. It describes the current role that data plays in

our society, gives examples for its structure, and presents a formal definition. Chapter 3

then covers Exploratory Data Analysis (EDA), a research area that is part of data mining. EDA explicitly deals with the quest to find techniques and methods for exploring data for new information. The chapter focuses on data representation to enforce a variety of different data-driven experiences in order to help users to derive structural information from the data under investigation. Working with data in an explorative manner particularly means to use interfacing technology to bridge the gap between the virtual data environment and the human perceivable reality. The research fields Human-Computer Interaction and

Interaction Design which will be described in Chapter 4. Their particular intention is

to design and analyse the interface between human and machine, i.e. our reality and the digital realm of automated data processing. A relatively new field into which HCI and IxD investigate are Tangible Interfaces. Since they play an important role in this thesis, they are

described and discussed in Chapter 5, in which I also develop considerations that I found

essential for a theory-building for Tangible Interfaces in general and Tangible Auditory Interfaces in particular. To get the processed information derived from the underlying data

to the user, display technologies are needed. They are examined in Chapter 6. Apart from

a brief overview of the widely known visual display technology and its possibilities for data representation, I focus on Auditory Displays and their potential especially in closed-loop

interfaces. This is followed by Chapter7, in which the integration of Tangible Interfaces

and Auditory Displays into Tangible Auditory Interfaces is introduced. Prior to a list of its key features, I will formulate a first proposal for a definition of Tangible Auditory Interfaces and its design guidelines.

Part II is structured as follows: Chapter 9 describes applications that I developed to

support the ideas behind the TAI paradigm, illustrating their usefulness and potential

regarding the features introduced in Chapter7. An brief overview of these applications is

given in Chapter 8. Chapter10introduces a software and a hardware framework that I

have developed in co-operation with others over the last years to support the design and especially the implementation of TAIs. The thesis closes with a summary of my findings

and an overall conclusion and outlook on further work in Chapter 11.

2

The content of the DVD can also be found athttp://LFSaw.de/tai.

3

(18)

(19)

Part I.

Interfacing Digital Content with

Auditory and Physical Devices

(20)

(21)

2. Data and its Central Role in Digital

Environments

Data is omnipresent in our society. Everything is measured, and the resulting data is collected, analysed and used to help to control many aspects of the industrial world. One movement, enforced by the artificial need for speed and accuracy, is the shift of communi-cation to use digital media. This trend turns communicommuni-cation away from the traditional face-to-face chat via low-tech media towards the use of digital media systems such as Voice over IP, internet-based chat, and electronic mail clients. This digital media, originally invented to be used for distance communications is increasingly used for communication in local settings, e.g. to distribute various information through an office.

It is, however, not only our communication that changed to the digital realm; our society itself depends increasingly on both digital data acquisition and automated analysis to gain new information. Essential application fields are for example marketing strategies, or the automated production of almost all devices and tools we use. In these applications, data on the production process has to be computed and analysed in both realtime and off-line to control the automated production units and monitor the build quality. The acquired data is digitally represented; coded either in numerical values or text. It often is composed into complex, non-linear structures in order to reflect the measurements and their interrelations.

Data is not only processed by machines, but serves also as a resource for human analysis. Data as a human

resource

The information extracted from that data is an integral part of our life that is used in both active and subconscious forms to understand and decide on living– and marketing strategies.

2.1. Examples for Common Data Domains

As a very abstract and general notion, data embraces a broad range of different shapes and meanings. Data taxonomy would surely differentiate between the two independent variables Data Structure and Data Semantics. While the semantic of data is particularly valuable for interpretation and reasoning, its structural appearance significantly influences which algorithmic exploration and analysis techniques may be applicable to get insights

into its semantical meaning. Geospatial data sets, for example, usually contain data records Geospatial data

that link a geographical location to observed values of measurements. These may be for example air pressure, temperature, humidity or wind speed. The structural relationship arises from the spatial distribution: possibly important information may be extracted from

this data when considering relative distances between data points. Another example for EEG data

data, now with a completely different semantic and structure is Electroencephalography (EEG) data, as it is measured by electrodes attached to a patient’s scalp. It is used e.g. in medical applications for disease analysis, serving there as a base for medical findings. Also research in human cognition makes use of such data. EEG can be interpreted as a spatial

(22)

2. Data and its Central Role in Digital Environments

map of the scalp, using the electrodes’ position as a three-dimensional location and the recorded current as a temporally changing feature vector. In addition to spatial relations, also timing aspects contribute to a meaningful interpretation of the data values, and should therefore be considered in their analysis.

2.2. Formal Definitions

Data is plural for lat. Datum, something given; it is, however, today also used in singular to

Origins of the term

Data _{represent a piece of information [}_McK05_{]. The origins of today’s interpretation of the term}

data as it is used in data mining and –exploration may be found in Riemann’s definition of

a manifold [Rie68]: a subset of a mathematical vector space with no dimensional semantics

attached. This abstraction from semantics of the measured dimensions – in geospatial weather data these are the origins of the measured values, i.e. temperature, humidity, etc. – opens the possibility to apply (non-linear) operations to the manifold resulting in a new manifold where it is impossible to interpret the dimensions. However, the new representation possibly makes it easier to identify structural patterns such as clusters or data item dependecies.

For data mining tasks, it is important to have a consistent mathematical representation of

Data as set

data sets. We define it as the set

X = {xα}_{α=0,...,m−1} (2.1)

of m data items xα_{∈ R}n_{, n ∈ N.}

Sorted data sets contain additional information that is implicitly given by their ordering. This circumstance motivates their mathematical description as a series

X = (xα)α=0,...,m−1 (2.2)

Another way to represent data sets is to put them into a matrix

Data as matrix Rm×n 3 X = x0, . . . , xm−1 =    x0₀ · · · xm−1₀ .. . . .. ... x0 n−1 · · · xm−1n−1   , (2.3)

where each column represents one data record. It is very close to the commonly used data structure in computers and has the benefit that all values are easily accessible by their indices. A disadvantage of such a representation, though, emerges, when trying to convert a discrete representation into an interpolated continuous vector field, since the dimensionality m of that matrix would be infinite. In this work, however, X will be used in the sense of

(2.3). Exceptions are otherwise explicitly noted.

Data sets are usually embedded into domains. For a mathematical definition, we call the set

Data Domain

Rn, n ∈ N together with the semantical description of its axes an n-dimensional domain

D. Since the semantical part of such a domain is very difficult to describe in mathematical terms, and therefore cannot make a contribution to this mathematical definition, I argue

that an n-dimensional domain can by sufficiently described by_Rn.

In an arbitrary data set X that is encoded numerically (e.g. by a Shannon-coding), every

data item can be described completely by an element x ∈_Rn. It can therefore be interpreted

(23)

2.2. Formal Definitions

For D, E being two domains, and k, l ∈_{N, the concatenation given by}

D ◦ E = {d ◦ e|d ∈ D, e ∈ E} , (2.4) with ◦ : Rk_{× R}l _{→ R}k+l _(2.5)           x0 x1 .. . xk−1      ,      y0 y1 .. . yl−1           7→      x0 x1 .. . xk−1      ◦      y0 y1 .. . yl−1      =               x0 x1 .. . xk−1 y0 y1 .. . yl−1              

forms a new (k + l)-dimensional domain.

On a computational level, data is often stored in structures that are closely related to the Data storage and

structures

mentioned mathematical descriptions. Defining an array in SuperCollider, the computer

language that is mainly used throughout the applications in this thesis [McC02], is done as

follows:

1 a = ["value", 23]; // a new Array with 2 entries

2

3 a[0].postln; // access a value and print it 4 a.do{|value, i|

5 "At % there is %.\n".postf(i, value) 6 }

Among other possibilities to store data, there is also a more high-level representation, called dictionary. Such a dictionary associates arbitrary values with keys:

1 a = Dictionary.new; // a new Dictionary 2

3 a[\key] = "value"; // assign values to keys 4 a[\data] = [1, 2, 3, 4];

5

6 a[\key].postln; // access a value and print it 7 a.keysValuesDo{|key, value, i|

8 "The value of %(%) is %.\n".postf(key, i, value) 9 }

Another common approach for data representation in computers is implemented in relational databases. In this setup, data are hold in tables; two-dimensional arrays that are optimised for combining and filtering large, mostly numerical or textual databases.

To handle data computationally, instructions in the form of algorithms and programs Computation and

algorithms

are needed. They determine how the data is treated, if and how it is filtered, sorted or processed to be finally presented to the user or stored in a database.

(24)

2. Data and its Central Role in Digital Environments

2.3. The Artificial Separation of Data and Algorithms

This section highlights the interdependencies between data and algorithms. It explains the difficulty in their separation, and outlines that, from a computational point of view, they are basically the same.

In our common environment, the reality, a clear separation between tool and material on the one side and its function on the other is noticeable: While the term object, be it considered a tool or its material is an abstract and general denomination for something physical, the

notion of function denominates the idea of their specifics, their intended semantic.1 The

New Oxford American Dictionary describes functionality therefore as [McK05]

the quality of being suited to serve a purpose well; practicality [. . . ] the purpose that something is designed or expected to fulfill [. . . ].

However, in digital environments these terms are different. In this case the separation between tool, material, or data but also meaning is blurred: An algorithm embodies both, the description of a process incorporating material, and the material itself, which makes it function, tool and material at the same time. My intent in this thesis is that the occurrence of data may always be interpreted as functionality, too. I think it is a good practice on the way to better understand the digital realm.

The fusion between algorithm and data is made explicit in Turing machines, where both the running algorithm and the data it operates on are stored on the same tape, making it possible to manipulate the program itself at runtime. This circumstance is for example

used extensively in the Lisp programming language [Fod91].

Today, however, with mainly imperative programming languages dominating the field, it is rare that programs do change themselves at runtime. Exceptions can be found in artistic programming situations such as live coding and just in time algorithmic music

performances [RdCW05]. Contrasting, the automated implementation of algorithms and

customised functionality is widely known and used, e.g. in the production of serial letters

with LA_{TEX. It’s power can be compared to machines that are able to produce physical but}

customised tools like rapid prototyping systems.

2.4. Data Processing

As mentioned earlier, data and information are the dominating material for computers.

They are designed to easily acquire, shape and display data. The diagram 2.1(a) shows

a typical data workflow of a computing and exploration process. Similar to traditional crafting, it consists of acquisition (i.e. data acquisition or measurements), and manipulation with a tool (the program). In difference, though, the appearance of the resulting material can be chosen almost independently from the crafting process, because data has no human-perceivable gestalt by itself. Working on data with a computer can therefore be seen as closely related to the traditional way of hand-crafting physical material with the help of appropriate tools. The next paragraph, though, states that this is not the case on a closer examination.

1

On a side note, this already incorporates a subjective view, since the function of an object depends on the user’s interpretation of the object.

(25)

2.4. Data Processing Data Program Human manipulates Display observes Digital Workflow Algorithm used in output

(a) Data Workflow

Material Tool Human used in Crafting Workflow Knowledge on tool-building observes manipulates (b) Crafting Workflow

Figure 2.1.: Similarities between the data mining workflow and the typical handcrafting workflow.

2.4.1. Data – the Non-materialistic Material

Due to its usage in digital environments, data is widely viewed as a material such as wood Implicit acceptance

of data as a material

or stone. This implies certain materialistic characteristics and a way to treat it that is based on our common experience with reality. This circumstance has its origin in our often subconscious understanding of it. Already the phrases data handling, data processing, or data mining implicate that data is widely recognised as a basic, materialistic resource. The used words originate in crafting or other physical work.

Data, though, is immaterial, and disembodied. Its physical shape, the modality it is represented in, does by no means determine or affect its content; even more, data is pure content. Neglecting this fact, data mining and data analysis handle data as a material: they process, analyse, and shape it like other work fields process, analyse and shape stones. Nevertheless, the nature of data being a “non-materialistic material” has some inherent features, marking it different to material in the common sense. One of these features is that a data set is not bound to one phenotype. This implies that a change of its modality does to no extent change the data itself: the text of a written book contains no other information than the same text represented as bits and bytes on a hard disk. A change of representation does, however, change the way people perceive a data set, and therefore the data-inherent structure they are able to identify. This is due to the complex interplay of the data’s representation and its perception by the human. So, data is independent of its representation type, but it is nevertheless bound to one (arbitrary regarding its meaning) physical representation.

If this representation is well-suited for an algorithmic processing by computers, it is – most Representation

duality

of the time – not in a form that supports human perception or structure recognition. The reason for this is not that the machine-oriented representation is too complex to understand, moreover the pure physical representation (binary values coded to voltage in semiconductors or magnetic forces on hard discs) is completely inappropriate for the human senses and cannot be decoded without appropriate tools.

(26)

(27)

3. Exploratory Data Analysis

Human intelligence, particularly imagination and creativity prove to be important resources in everyday live. Intelligence enables us to discover unexpected structures, find previously unknown coherence, and gain new insights; all based on our knowledge of the world. These cognitive abilities can be seen as a result of human creativity, a unique way of thinking and creating, an ability to connect previously unconnected aspects of the world. Darwin’s theory of evolution supports the argument that the characteristic shape of our imagination is based on the physical conditions of our immediate vicinity. Therefore, we are well-trained to unveil and manipulate structures in our everyday environment; spatial thinking and imagining future scenarios arising of current situations are easy for us. This can be exemplified with the help of the examination of a hypothetical passing of a hard-to-find pathway in the hills: We are not only able to easily spot its almost hidden course, but we can also manage to walk it without stumbling, and can at the same time anticipate how it possibly continues. This capability is the result of the harmonic interplay between the evolutionary deployed combination of body and brain.

However, the imagination respectively recognition of abstract structures in both theoretical and especially mathematically formalised spaces is considerably more difficult for us. An example for this is playing chess, a board game that requires thinking on a highly symbolic

and abstract level. This game is generally accepted as difficult and highly complex,

incorporating theoretical and non-linear thinking. Nowadays, it is played far better by computers than by humans, although these computers cannot be considered as intelligent, or even more intelligent than the people they beat in chess.

Disregarding the circumstance that we do not have a considerably good performance in the The usefulness of

EDA

analysis of abstract data and algorithms, we still have to invest a huge and constantly growing part of our time into their analysis and exploration. Such tasks arise e.g. from working with pre-recorded data from investigations in a supermarket, astronomical measurements, computer programming or from the work on a quantum-physical experiment that helps us understanding our environment. The recorded data lack a physical representation that is easy to perceive. The measurements are, after all, primarily composed of abstract numbers attached with a description and a certain (probably not linear) correlation to each other. Our intent then is to find these structures and use them for decision making.

As described in Chapter 2, the view on data, abstracting from its semantics (i.e. its Riemanns Manifolds

and its impact on data mining

description) can, according to Riemann, be interpreted as a manifold that is embedded into

a high-dimensional vector space [Rie68]. Although this abstraction turns the data even

more into the abstract, this techniques is essential for an algorithmic processing, which is necessary to either perform an automated analysis regarding known structural features, or prepare it for a representation that allows people to use their innate pattern recognition capabilities. This automated and computer-assisted information retrieval is called data

mining [BH03] [FPSSU96]. It can be differentiated into Confirmatory Data Analysis (CDA)

(28)

3. Exploratory Data Analysis something interesting Random Search Directed Search general model Data specialised methods, statistical data analysis Inducement Intuition no no yes yes

(29)

3.1. Workflow in Exploratory Data Analysis

It is the aim of EDA to invent and further develop virtual tools and artefacts that can be used for data exploration. Therefore, one of EDAs dedicated tasks is to actively support human creativity by representing abstract data and algorithmic processes in such a way that is in line with the particular requirements of the human structure recognition system. One very common approach is the visual representation of data. It embeds otherwise abstract data into a humanly well-perceivable form, adding the possibility to use the visual sense and its characteristics in order to recognise structure and information. Other approaches

use auditory and physical representations [BHR05] [HBRR07], or develop a completely

virtual data space surrounding the user in a multi-modal fashion [dHKP02].

3.1. Workflow in Exploratory Data Analysis

Due to its exploratory nature, the EDA workflow tends to vary a lot. In order to still convey a general impression on how Exploratory Data Analysis usually works, a typical

iteration will be described next and is also shown in the flowchart in Figure 3.1.

An EDA usually starts with data acquisition, followed by preliminary data preparation

that includes steps like the elimination of missing data and sphering.1 As a next step, the

cleaned data is usually pre-processed by dimensional reduction methods (e.g. PCA [Jol86],

or ICA [Com91]), or analysed for clusters [JD88] or other higher level features. The result

is then presented to the user via display technologies such as visualisation or Sonification.2

It is possible for the researcher to manipulate that representation and apply human-aided feature extraction algorithms. Usually, this workflow is repeated until an idea of a general model is discovered. In this situation, the researcher eventually changes his strategy from a random search to a directed search process. If a general model or source of structurally relevant information arises, he will leave the field of exploratory analysis in order to move on to more specialised methods that are dedicated to find significant validations for his theory.

3.2. Standard Techniques

In general, all approaches of data exploration change the representation of the data under exploration to an extent that specific features like local density, clustering or correlations can be better recognised by human perception. Main differences of these approaches, though, can be observed in (a) the used modalities, (b) the amount of possible user interaction, and (c) the level of abstraction from the original data. Technically, (a) has an important influence on how data can be transmitted technically – each medium has its specific character and therefore features – and which information can be carried out in what quality

by the perceiving human. This aspect is covered in Chapter6 in more detail. The amount

of possible user interaction (b) highly depends on the way data is represented and how many parameters of this representation can be changed by the user in realtime. As stated before, Exploratory Data Analysis is the collection of techniques and methods to unveil unknown information and structure from data; a process in which creativity, intuition and

1

Sphering is also known as whitening, and means to remove biases in all measured dimensions, followed by a variance normalisation. See standard literature on statistics like Steel [ST60] for further details.

2

(30)

3. Exploratory Data Analysis

curiosity plays an important role. The more direct and immediate activity is possible with data exploration systems, the better will be the results that can be achieved. The level of abstraction (c), finally, decides on the granularity of information presented to the user. In this case, eventually, a compromise between analogue and direct representation on the one side, and a symbolically higher representation that is based on automated analysis has to be found. While a quantitative and direct representation features a broad variety of possibly useful information at once that, however, may be difficult to understand since all feature extraction has to be done by the human and transfers the full complexity of the data under exploration, a more symbolic representation is based on the pre-processed and –analysed data. Although better to understand, it carries the risk to miss important structural information of which was never thought and therefore not searched for. By means of meta-controls, it is possible to change this level of abstraction to allow users to fluidly change between analogue and symbolic representations. This practice will be described for

Tangible Interfaces in Section 5.4.3.

As pointed out, the perception of display technologies carries specific characteristics that

Why different

approaches _{heavily depend on their modality. Each of them helps users to detect specific structures,}

whereas it does not feature other, maybe equally important aspects. It is fairly easy, for example, to use graphics and prepare data in a way that the viewer is able to recognise

cluster-like structures. Such standard techniques for visual analysis are described in

Section6.2 A phase shift in a quasi-periodic signal on the contrary can only be made visible

with considerable effort.

Although vision is clearly the preferred modality used for data exploration, audio-based displays gain more and more attention, since they focus on other aspects of the data under exploration. Aspects that are difficult to perceive visually. Up to now, there are no widely established standards in Sonification, however, the International Community

for Auditory Displays is in the process to establish such standards [KWB+97]. Albeit,

there are some promising fields aiming for attention regarding data exploration, since they are especially suited for multi-variate data. These fields include Parameter Mapping Sonification, Audification, and Model-Based Sonification. Among others, these techniques

will be described in detail in Section6.3.

3.3. Neighbour Fields

As mentioned in the last section, EDA is part of data mining. Closely related is analytical data analysis. Accompanied by statistical data analysis, it is most commonly used to get valid and verifiable results for structural relations. First hints into this direction are often found with the help of EDA. Other fields related to EDA are computer science and display technologies. They provide the algorithmic theory respectively hardware to be used in EDA applications. HCI and psychology, finally, are needed to get information on how a data exploration system should be designed in order to fulfil human requirements.

Apart from these direct interconnections, data monitoring is also related to EDA. It uses the same methods and mechanisms, but for a different purpose: While data exploration aims for new insights into (probably already known) data domains, data monitoring tasks are intended to increase the perceivability of data features. These features are, however, often already well understood. In a medical context, for example, data monitoring is often needed

(31)

3.4. Data Representations Exploratory Data Analysis Data Monitoring HCI Computer Science Psychology Statistics Statistical Data Analysis Data Mining

Figure 3.2.: Fields related to Exploratory Data Analysis.

to warn people according to the situation of patients. This situation can be supported by representation systems that represent e.g. EEG data by sound or vision that integrate into the staff’s ambient environment.

3.4. Data Representations

Human Perception Digital

Realm

Information Processing Digital Interface Hardware Human

Figure 3.3.: Data transcribed from a digital storage to human perception has to go through several layers.

All data is represented in a certain way. The form of its representation hereby heavily relies on the context in which it is intended to be used. In a digital envi-ronment, for example, data should be optimised for digital data processing, whereas in a human related environment, data should be optimised according to human perceptual skills.

A digitally optimised data representation can be fully described as a valid element of a superset of symbols of a predefined alphabet. Take for example data from a digital photo camera that has to be saved for further usage and processing. This is done by filling a list of Integer variables with values ranging from 0 (black) to 255 (white) according to the brightness of points

in the photography.3 To be perceived by a human,

however, the digital data4 has to be transcribed into

a human perceptible representation. For this, it has to pass at least one digital processing stage (usually a software that turns the value of a

3 _{For the sake of simplicity, only the case of monochrome image processing is covered, data structures for}

coloured images are more complex, but are based on the same principals.

4

Digital data actually is a wrong term: it is not the actual data that is digital but its form of representation. To not impede the reading flow by over-complex terms, I consider the term digital data to be an equivalent to digitally represented data.

(32)

list item into the value of a virtual pixel) and a hardware interface, which transcribes the prepared digital data (the pixel value) into a human perceivable event (the brightness of one point per list entry on the display’s surface). This data transcription process from the

digital realm to a perceptual stage is exemplified in Figure3.4.

When examining data representations, it is essential to separate the following terms: Data transcription is the formal act of moving data from one medium to another. Data representation is how data is stored.

Data perceptualisation is how data is perceived.

An example for data transcription is the act of copying data from a hard disk to a DVD,

Data transcription

but also the act of printing a visualisation of formerly digitally stored data to a sheet of paper. Each stage, be it the digital processing or the hardware interface, introduces specific properties that may hide, emphasise or even omit parts of the original data-inherent information. Each of the forms in which the data appears in these examples is referred to

Data representation

as data representation, i.e. their visual apearance on paper, their structure that is linked with the optical representation of bits on the DVD, or the structure and magnetically stored bits on the hard disk. Data always is represented with the help of a medium; its representation can be decoded with help of a (sometimes implicitly available) grammar. Data perceptualisation, finally, describes the way data is perceived. The perception heavily

Data

perceptualisation _{depends on the current representation and the perceiver. The perception process includes the}

perceiver’s interpretation as well as his abilities for structure recognition and information retrieval based on the particular data representation. It also covers the perception of representation-inherent artefacts and their potential misleading.

3.4.1. Representation Classifications

This section discusses and proposes indicators that may be used when describing data transcriptions, representations or perceptualisations. For this, I propose that a description

Requirements

of a data transcription into a new representation form should include information on 1. the incorporated sensorial modalities,

2. which data-inherent structures are emphasised,

3. the level of interaction between user and representation system,

4. the level of reality (as described in RBI, see Section4.5.1),

5. the supported and preferred types of input data (e.g. sequential or cartesian), 6. the percentage of passed-through information, and

7. the symbolic level, i.e. whether high- or low-level symbols are used.

The requirements 1 through 5are – given a specific transcription – more or less easy to

deduce from the technical parts of the representation system. The requirements 6and7,

though, need a closer look. In the following, I describe classification systems and other approaches that can be used as indicators for these parts. Because of the focus of this work, I chose most of them because of their close relation to Auditory Displays.

(33)

3.4. Data Representations

Sloman’s Analogical and Fregean Representations

Sloman explains in his Afterthoughts on Analogical Representations the difference between

Analogical and Fregean representations [Slo75]. While he defines analogical representations

to be complex representations of complex data, obligatory having a structure that corre-sponds to the structure of the represented, Fregean representations do not need to have an obvious correspondence to the data’s structure. For Sloman this especially means that the interpretation

Analogical representations are continuous, Fregean representations discrete as cited in his paper is a misinterpretation, because

[there are] examples of discrete analogical representations, e.g. a list whose elements are ordered according to the order of what they represent.

However, a differentiation between continuous and discrete streams of information represen-tations is often obvious in human-computer interaction contexts.

Kramer’s Analogic/Symbolic Chart

A similar approach based on Sloman is Kramer’s Analogic/Symbolic placement scale [Kra94],

in which he claims that

[an] analogic representation is one in which there is an immediate and intrinsic correspondence between the sort of structure being represented and the repre-sentation medium. The relations in the reprerepre-sentation medium are a structural homomorph of the relations in the thing being represented. A change in the representation medium [. . . ] has a direct correspondence with the thing being represented [. . . ],

whereas

[b]y symbolic representation we refer to those display schemes in which the representation involves an amalgamation of the information represented into discrete elements.

Kramer proposes that – in difference to Sloman – the classification and differentiation of Sonifications into his system is continuous. He proofs it by filing representative examples for Auditory Displays into his classification system. Although a continuous mapping space, Kramer’s analogic/symbolic chart does not cover the above-mentioned, seemingly natural, discrimination between discrete and continuous data representations.

De Campo’s Sonification Map

A third theory to classify – purely sonic – data representation is described by de Campo in

his PhD thesis [dC09a]. The there-introduced Sonification Design Space Map (SDSM ) draws

a three-dimensional figure on how sound and meaning can be connected to render a sonic data representation. The aim of the SDSM is less to analyse existing data representations, furthermore, it should support to

find transformations that let structures/patterns in the data (which are not known beforehand) emerge as perceptual entities in the sound which jump to

(34)

(a) Kramer’s Chart on Analogic/Symbolic Continuum [Kra94].

(b) DeCampo’s Sonification Design Space Map [dC09a].

(35)

3.4. Data Representations

the foreground, i.e. as identifiable ‘interesting audible objects’[. . . ] Therefore, the SDSM can be used

to achieve improvements to solve the most general task in data Sonification designs for exploratory purposes [, namely] to detect auditory gestalts in the acoustic representation, which one assumes correspond to any patterns and structures in the data one wants to find.

De Campo’s intend can therefore be entitled as to guide the design of a data representation process in such a way that it fits the needs of the researcher. In his ibid. developed Sonification designs, he points out paths through the SDSM rather than concentrating on fixed points, which enables him to describe the actual design process as a continuous and intentional series of decisions based on user experience and the goals of the resulting system. This massively increases the usability of the SDSM and introduces an indicator for changing what I call the level of abstraction of a Sonification. Together with the definition

of a level of abstraction for Tangible Interfaces (as described in Section 5.4.3) this forms a

powerful toolbox for Tangible Auditory Interfaces.

3.4.2. Considerations based on the presented classification strategies

In the last subsection, an overview on common techniques to represent data and its structure was given. With Sloman’s analogical-fregean, and Kramer’s analogic-symbolic ranges, we get two closely related indicators that are based on subjective interpretations of the representation under exploration, since they rely on the characteristic of the information perceived by the human. De Campo’s SDSM on the other hand introduces descriptive dimensions like the number of data points, the number of data properties, or the number of audio streams. Their combination is used to indicate an appropriate representation method. This strategy elegantly avoids the need to classify these methods according to their information preservation. Instead, the choice of Sonification strategies (and also strategies that include other modalities) is based on the experience of experts.

Many researchers, however, would prefer to actually use quantitative measures to compare representation techniques with each other in order to make decisions regarding their quality. Unfortunately, already the computation of the norm

k.k_S: S → R (3.1)

with S the set of all representations, determining the valuable information in that rep-resentation is impossible. At least when humans are incorporated into the perceptual process. The individual information content of a data representation is highly subjective; only when the states of all incorporated systems are known, the actual information content of a representation can be determined. In the case where the quality of a representation is based on human perception and analysis, only estimations based on quantitative and qualitative evaluations can be made. In these cases, it still remains difficult to generalise the performance of individuals. This makes indicators for e.g. the level of detail of a representation or its information to noise ratio unreliable.

Another aspect speaking against a quantitative measurement at least for representations that are intended for Exploratory Data Analysis is the aim of these representations: Estimations based on quantitative measurements abstract from the participants’ individual performance

(36)

in favour for a better generalisation. Although this might be effective in situations where the majority’s performance is relevant, it does not make sense for explorative situations, in which it is essential to find any hint on any structure. If only one person can effectively use the representation to unveil unknown structural information, this has a significant impact and is considered as relevant. The general performance of the prototypical human backs out in favour to the individual. Qualitative methods, e.g. those based on grounded theory

(as it will be described in Section 4.6in more detail) are able to emphasise these aspects.

They also build a solid basis for the analysis of the indicators by Sloman, Kramer and de Campo.

Tangible auditory interfaces : combining auditory displays and tangible interfaces