Visualization of Unfolding Emotions

Three approaches were used for the visualization of the self-report data. The easiest and most obvious way of displaying such time-series data would be to plot the averaged data as shown in Figure 5.5. However, the use of a third dimension enables the user to retrace the trajectories of the emotions the 38 participants had indicated. At approximately 90 s, a significant break in the emotional impact of the music of ID 3 can be seen (see Figure 5.4d),

Consistency of Self-reporting

Ratings in the continuous self-reporting were not always coherent, despite distinct events, such as beginnings of new sections. The means of valence and arousal are shown for the film music “Main Titles” (Portman, 2000) in Figure 5.5. One can easily see a change in self-reporting around 90 s.

Finding such changes in self-reports can be facilitated by looking at the moving variances in the self-reports. This is shown in 12-second windows, all taken from the same musical piece, in Figure 5.6.

Animations

An impressive account of the unfolding of emotions over time becomes visible when animated data is used using films as a synopsis of the sampled self-report

5.1. USABILITY OF THE EMUJOY-SOFTWARE

Figure 5.4: Self-report data of participants who were listening to two musical pieces displayed from bird’s eye view (same mode of visualization as in Figure 5.2). In the two upper panels, emotional self-reports from hearing the death metal music piece “Skull full of maggots” by Cannibal Corpse are displayed. In contrast to Figure 5.2 and the lower right panel, no columns can be identified, whereas the horizontal patterns demonstrate high individual constancies. In the lower panels self-reports from hearing the pop-music piece ’Making Love out of nothing at all’ by Air Supply are depicted. As can be seen on the lower right panel, there is an inter-individual consistent change in self-reported arousal in around 70, 100 and 140 s. However, an overall tendency towards aversion (a) and high arousal (b) is ascertainable amongst all participants in response to the death metal music piece.

46

0 50 100 150

Figure 5.5: Mean of all participants’ rating of arousal (left) and valence (right). Only very strong changes are visible.

data from all participants. In all animations, there is a preference towards one of the quarters of the emotion space. However, when there are breaks in the music, such as the beginning of a new part, the entrance of voices, a change of key, etc., an overall tendency towards movement can be seen in all data points. The colors of the worms, used to represent individual participants, are meaningless.

A worm’s head changes its shape from a circle to a rosette when the participant reports a chill. The animations are available on DVD in the Appendix D; a screenshot from the visualization is shown in Figure 5.7, the birds’ eyes’ view on the data was presented in Figure 5.4 (page 46). In the PDF-version, the video starts by clicking on Figure 5.7.

There are several advantages compared to study only averaged data and there is an additional aesthetic value. Changes in emotions experienced in the same di-rection but on different levels, i.e. starting at different levels of valence or arousal, are more easily recognized. Furthermore, diverging trajectories, whose averages cancel each other out, can still be observed. The third approach for visualizing changing emotions has already been shown for images from the IAPS (see Figure 5.2) and for musical pieces (see Figure 5.4).

5.1. USABILITY OF THE EMUJOY-SOFTWARE

Figure 5.6: Changes become emphasized by calculating moving variances. This is done here for 12-second windows.

Figure 5.7: Emotional self-reports over time are visualized within animations.

48

5.2 Experiment I

The goal of the first experiment was to find out which participants when experi-ence chills, the music with which this occurs and the psychoacoustical parameters that play a role. With respect to the listeners, I was interested in the ongoing psy-chological and psychophysiological processes during the music listening. These were investigated by concentrating on the peripheral indicators of psychophysiol-ogy rather than brain imaging.

Patterns in physiological changes and temporal dynamics in these patterns were to be defined. There is a delay between musical event and the emotional correlate in the self-report (Schubert, 2004b). However, an investigation needed to be made as to whether there is also a delay between physical indicators of SEMs and the emotional self-report.

The focus of the experiments was laid on chill events and their correlation to psychophysiological changes, such as HR and its variance, SCR, and EMG, as shown by data gathered as the participants listened to full length musical pieces in a laboratory setting.

Physiological parameters for full length musical pieces that elicited chills were compared to those that did not elicit chills. Physiological time series at chill onsets are averaged in order to find patterns in psychophysiological parameters.

Additionally, the relation of chills to self-reports were investigated in valence and arousal. This raised the question as to whether psychophysiology without self-reports from participants is always appropriate for predicting chills.

Participants were asked to listen to music that they had brought with them, which was expected to lead to an experience of chills. Other musical pieces were pre-selected and played to all participants. They reported the emotions they felt during the listening with the computer software EMuJoy (see Section 4.4).

Physiological parameters for identifying correlates with strong emotions during the listening were recorded during this experiment.