The Neural Processing of Emotional Pictures:
Evidence from Evoked Potentials and Functional Magnetic Resonance Imaging
Dissertation
zur Erlangung des akademischen Grades des Doktors der Naturwissenschaften an der Universität Konstanz
Mathematisch-Naturwissenschaftliche Sektion Fachbereich Psychologie
vorgelegt von
Tobias Flaisch
aus Mengen Konstanz, März 2007
Tag der mündlichen Prüfung: 21.06.2007 1. Referent: Prof. Dr. Harald Schupp 2. Referentin: Prof. Dr. Johanna Kißler
Konstanzer Online-Publikations-System (KOPS)
Index
Index 2
Danksagung / Acknowledgments 3
Summary 4 Zusammenfassung 5
General introduction 6
The bioinformational theory of emotion 7
The picture viewing paradigm 8
Peripheral physiology 10
Functional magnetic resonance imaging 11 Event-related brain potentials 13
Motivated attention 16
The present thesis 17
Emotion in your hand: The selective processing of affective hand gestures in
visual cortex 21
Introduction 22 Method 27 Results 35 Discussion 41 Fleeting fingers: The rapid discrimination of emotional hand gestures 45
Introduction 46 Method 50 Results 54 Discussion 60 Rapid picture processing: Affective primes and targets 64
Introduction 65 Method 68 Results 71 Discussion 77 ERPs reveal interference between sequential affective pictures at distinct
processing stages 81
Introduction 82 Method 86 Results 90 Discussion 98
General discussion and outlook 103
References 106
Danksagung / Acknowledgments
Zu allervorderst gebührt der größte Dank meinem Betreuer Prof. Harald Schupp. Ohne seine kontinuierliche fachliche Unterstützung und persönliche Ermutigung wäre diese Arbeit vermutlich nie fertig gestellt worden.
I want to express my great appreciation and deep gratitude to Prof. Peter J. Lang and Prof.
Margaret M. Bradley who welcomed me at their fantastic lab in Florida and thereby gave me the exceptional opportunity to experience what good science is all about and what it means to not hope for specific results.
Besonders bedanken möchte ich mich des Weiteren bei allen Kolleginnen und Kollegen der Arbeitsgruppe Allgemeine Psychologie für die freundschaftliche Zusammenarbeit und die entspannte Atmosphäre. Namentlich nicht unerwähnt lassen möchte ich hierbei Ines Krug – Danke für die vielen Daten –, Sabine Widmann-Schmid – Danke für den Papierkrieg –, Andrea De Cesarei – Thanks for pretty much everything, Dude – , Peter Peyk – Danke für EMEGS –, Florian Bublatzky, Ralf Schmälzle, Margarita Stolarova – Danke für das Teamwork – und Frank Häcker – Danke für den Fahrdienst. Nicht fehlen darf hier natürlich auch Dr. Markus Junghöfer, der mir immer mit Rat, Tat und exemplarischer Gelassenheit zur Seite stand.
Ein ganz großes Dankeschön geht an meine gute Freundin und Lieblings-Kollegin Dr. Jessica Stockburger, die diesen Weg von Anfang an gemeinsam mit mir beschritten hat und mir wohl auch in Zukunft einige große Schritte voraus sein wird.
Ein wichtiger Begleiter war und ist mir weiterhin Dr. Marcus Meinzer. Danke für die Freundschaft, die beständige Aufpäppelung meines Egos und natürlich die sich anbahnende Flut von Zweit-Autorschaften.
Nicht viel sagen möchte ich zu meinem besten Freund Dr. Marco Steinhauser. Nur soviel:
ohne Dich wäre ich nie und nimmer in der Verlegenheit, bald einen neuen Ausweis bestellen zu können.
Vielen Dank auch allen meinen Freunden, die mir von fern oder nah oft genug dringlichst notwendige Stütze waren und sind: Iris & Ingo, Tina & Rainer, Simon, Elke & Ernst, Jörni, Marc, Matthias und Ralph.
An dieser Stelle gebührt die Ehrenposition natürlich meinen Eltern Mine und Walter Flaisch und meiner ganzen Familie, Alex & Frank, sowie Magge & Claudia. Ohne meine Eltern wäre ich niemals an diesem Punkt angelangt. Danke, dass ich immer eine Heimat habe und weiß, dass ich im Leben nicht alleine sein werde.
Tobias Flaisch
Konstanz, im März 2007
Summary
The present thesis examines the neural processing of emotionally salient picture stimuli. Previous research has determined that passively viewed emotionally evocative images are potent elicitors of a number of neurophysiological responses.
Functional brain imaging studies served to detail the neuronal structures involved in selective emotion processing. Moreover, results from EEG studies allowed identifying distinct sub-processes sensitive to the emotional quality of picture stimuli. The present thesis extends this research by illuminating two aspects of emotion processing.
In the first part the question is pursued to what extent the well-established effects of selective neuronal encoding of emotional images are dependent on the evolutionary significance of the eliciting stimulus materials. Two studies are presented using functional magnetic resonance imaging and event-related potentials, respectively. In the reported studies, symbolic hand gestures bearing distinct emotional meaning are introduced as a new stimulus class in the domain of emotion research.
The obtained results show that emotional gestures are associated with both increased functional activation in extrastriate visual cortex, as well as a modulation of early electro-cortical indices of emotion processing. It is concluded that the emotional significance evoking selective neuronal processing is not limited to evolutionarily important stimuli but that this significance may also be acquired through ontogenetic learning.
The second section investigates the hypothesis of sequential processing interference between successively presented pictures of varying emotional content. It is comprised by two studies utilizing event-related potentials. Specifically, it was tested whether features of interference are apparent at two distinct emotion-sensitive sub-stages. Consistent with the notion of temporal processing interference, the findings reveal diminished amplitudes for both early, as well as relatively later indices of selective emotion processing as a function of the emotionality of preceding stimuli.
These results are interpreted in support of a multi-stage account of neuronal emotion processing.
Zusammenfassung
Die vorliegende Arbeit untersucht die neuronale Verarbeitung emotionaler Bildreize. In früheren Studien wurde gezeigt, dass passiv betrachtete emotionale Bilder potente Auslöser für eine Reihe neurophysiologischer Prozesse darstellen. Bei Untersuchungen mittels funktionell-bildgebender Verfahren wurde festgestellt, dass emotionale im Vergleich zu affektiv neutralen Bildinhalten differenzierte neuronale Aktivierungsmuster induzieren. Darüber hinaus gelang es mit Hilfe von elektrophysiologischen Untersuchungsmethoden, distinkte emotions-sensitive Unterprozesse voneinander zu unterscheiden. Die vorliegende Arbeit erweitert diese Forschung durch die Untersuchung zweier Aspekte der emotionalen Verarbeitung.
Im ersten Teil wird der Frage nachgegangen, inwieweit die bekannten Effekte selektiver neuronaler Enkodierung emotionaler Bildreize von deren evolutionären Bedeutsamkeit abhängig sind. Es werden zwei Studien vorgestellt, die sich methodisch der Funktionellen Kernspintomographie, bzw. Ereigniskorrelierter Potenziale bedienen. Hierbei werden erstmals emotional bedeutungshaltige symbolische Handgesten im Bereich der Emotionsforschung untersucht. Die Verarbeitung emotionaler Gesten war zum einen durch eine erhöhte Aktivation extra- striärer Kortexareale, sowie zum anderen durch eine spezifische Modulation früher elektrokortikaler Indikatoren emotionaler Verarbeitungsprozesse gekennzeichnet.
Durch die Einführung dieser neuen Reizklasse wird gezeigt, dass selektive Reizverarbeitung im Gehirn nicht auf evolutionär bedeutsame Stimuli beschränkt ist, sondern auch durch ontogenetisch erworbene emotionale Signifikanz ausgelöst wird.
Im zweiten Abschnitt wird untersucht, ob die Verarbeitung affektiver Stimuli zeitlichen Interferenz-Effekten unterliegt wenn Bilder unterschiedlichen emotionalen Gehalts in schneller Aufeinanderfolge präsentiert werden. Die hier berichteten Ergebnisse aus zwei EEG-Studien zeigen, dass sowohl frühe, als auch spätere emotions-sensitive EEG-Komponenten mit einer Verminderung ihrer Amplitude als Funktion der Emotionalität unmittelbar vorhergehender Bildreize einhergehen. Diese Resultate sind sowohl mit der Hypothese der sequenziellen Interferenz, als auch der Annahme distinkter Unterprozesse bei der neuronalen Verarbeitung emotionaler Bildreize vereinbar.
General introduction
Everyone knows what an emotion is. In reference to a famous quotation by the great William James regarding attention, this statement points to the outstanding importance of emotion for every single human being. When being asked, everybody would admit without hesitation that emotions exert a great influence on our everyday life by shaping how we perceive events, what conclusions we draw from experiences and consequently what decisions we make in our lives. Still, for long psychology has largely left out emotion in the endeavor of understanding the human mind and human behavior. Just in the last decade, researchers have developed an increasing interest in the significance of emotion for mental processing and behavior. In fact, the idea that emotion is not only a subjective personal experience but rather constitutes one of the prime psychological functions affecting human behavior is an old one. One of the first to explicitly acknowledge its behavioral significance was already Darwin (1872) who proposed that emotions are not private mental episodes but would instead have adaptive behavioral value, promoting favorable actions under given situational circumstances. In his view, emotions and particularly emotion displays such as body language in animals or facial expressions in humans are thought to be a result of each species’ specific phylogenetic history and as such are shaped by evolution.
However, even though the first attempts to consider emotion from a scientific perspective date back to the early days of Darwinian evolution theory, science has yet to agree on a unified definition of the research subject. One of the major disagreements between the different views is whether the study of emotion is best approached by categorizing them into a set of discrete basic (or “primary”) emotions (see e.g., Ekman, 1999; Panksepp, 1992) or whether they should rather be understood as a dimensionally organized phenomenon (see e.g., Cacioppo et al., 1999; Lang et al., 1997). Based on subjective experience, folk psychology easily accords to the existence of qualitatively distinct emotions such as anger, fear, disgust, happiness, surprise, etc.
However, acknowledging the behavioral relevance of emotion it has long been a scientific endeavor to characterize distinct basic emotions not only in terms of subjective introspection but rather based on their respective physiological response patterns. Yet, empirical evidence in support of this account is sparse and science is still struggling to unequivocally distinguish between qualitatively different emotions on the basis of physiological responses. As a result, theorists have greatly disagreed on both
the number of distinct basic emotions, as well as their respective qualitative nature (Ekman, 1999; Izard, 1972; Panksepp, 1992; Plutchik, 1980; Watson, 1924). To complement this approach, other researchers have advocated a dimensional access to the scientific investigation and the definition of emotion (see e.g., Cacioppo et al., 1999; Lang et al., 1997). This view emphasizes that affects are fundamentally organized by overarching motivational factors. These factors include one dimension which localizes emotion and emotion related behavior on a continuum of hedonic valence, basically distinguishing between events that promote (i.e., pleasant, appetitive, preservative) or threaten (i.e., unpleasant, aversive, protective) an organism’s life. The second dimension however is supposed to reflect the strength or intensity (i.e., “arousal”) of the respective emotion. Whereas this conception provides less specificity, it provides a powerful framework to theoretically combine current empirical results from various data sources. Specifically, some authors have pointed out (see e.g., Bradley, 2000) that the study of emotion has to rely on the measurement of three output systems:
1.) subjective report, e.g., verbal descriptions or ratings of emotion
2.) overt action, e.g., expressive language, vocalization and performance measures or observable facial expressions
3.) physiological response, e.g., autonomous or somatic reactions
Based on this database, dimensional theories of emotion have proven to be fruitful to integrate empirical evidence from the various emotional response channels.
The work reported in the present thesis largely adopts the perspective put forward by the dimensional approach to emotion, in particular it is theoretically grounded on the
“bioinformational theory of emotion”, as conceived by Lang and colleagues (Lang et al., 1998a).
The bioinformational theory of emotion
Many theorists propose that emotions are founded in a fundamental biphasic organization of the underlying affect system (Cacioppo et al., 1999; Dickinson &
Dearing, 1979; Dickinson & Balleine, 2002; Konorski, 1967; Lang et al., 1990; Lang, 1995; Lang et al., 1997; Schneirla, 1959). As such, two distinct motivational sub- systems subserve two basic behavioral alternatives. On the one hand, withdrawal from and defense against potentially harmful environmental cues is controlled by the
protective defensive system. Approach-related behaviors such as foraging, ingestion, copulation, and nurture of progeny on the other hand are coordinated by the self- preservative appetitive system. In this bivariate view, the motivational sub-systems mediating emotions up to their most complex manifestation in man have evolved from simple organisms and have their physiological bases in distinct sub-cortical networks.
Early analyses of emotionally evocative stimuli (e.g., Mehrabian & Russell, 1974;
Osgood et al., 1957; Russel, 1980; Wundt, 1896) resulted in a semantic structure suggesting two affective dimensions, valence and arousal. Built on these theoretical foundations, Lang and colleagues (1997) proposed a model which views emotions as action dispositions. As such, they are comprised by motivational, physiological/behavioral, as well as cognitive components. The basic organization of these action dispositions is built up by appetitive and aversive tendencies.
Accordingly, in this model emotions can be localized along two continuous dimensions. The first one is constituted by the ‘valence’ dimension reaching from pleasant to unpleasant and indicating the differential engagement of two basic motive systems. The second one is the ‘arousal’ dimension reaching from calm to aroused and is supposed to modulate emotional behavior regarding activation or intensity. It is furthermore proposed that the valence dimension is rooted in underlying motivational sub-systems, whose anatomical substrates are presumed to be located largely in sub- cortical neuronal structures As opposed to valence, the arousal dimension is not believed to be a third system having its distinct neuronal substrates, rather it is an indicator for the degree of involvement of either one or both of the basic motive systems. So far, research has largely focused on the neural circuitry underlying the defensive motivational system. Accordingly, animal research suggests that the function of this system is mediated by a network organized around the amygdala as a core structure (see e.g., Davis, 1997; LeDoux, 2000). The neuronal substrates of the appetitive motivational system on the other hand are to date much less understood.
The picture viewing paradigm
To investigate the psychophysiology of emotion, Lang and colleagues (Lang et al., 1997) have strongly advocated a methodological approach that vastly relies on passive picture viewing. One major advantage arising from this paradigm is that it provides a highly controlled experimental setting. In particular, it has been stressed that emotional responses are strongly dependent on the respective situational context.
For instance, whereas the confrontation with an aversive stimulus in one situation results in flight and avoidance, it may elicit aggression and approach-related actions in another context given the situational circumstances render other behavioral alternatives impossible. However, in both cases the emotional state of the organism is the same. Therefore, Lang and colleagues (1997) distinguish between strategic and tactical aspects of emotional behavior. Accordingly, the strategic frame sets the basic configuration of the underlying affect system whereas the contextual tactics are determined by emotional action tendencies and the respective situational demands.
Therefore, the fundamentals of emotion need to be studied in an experimental context that reduces and constrains local actions of the research participant, in effect ensuring the comparability of empirical results and enabling to draw inferences about underlying processes.
Figure 1.1
Affective Space spanned by the two basic dimensions of emotion, valence and arousal. Illustrated is the distribution of IAPS-pictures within this space, standardized by means of self-report scales (from Lang et al., 1998a)
Specifically, the responses measured during passive picture viewing are primarily of perceptual nature and index the motivational strategy strongly controlled by the stimulus. Apart from this major benefit, picture stimuli also provide high experimental control in terms of exposure time and regarding perceptual characteristics. To promote these advantages gained from the picture viewing paradigm, Lang and colleagues (1999) have developed a set of photographs (“International Affective Picture System”, IAPS; see Fig. 1.1) which is calibrated with respect to self report scales (“Self-Assessment Manikin, SAM; Bradley & Lang, 1994;
see Fig. 1.1, axes) and thus provides an experimental standard for emotion research.
Moreover, due to this standardization the emotional output systems measured by subjective reports and physiological responses can be easily related to each other.
When considering the affective space spanned by the subjective evaluation of these pictures, it becomes apparent that the two emotional dimensions are not completely independent from each other. In particular, it is obvious that high arousal is strongly related to high evaluations of pleasure, or unpleasantness respectively.
Peripheral physiology
By strongly relying on the picture viewing paradigm, predictions and assumptions drawn from the two-dimensional theory of emotion find support in a wealth of studies employing peripheral physiological measures. Several responses covary systematically with the judged valence and/or arousal of picture stimuli. First, electro-dermal activity (EDA) – measured by means of electrical skin conductance – is modulated as a function of arousal (see Fig. 1.2, lower panel, left). That is, both pleasant, as well as unpleasant arousing pictures elicit heightened skin conductance responses as compared to neutral images (e.g. Bradley et al., 2001). Second, heart rate patterns (electrocardiogram, ECG) are selectively modulated as a function of valence (see Fig. 1.2, upper panel, right), showing more pronounced initial deceleration for unpleasant pictures and a larger accelerative peak for positive pictures (e.g., Bradley &
Lang, 2000; Hamm et al., 2003). Third, specific affective picture contents are associated with distinct variations in face muscle activity (electromyography, EMG).
That is, whereas the corrugator supercilii or ‘frown’ muscle is associated with increased responses specifically while watching unpleasant images (see Fig. 1.2, upper panel, left), the zygomatic or ‘smile’ muscle displays heightened activity when being confronted with pleasant picture contents (see Fig. 1.2, upper panel, middle; e.g.,
Bradley & Lang, 2000). Finally and most importantly, it is reliably observed that the startle reflex is modulated as a function of stimulus valence. The startle reflex is a defensive reflex evoked by administration of an aversive probe, typically a short noise burst. It is assessed by measuring the strength of the associated eye blink. Specifically, as compared to neutral pictures startle reflex magnitude is decreased when watching pleasant images, while potentiated when viewing unpleasant stimuli (e.g., Bradley &
Lang, 2000). This provides empirical corroboration for theoretical considerations assuming that the basic motive-systems are linked to associations, representations and action programs, which are more likely to be activated in the presence of congruently valenced environmental stimuli (Lang et al., 1997).
Figure 1.2
Illustration of the relationship between several peripheral physiological measures and the valence and arousal dimensions, respectively (from Lang et al., 1998a)
Functional magnetic resonance imaging
Presumably, emotional modulations are not limited to the autonomous behavioral output channels. They should also be evident during the preceding perceptual and evaluative processing of emotional cues. A powerful method to study according effects of emotion modulated processing at the neural level is provided by
functional magnetic resonance imaging (fMRI). Measuring regionally specific metabolic changes associated with neural events, fMRI allows identifying the anatomic locus of emotion-related processes with high spatial accuracy. Essentially, MRI takes advantage of some fundamental physical characteristics of hydrogen protons (a.k.a. “protons”) which exist in abundance in biological tissue. As first shown independently by the later Nobel Prize winners Bloch and Purcell (Bloch et al., 1946;
Purcell et al., 1946), various atomic nuclei – including hydrogen – possess both a small magnetic moment, as well as an angular momentum, a property called ‘spin’. As a result, when placed into a strong external magnetic field the spin axes of protons align with that magnetic field’s orientation. Whereas some protons align in parallel direction, some will do so in the opposite way, i.e. anti-parallel. After some time, the entire system will have reached equilibrium, the tissue is said to be magnetized.
However, the protons’ axes are not in perfect parallel alignment with the external magnetic field but are rather slightly tilted, much like the earth revolving the sun (called ‘precession’). Thus, some of the protons can assume a higher energetic state when a radio pulse having the protons’ specific precession frequency is applied.
Afterwards, these protons return to equilibrium by emitting the absorbed energy.
Essentially, this is the basis of the signal measured during MR-scanning. Lauterbur (1973) was the first to suggest that this phenomenon could be utilized to form images.
Whereas the first MR-scanners were put into operation by the late 70’s, it wasn’t until the early 90’s when MRI was first used to conduct functional studies. In a seminal study, Ogawa and colleagues (Ogawa et al., 1990) demonstrated for the first time that due to oxygenation variations blood can be used as endogenous contrast agent to trace regions of increased activity in the brain (‘blood oxygenation level dependent’, BOLD-effect). It was finally Logothetis and his colleagues (2001) who could convincingly link the metabolic changes during the BOLD-response to actual neuronal activation differences, paving the way for cognitive neuroscience.
Applying fMRI in the field of emotion research, it could be demonstrated that affectively salient picture stimuli are accompanied by distinct neuronal activation patterns. When viewing emotionally arousing pictures of either valence as compared to neutral ones, enhanced brain activity is consistently observed in several areas implicated in visual information processing, including the occipital and fusiform gyri, as well as the vicinity of the superior temporal sulcus when contrasting emotional and neutral picture contents (e.g., Bradley et al., 2003; Junghöfer et al., 2005; Lang et al.,
1998b; Sabatinelli et al., 2004; Sabatinelli et al., 2005; see Fig. 1.3). Furthermore, recent studies observed that the signal changes in the visual cortex varied strongly with self reports of emotional arousal. Specifically, pleasant and unpleasant pictures high in arousal were associated with strong and widely distributed visual cortical activations compared to low arousing emotional or neutral materials (Bradley et al., 2003; Junghöfer et al., 2005). Using angry as compared to affectively neutral prosody as experimental stimuli, enhanced activation of associative sensory cortex has not only been observed in the visual, but also in the auditory modality (Grandjean et al., 2005).
Finally, emotional modulation of neuronal activity is also reported for several structures of affective cortical and sub-cortical networks, including the amygdala (e.g.
Hadjikhani & de Gelder, 2003; Hamann et al., 2004; Junghöfer et al., 2005; Norris et al., 2004; Sabatinelli et al., 2005; Vuilleumier et al., 2001; Whalen et al., 1998), the insula (e.g., Junghöfer et al., 2005; Simmons et al., 2004), as well as orbito-frontal cortex (e.g., Karama et al., 2002; Murphy et al., 2003; Nitschke et al., 2004;
Vuilleumier et al., 2001).
Figure 1.3
Illustration of increased BOLD-activation in extrastriate visual cortex while watching emotionally arousing pictures (from Sabatinelli et al., 2004)
Event-related brain potentials
Brain imaging studies are helpful for demonstrating selective emotion processing and for identifying the neuroanatomical substrates mediating the underlying processes. However, due to limited temporal resolution they only give sparse insight into their temporal dynamics. One means to overcome these limitations is provided by event-related scalp potentials (ERP). Measuring electro-cortical activity at the scalp surface, ERPs make it possible to trace the unfolding cascade of neural
emotion processing with high temporal accuracy. Thus, ERP measures allow the examination of the selective processing of emotional cues at the level of specific sub- systems. Event-related brain potentials provide a voltage measurement of neural activity that can be recorded non-invasively from multiple scalp regions (Birbaumer et al., 1990). More specific, ERPs are considered to reflect summed postsynaptic potentials generated in the process of neural transmission and passively conducted through the brain and skull to the skin surface where they contribute to the electroencephalogram (EEG). Since ERPs are usually hidden in the larger background EEG activity, it is necessary to use multiple stimulus presentations and the calculation of stimulus-locked signal averaging to extract the ERP signal from the background EEG activity. Biophysical considerations suggest that large-amplitude ERP components reflect widespread, synchronous sources in cortical regions (Lutzenberger et al., 1987). Brain activity locked to the processing of a stimulus becomes apparent as positive and negative deflections in the ERP waveform. The amplitude and latency of specific ERP components provide information regarding the strength and time course of underlying neural processes. Furthermore, given appropriate spatial sampling (Tucker, 1993) the topography of ERP components can be used to estimate the neural generator sites by advanced analytic tools such as Current-Source-Density (CSD;
Perrin et al., 1987) or L2-Minimum-Norm-Estimate (L2-MNE; Hamalainen &
Ilmoniemi, 1994; Hauk et al., 2002).
Until today, several components have been identified indexing distinct emotion sensitive processes. The first ERP component reflecting the differential processing of emotional compared to neutral stimuli is the early posterior negativity.
The temporal and spatial appearance of the early indicator of selective processing of emotional pictures is illustrated in Figure 1.4 presenting data from research utilizing the rapid serial visual presentations (RSVP) of IAPS-pictures (Schupp et al., 2004b).
A pronounced ERP difference for the processing of emotionally arousing and neutral pictures developed around 150 ms which was maximally pronounced around 250 - 300 ms. This differential ERP appeared as negative deflection over temporo-occipital sensor sites and a corresponding polarity reversal over fronto-cental regions. Despite differences in the overall topography, research presenting pictures discretely and with longer presentation times (1.5 s and 1.5 s intertrial interval) also demonstrated a more pronounced negative potential for emotional pictures in the same latency range over temporo-occipital sensors (Schupp et al., 2003b; Schupp et al., 2004c). Additional
analyses served to test the prediction derived from the biphasic view of emotion that the differential processing of pleasant and unpleasant cues varies as a function of emotional arousal. Consistent with this notion, the EPN covaried with the arousal level of the emotional pictures. Specifically, highly arousing picture contents of erotic scenes and mutilations elicited a more pronounced posterior negativity compared to less arousing categories of the same valence (Junghöfer et al., 2001; Schupp et al., 2004b).
Figure 1.4
Illustration of the time course (upper panel) and the topographical distribution (lower panel) of the Early Posterior Negativity (EPN, from Schupp et al., 2003a)
As shown in Figure 1.5, subsequent to the modulation during perceptual encoding, it is consistently observed that emotional (pleasant and unpleasant) pictures elicit increased late positive potentials over centro-parietal regions, most apparent around 400-600 ms post stimulus (also referred to as P3b; Amrhein et al., 2004;
Cuthbert et al., 2000; Keil et al., 2002; Palomba et al., 1997; Schupp et al., 2000;
Schupp et al., 2003b; Schupp et al., 2004a; Schupp et al., 2004b). This LPP modulation appears sizable and can be observed in nearly each individual when calculating simple difference scores (emotional – neutral). In addition, the LPP is also specifically enhanced for pictures that are more emotionally intense (i.e., described by viewers as more arousing, and showing a heightened skin conductance response).
Extending the valence by arousal interaction, a recent study examined the LPP amplitude associated with the processing of specific categories of human experience.
Focusing on specific pleasant and unpleasant picture contents, it was found that picture contents of high evolutionary significance such as pictures of erotica and sexual contents and contents of threat and mutilations were associated with enlarged LPP amplitudes compared to pictures categories of the same valence but less evolutionary significance (Schupp et al., 2004a).
Figure 1.5
Illustration of the time course of the Late Positive Potential (LPP; from Schupp et al., 2004b)
Motivated attention
Together, these effects of emotion-modulated selective neuronal processing may be integrated by considering them from an attentional perspective. Specifically, efficient preparation and organization of appropriate behavioral responses require the rapid extraction of critical information from the environment. In this respect, emotional cues direct attentional resources (Öhman et al., 2000a). In general, mechanisms of selective attention assure the prioritized processing of some objects,
events, or locations and multiple avenues to command attention are indicated by distinguishing among active and passive forms of attentional control (Öhman et al., 2000b). In passive attention, the power to capture attention derives from simple qualities of the stimulus such as intensity, suddenness of onset, or novelty. In active attention, priority processing reflects the intentional effort to look for selected stimuli based on instructions, self-generated intentions, or associative learning. In addition, certain kinds of stimuli trigger selective attention due to their biological meaning.
Organisms respond to environmental cues according to their emotional/motivational significance (Lang et al., 1997; Öhman et al., 2000b). Drawing on theoretical foundations provided by the two-dimensional model of emotion, the attention capture of emotionally relevant stimuli has been dubbed ‘motivated attention’ referring to a natural state of selective attention, “… similar to that occurring in an animal as it forages in a field, encounters others, pursues prey or sexual partners, and tries or avoid predators and comparable dangers” (Lang et al., 1997, p. 97). An evolutionary perspective suggests that this form of attentional control is highly adaptive for survival giving primacy - in terms of attentional selection and response - to appetitively and aversively relevant events (Lang et al., 1997). Furthermore, motivated attention may strongly reflect our evolutionary heritage and may therefore proposedly be most apparent for stimuli with high evolutionary significance, that is, prototypical stimuli related to threat and survival strongly engaging basic motivational systems (Bradley et al., 2001).
The present thesis
Building upon this framework provided by the motivated attention account, the present thesis pursues two lines of research to shed light on the foundations of neural emotion processing from two different angles:
a) by assessing the dependency of emotion processes on the nature of the eliciting stimuli used in laboratory research
b) by detailing the specifics of the sequential processing of affectively evocative picture materials
The influence of evolutionary stimulus significance
To examine effects of emotion modulation, it is necessary to induce emotion related processes in the laboratory. For this purpose, previous research has largely relied on emotional pictorial stimuli and the picture viewing paradigm. The vast majority of studies reported to date utilized pictures depicting emotionally arousing naturalistic scenes (for an overview see e.g., Bradley & Lang, 2000; Schupp et al., in press), affective facial expressions (e.g., Schupp et al., 2004c; Pourtois et al., 2004;
Vuilleumier, 2002) or emotional body posture (for an overview see de Gelder, 2006).
According to the conception of several theorists (Bradley et al., 2001; de Gelder, 2006;
Lang et al., 2000; Öhman & Mineka, 2001; Vuilleumier, 2002), these stimulus classes possibly share an inherent affective significance which may be rooted in genetic predisposition and may thus reflect the phylogenetic heritage of humans. As suggested by Öhman and Mineka (2001), these stimuli might accordingly be characterized by a general biological preparedness, in effect promoting selective perceptual processing, attention capture and rapid aversive learning. However, this assumption is mainly theoretical and still awaits explicit empirical substantiation. Thus, it would be informative to extend the current knowledge of emotion processing to new stimulus classes which have not yet been investigated in the context of neuroscientific emotion research. To illuminate this issue, the first part of the present thesis reports two studies examining the neural processing of symbolic expressive hand gestures with culturally specific emotional meaning.
The investigation of this new emotional stimulus class yields the promise to give insight into several questions. First, information could be provided regarding the dependency of emotional processes on the evolutionary preparedness of the eliciting stimuli. Since hand gestures do not possess culturally universal meaning, their identification as affectively relevant environmental cues would essentially need to be a result of ontogenetic acquisition. This would reveal whether emotional significance inherent to external stimuli is strongly dependent on biological preparedness or whether it can also be acquired by our brain during the course of life. Second, replicating the characteristic neural signatures of emotion processing utilizing this new stimulus class would provide further corroboration of theoretical believes regarding the prioritized processing of affectively salient stimuli by means of motivationally guided attention. And third, additional clarification could also be provided regarding
the influence of physical stimulus features on the observed emotion effects. Due to the great visual complexity of conventional stimulus materials, it still remains unclear to what extent these effects are influenced not only by semantic meaning but rather by physical stimulus characteristics. These new experimental materials could therefore provide better physical stimulus control.
Utilizing functional magnetic resonance imaging, the first study of the present thesis examines whether selective neuronal processing specifically in associative visual cortex is also observable as a function of the emotionality of highly- symbolic expressive hand gestures. Following up on this pilot-study, study number two constitutes the attempt to corroborate the findings from study one by investigating the temporal specifics of emotional gesture processing in more detail. Specifically, by utilizing EEG methodology it is tested whether emotional expressive hand gestures are subject to rapid emotion discrimination as demonstrated with naturalistic scenes in previous research (i.e., EPN, Junghöfer et al., 2001). Taken together, both studies may thus provide methodologically converging evidence for the prioritized processing of emotionally salient gestures.
The effects of sequential picture perception
In the second part of this thesis, an attempt is made to shed light on the exact nature of the processes underlying typically observed effects of neural emotion processing. Prior research has adopted a paradigm that utilizes fast repetitive picture presentations (rapid serial visual presentation, RSVP) in which images of varying emotional content are shown in quick succession without intertrial gaps in between (e.g., Junghöfer et al., 2001; Schupp et al., 2003a; Schupp et al., 2006; Wieser et al., 2006). Utilizing this paradigm was originally intended to allow for the quick examination of automatic emotion discrimination which would employ minimal strain on the experimental participant. Before, this issue had gained special importance in the context of clinical research demonstrating that patients suffering from various mental conditions show characteristic deviations in terms of neuronal processing patterns when viewing clinically relevant emotional picture stimuli. However, the use of this paradigm has several implications which have not yet been addressed by existing research, possibly the most obvious one being the specific influence of the sequential processing of emotional pictures.
Specifically, the motivated attention account suggests that during the processing of the continuous picture stream emotional images consistently draw more attentional resources than neutral ones. Accordingly, the question arises whether the involved processes are completed by the time the next picture is presented or whether they interact with subsequent processing. Relating to research showing that stimuli competing for capacity limited processing resources interfere with each other (e.g., Keysers & Perrett, 2002; Pessoa et al., 2002a; Potter et al., 2002), a motivated attention perspective would therefore predict deteriorating effects of emotional pictures on subsequent processing. Alternatively, it might also be expected that sequential stimuli of congruent valence do not result in interference but rather in the facilitation of subsequent picture processing. Such an outcome might be derived from literature showing that behavioral responses to emotional stimuli are facilitated when being primed by cues of congruent emotional valence (for an overview see Klauer &
Musch, 2003). Thus, to be able to properly appreciate results from RSVP studies, it is essential to clarify this issue.
Pursuing this aim, the second part of this thesis presents two studies which utilize EEG methodology to assess whether corresponding effects act on the different stages of selective emotion processing. Applying ‘classical’ RSVP parameters in terms of presentation frequency, study three investigates this research question by specifically assessing the early stage indexed by the EPN. Since it has been proposed that attention may act upon separate stages (Luck & Hillyard, 2000), study four extends this rationale by applying longer presentation rates, in effect enabling to examine the later stage of emotion processing indexed by the LPP.
In summary, the present thesis provides empirical neuroscientific evidence to further the deeper understanding of emotion processes in the human brain. This objective is pursued by integrating two converging avenues to the research subject – one approaching emotion from the characteristics of external evocative stimuli, the other dealing specifically with the nature of the elicited processes itself.
Emotion in your hand: The selective processing of affective hand gestures in visual cortex
Abstract
Emotionally salient stimuli such as naturalistic scenes, facial expressions and emotional body language elicit increased neuronal activation in associative sensory cortex, as well affective cortical and sub-cortical networks. It has been suggested, that the affective quality of these stimuli is largely a result of evolutionary predisposition and that their selective perceptual encoding may thus be biologically prepared. The present study tested whether comparable neuronal processing patterns are also elicited by expressive hand gestures, bearing distinct emotional yet highly symbolic meaning.
During functional scanning, 33 subjects passively viewed blocked picture presentations of pleasant, neutral and unpleasant hand gestures. Consistent with a priori assumptions, emotionally meaningful gestures were associated with increased BOLD activations in extrastriate visual cortex, as well as in parietal areas. These results suggest that emotional significance detected by the brain is not exclusively founded in the genetic heritage of the human species but may also get attached to highly symbolic stimuli by social learning.
Introduction
An increasing number of studies confirm that the brain systematically discriminates between emotionally salient and neutral environmental stimuli. From an evolutionary point of view, this is highly profitable for organisms because as a result optimally adaptive behavior enhances the organism’s survivability and reproductive success (Cacioppo et al., 1999; Lang et al., 1997; Öhman et al., 2000a). Evidence in support of such selective emotion processing is provided by brain imaging studies showing that emotional as compared to neutral stimuli are associated with increased neuronal processing in various brain structures. In particular, enhanced brain activity is consistently observed in several areas implicated in visual information processing, including the occipital and fusiform gyri, as well as the vicinity of the superior temporal sulcus when contrasting emotional and neutral picture contents (e.g., Bradley et al., 2003; Junghöfer et al., 2005; Lang et al., 1998b; Sabatinelli et al., 2004;
Sabatinelli et al., 2005). Furthermore, recent studies observed that the signal changes in the visual cortex varied strongly with self reports of emotional arousal. Specifically, pleasant and unpleasant pictures high in arousal were associated with strong and widely distributed visual cortical activations compared to low arousing emotional or neutral materials (Bradley et al., 2003; Junghöfer et al., 2005). Moreover, affectively enhanced activation of associative sensory cortex has not only been observed in the visual, but also in the auditory modality (Grandjean et al., 2005). Finally, emotional modulation of neuronal activity is also reported for several structures of affective cortical and sub-cortical networks, including the amygdala (e.g. Hadjikhani & de Gelder, 2003; Hamann et al., 2004; Junghöfer et al., 2005; Norris et al., 2004;
Sabatinelli et al., 2005; Vuilleumier et al., 2001; Whalen et al., 1998), the insula (e.g., Junghöfer et al., 2005; Simmons et al., 2004), as well as orbito-frontal cortex (e.g., Karama et al., 2002; Murphy et al., 2003; Nitschke et al., 2004; Vuilleumier et al., 2001).
A great number of these studies used picture materials depicting contents of high evolutionary relevance, such as mutilated bodies, threatening animals and erotic scenes. However, being a species relying heavily on complex social interactions humans not only need to identify and react to environmental stimuli signaling immediate physical danger or reproductive opportunity. Instead, it is also of eminent importance to perceive socially relevant signals, in effect allowing for optimized
behavior in a complex social environment. Accordingly, verbal and non-verbal social cues provide information about the motivational and emotional states of interaction partners, as well as the surrounding environment (fearful facial expressions may e.g.
indicate immanent danger). Consequently, they can be used to predict intentions, and likely future interactions (see Keltner & Kring, 1998). Because of this, it has been stressed that social and emotional processes are strongly interdependent and that socially relevant stimuli may thus be characterized by inherent emotional significance (Keltner & Kring, 1998).
In the context of emotion research, special importance has been attributed to communicative aspects of the face and the eyes as this class of non-verbal behaviors is regarded to have a major impact on the expression of emotion and on the regulation of inter-personal interactions (see Ekman & Oster, 1979; Öhman, 1986). Like naturalistic pictures, emotional facial expressions are processed preferentially as compared to neutral ones. In behavioral experiments, schematic threatening faces are recognized faster than friendly ones within an array of neutral distractors (Öhman et al., 2001b).
On the neuronal level, studies using EEG-measures found distinct electro-cortical indices of selective emotion-processing to be sensitive to the emotionality of faces (Schupp et al., 2004c). Also, a great number of studies have shown that emotional faces automatically attract attention (see e.g. Pessoa et al., 2002b; Vuilleumier, 2002).
Corresponding to results obtained using naturalistic scenes, emotional faces elicit heightened neuronal processing already in visual sensory areas, as well (e.g., Breiter et al., 1996; Pessoa et al., 2002a; Vuilleumier et al., 2001). Interestingly, this increased neuronal sensitivity to emotional facial expressions is not unique to the human species.
Face-sensitive neurons in the temporal visual cortex of macaque monkeys also respond with an increased firing-rate when viewing emotionally aroused as compared to neutral faces (Sugase et al., 1999).
Additional evidence for the preferential processing of affect-laden social cues is provided by studies examining ‘emotional body language’ (EBL, de Gelder et al., 2004; Hadjikhani & de Gelder, 2003; Meeren et al., 2005; Stekelenburg & de Gelder, 2004; for an overview see de Gelder, 2006). EBL involves hand and arm movements, as well as body posture, in combination providing revealing insights into somebody else’s emotional condition. Corresponding to the processing of emotional faces, the perceptual analysis of emotional body language as compared to neutral
postures is associated with increased neuronal activation of higher-order visual cortical areas (de Gelder et al., 2004; Hadjikhani & de Gelder, 2003). This was also confirmed by a recent study by Grosbras and colleagues (2005), reporting increased BOLD activity in temporo-occipital cortical regions when subjects viewed hand movements carried out in an angry as opposed to an emotionally neutral fashion. Even though no direct electro-cortical differentiation between emotional and neutral body postures has been reported to date, ERP-measures are still sensitive to the emotionality of body language, as demonstrated by an emotional congruency effect of faces and body postures apparent at the P1-component, starting at around 100 ms and observed over occipital leads (Meeren et al., 2005).
Altogether, emotional facial expressions, body language and naturalistic scenes share an emotional significance that may inherently be rooted in the phylogenetic heritage of our species. This idea was formulated early on by Darwin (1872) who proposed that emotions would not be private mental episodes but would instead have adaptive behavioral value, promoting favorable actions under given situational circumstances. In correspondence with this assumption, the evolutionary significance of such environmental cues promotes selective perceptual processing, attention capture and rapid aversive learning (see Öhman & Mineka, 2001). As Öhman and Mineka (2001) suggested, such stimuli might be considered to be ‘biologically prepared’. Specifically, whereas their inherent emotional significance may not be entirely innate, it may rather determine a general readiness to respond to fear-relevant stimuli such as angry faces or snakes and to quickly learn the association with according emotional states in aversive contexts. The observation that phobias most prevalently occur in dependency of evolutionarily significant fear cues also points in this direction (e.g., Marks, 1969). Furthermore, these stimuli appear to be greatly characterized by cultural universality. More specific, pictures of mutilated bodies or snakes may evoke aversive emotional reactions in people all around the world, regardless of their respective cultural background. Also, emotional facial expressions, at least involving those considered by Ekman to represent basic emotions, have been reported to be unambiguously identifiable for individuals of widely differing cultural background, including people of European, African and Asian origin (Ekman & Oster, 1979). Therefore, certain environmental cues as well as species-specific expressive emotional non-verbal behavior having its roots in adaptive actions (see also Schmidt &
Cohn, 2001) might be considered to be ’biologically prepared’ (see Bradley et al.,
2001; Lang et al., 2000; Öhman, 1986; Öhman & Mineka, 2001) and may thus be an integral part of each species’ evolutionary heritage.
Based on these considerations, it remains an open question however to what degree the signature of neuronal emotion processing may not only reflect the evolutionary preparedness of the eliciting stimuli, but may also rather be a function of culture-specific social learning. Specifically, to what extent are evolutionarily insignificant stimuli whose emotional meaning is culturally mediated also subject to preferential processing in the brain? Moreover, does such selective encoding resemble typically observed processing patterns, i.e. regarding involved brain areas and time course of the underlying processes? Accordingly, we suggest that another class of non- verbal communicative signals may serve as an adequate means to investigate these questions. More specific, simple static hand gestures are widely used across all cultures all over the world (for a comprehensive overview about worldwide use and meaning of numerous expressive hand gestures see Morris, 1994). Similar to the aforementioned social cues they are also carriers of socially important and emotionally relevant information. Moreover, meaningful hand gestures have been shown to be processed semantically (Wu & Coulson, 2005) and are associated with distinct neuronal activation patterns (Gallagher & Frith, 2004; Nakamura et al., 2004).
However, as opposed to the emotional non-verbal signals investigated in prior research, the meaning attached to hand gestures is highly symbolic and conventionalized, being strongly dependent on a shared cultural context between the sender and the perceiver of the non-verbal message. Therefore, whereas a particular hand gesture may be associated with a strong emotional meaning in one cultural communication context, it may not be associated with any semantic meaning at all in another culture.
Thus, in the present study we investigated the neuronal correlates of the processing of expressive hand gestures by means of functional magnetic resonance imaging. We assumed a priori that hand gestures posses culturally mediated emotional significance and that this would be reflected both at the behavioral, as well as the neuronal level. Behaviorally, we expected a differentiation between the pre-selected gesture categories on self-report scales judging the emotionality of the used pictures.
In terms of hemodynamic responses elicited by the different hand gestures, we expected an increase in neuronal activation while viewing emotional as compared to
neutral gesture categories. Specifically, since various classes of emotional stimuli have been consistently reported to be associated with stronger activation in associative sensory areas, we formulated our strongest anatomical hypothesis for the extrastriate visual cortex. Additionally, increased activation during the viewing of emotional gestures was also expected in affective cortical and sub-cortical networks frequently reported in previous studies, including the amygdala and the insula, as well as orbito- frontal cortex.
Method
Participants
33 subjects participated in the study (16 male). All of them received monetary compensation or course credit and a copy of their structural T1-MR-image for their participation. Participants were between the age of 19 and 33 years (M = 23.1) and were naïve with respect to the study’s aims and hypotheses, i.e., had no prior experience with the used picture stimuli. All of them were familiar with the meaning of the middle finger jerk and thumb up gesture. However, three subjects (all female) reported exclusive primary meanings of the finger points related to threat (pointing towards a companion) or punishment and were therefore excluded from further analyses. Subjects were also selected with respect to the specific requirements set by the MR-safety procedures, i.e. no non-removable magnetic objects of any kind within their body and no history of epilepsy and claustrophobia. Also, in order to fit the stimulation goggles over the subjects’ eyes within the head coil, only subjects were selected with average or small head size. Before the experiment, subjects read and signed forms of consent containing detailed information regarding MR-safety- procedures and possible risks linked with magnetic resonance imaging examinations.
Stimulus Materials
A stimulus set of 24 pictures overall comprised of three differently valenced meaningful hand gestures was generated. Hand gestures were chosen according to a number of criteria:
1) All gestures should be associated with a distinct semantic meaning, in a broad sense representing emblems (see Ekman & Friesen, 1969). 2) Their meaning should be carried by a static hand gesture, neither involving dynamic hand movements nor a hand gesture related to other body parts. Importantly, no accompanying speech should be necessary to understand this meaning. 3) Their meaning should be shared by the vast majority of people belonging to the cultural context common to the examined research participants. 4) The gestures should be clearly distinguishable in terms of inherent emotional significance. 5) All gestures should be perceptually as similar as possible. Specifically, each gesture should only consist of the back of the hand and a single extended finger in various orientations.
Based on those criteria, we pre-selected three different hand gestures bearing pleasant, neutral and unpleasant emotional meaning, respectively:
First, the pleasant gesture was realized by the “thumb up” sign which displays a horizontally oriented hand with vertically extended thumb (see Fig. 2.1). In Germany, as well as in many other countries, this gesture is understood as a sign of approval or commendation (see Morris, 1994). The origin of this gesture is traced back to gladiator fights in ancient Roman Colosseum in which the covered thumb saved the life of gladiators because of the audience’s approval. Due to mistranslations or ignorance, the original “thumb thrusted” and “thumb covered up” changed to “thumb up” and “thumb down” gestures, meaning something good or bad.
Second, the neutral gesture was implemented by the pointing forefinger, being comprised by a horizontally oriented hand with horizontally extended index finger (see Fig. 2.1). In many parts of the world including Germany, this gesture is understood as a request to look or orient towards the indicated direction. Conceivably, this gesture might culturally even be rather universal. However, many cultures around the world do employ different gestures as pointing display (see Morris, 1994). Also, it might be arguable that this gesture effectively does have emblematic meaning. In prior assessment however, subjects were able to clearly verbalize the attached meaning based on static pictures with no accompanying speech and conversation.
Finally, the unpleasant gesture category was represented by the “middle- finger jerk”, realized by a vertically oriented hand with a vertically extended middle finger (see Fig. 2.1). In Germany, as well as in many other western countries, this sign is understood as an explicit gesture of insult (see Morris, 1994). The sexual connotation of this gesture is apparent by the middle finger symbolizing an erect penis and the adjacent digits, curled on either side of it, representing the testicles. This offensive gesture is one of the oldest sexual insults known and has been tracked in classical records to Ancient Rome and Greece (see Bäuml & Bäuml, 1975; Morris, 1994). Apparently, Diogenes insulted Demosthenes with this gesture and Caligula is said to have strongly insulted subordinates showing it when offering his hand to be kissed. The meaning of the gesture, in Ancient Rome referred to as ”digitus impudicus” (the indecent finger), appeared invariant across time and nowadays still provokes strong social control and disdain in the public.
Figure 2.1
Examples of the picture stimuli used in the present study.
Four women and four men served as models for the different picture stimuli.
Accordingly, the picture set consisted of the photographed hands of eight different people, which were the same for each stimulus category. All gestures were displayed with the back of the hand rotated towards the viewer and with a neutral single-colored grey-blue background (color: R=56, G=64, B=80; 24 Bit). The exact location of each hand within a square-shaped image was kept constant by approximately positioning the back of the hand to the center of the picture. All pictures also appeared mirrored along the vertical axis to control for possible lateralization effects.
Experimental Procedure
Following experimental instruction, subjects lay on the patient table of the MR-scanner and the visual stimulation device (VisuaStim XGA; Resonance Technology, Inc., Northridge, California, USA) was positioned within the head coil in front of the subjects’ eyes. After optimizing the visor’s position, it was fixated with adhesive tape to the frame of the head coil. When necessary, the subjects’ vision was corrected with lenses getting attached to the VisuaStim-system. To protect the participants from loud scanner noise, they were also required to fit ear-plugs into their ears. In addition, to minimize artifacts due to head movements, the subjects’ head was subsequently fixated sideways using rubber-foam pads. During MR-measurement, subjects were able to call for assistance via the intercom system or to interrupt the experiment at any time by pressing an alarm bell.
After measurement of two short overview scans, subjects were instructed to passively view the experimental stimulation and to lie as still as possible while being
scanned. During functional imaging, stimuli were presented in blocks of 12 s each, which followed each other in a pseudo-randomized order. Within each block, all eight different pictures of each category were presented twice, resulting in 16 picture presentations overall. Hereby, all eight different images were shown in randomized order before they were repeated. Each picture was shown for 330 ms, immediately followed by a 420 ms period of a blank screen. This resulted in a flickering of the single stimuli and was intended to specifically maximize the magnitude of the visual hemodynamic brain response.
In addition to the three experimental conditions, a separate baseline- condition showing a blank screen with a small white fixation cross in the center was included. This served the purpose of subsequent calculation of main effects of visual stimulation for each category and to empirically determine a posteriori functional regions-of-interest representing the overall effect of visual processing. To control for lateralization effects, each category was shown half of the time as flipped images, mirrored along the vertical middle axis. The two versions of each experimental condition were presented equally often over the course of the experiment and their occurrence was randomized over the entire stimulation sequence. Subsequently, the two versions were collapsed to one category. Within a single block, only pictures of the same version were shown.
Within the overall sequence of blocks, all four conditions were presented in randomized order before they could be repeated the next time. Also, no condition could follow a block of the same category. With this exception, the transition frequencies of all remaining two-block sequences were kept constant. Over the entire experimental session, each category was shown 14 times (7 times flipped) resulting in 56 blocks overall and adding up to a total of 12 minutes of functional scanning.
Following MR-scanning, subjects were asked to rate the viewed gestures according to their perceived pleasantness and arousal using the Self-Assessment Manikin (SAM, Bradley & Lang, 1994). The different gestures were each presented within an overview slide, being comprised of all different instances of each picture category. For exploratory reasons and to provide a broader pictorial and interpretative frame to base the subjective evaluations on, subjects had to rate both the three gestures used as experimental categories, as well as nine additional ones, all of them both in their original as well as flipped versions. Finally, subjects were asked to write down
the semantic meaning associated with each gesture. They also had to indicate how strongly they would link this meaning to the respective gesture (ranging from 1=not at all to 5=very strongly).
MR - scanning parameters
Scanning was conducted using a 1.5 Tesla Philips Intera MR-System equipped with Power Gradients. For functional scanning, a T2*-weighted Fast-Field- Echo, Echo-Planar-Imaging (FFE-EPI) sequence utilizing parallel scanning technique (SENSE; Pruessmann et al., 1999) was used. Images were acquired in transversal orientation parallel to the AC-PC line. Each dynamic volume consisted of 36 slices measured in interleaved acquisition order with a thickness of 3.5 mm each with no inter-slice gap. In-plane resolution was 2.9 x 2.9 mm with a squared Field-Of-View at a size of 230 mm (acquisition matrix of 80 x 80 voxels; TE = 40 ms; flip angle = 90°).
One complete whole-head scan was measured every 3 s (TR) and overall the sequence consisted of 230 continuously acquired volumes adding up to a total of 11:30 min of functional scanning. At the beginning of the experimental session, 6 dummy scans were acquired which were later on discarded to allow for T1-equilibration. After the experiment, a standard T1-weighted high-resolution structural scan was obtained (in- plane-matrix 256 x 256; voxel size 1 x 1 x 1 mm).
Data analysis
Statistical analysis of the functional images was conducted using Statistical Parametric Mapping (SPM2; Friston et al., 1995;). Preprocessing included correction for slice-time differences and spatial realignment to the first volume in the image series to adjust for head movements during the course of the measurement. Also, an unwarping algorithm was applied to correct image distortions typical for EPI-images due to local field inhomogeneity and susceptibility artifacts. Afterwards, the functional volumes were normalized to MNI standard stereotactic space and resampled to an isotropic voxel size of 3 mm. In a last step, images were spatially smoothed with a smoothing kernel of 8 x 8 x 9 mm full-width-at-half-maximum (FWHM).
After spatial preprocessing, the data were submitted to statistical analysis implementing the General Linear Model (GLM). The corresponding design matrix was comprised by three covariates-of-interest representing the experimental
conditions’ onsets and the duration of the different picture presentation epochs, as well as covariates-of-no-interest including the modeled response functions’ time and dispersion derivatives, six movement parameters obtained during realignment and one covariate incorporating an overall intercept to the model. The Null-baseline was modeled implicitly. The covariates-of-no-interest were included to improve overall model fit to the empirical data and to reduce residual error variance, in effect increasing overall statistical power of the model. This procedure was based on experience from previous studies, showing that in the context of passive picture viewing the statistical power gained by these covariates-of-no-interest’s inclusion into the model exceeds the statistical disadvantages associated with the loss of degrees-of- freedom. Before estimating the modeled regressors, a high-pass filter with a cutoff period of 128 s, as well as global scaling was applied to the data.
Following estimation of the overall model, planned contrasts-of-interest were calculated for each subject. These included separate comparisons of all experimental conditions with the Null-baseline to determine areas of visual activation (thumb >
Null; index > Null; middle > Null). These contrasts were entered into separate 2nd- level random-effects group analyses. Based on the resulting activation clusters, we defined a functional region-of-interest (fROI) to assess the experimental hypotheses in an anatomically and statistically more detailed manner. Specifically, after visual inspection of all basic contrasts four main clusters of activation were apparent in posterior cortical areas. Based on our a priori anatomic hypothesis, we identified the voxels of maximal activation within these clusters for each contrast. Then, spherical regions-of-interest with a radius of 8 mm were defined around each of those voxels. In a next step, all resulting regions-of-interest were collapsed across all contrasts. Finally, we identified voxels within these clusters which showed significant activation (voxel threshold: T > 2.46) for each contrast and rejected all others, resulting in a combined fROI being comprised of four distinguishable clusters in the bilateral medial cuneus and bilateral temporo-occipital cortex (see Fig. 2.2). This procedure was chosen to obtain a fROI which is unbiased by single experimental conditions. By simply contrasting all conditions together against the Null-baseline, any condition prompting stronger activations than the others would have contributed more strongly to the resulting ROI, selectively choosing voxels with according activation differences.
Therefore, the fROI used for the present analyses represents a conjunction contrast between all experimental conditions and the Null-baseline, in effect incorporating
voxels that are activated by each experimental condition separately. For statistical analysis, the average of the model estimation parameters of all positively activated voxels within this functional region-of-interest was extracted for each subjected and entered into a two-factorial ANOVA with repeated measurement on the factors picture category (thumb, index, middle) and brain laterality (left, right). Where appropriate, the Greenhouse-Geisser epsilon was utilized to correct for violations of sphericity.
Figure 2.2
Localization of the medial (left) and lateral (right) parts of the functionally defined Region-of-Interest entering analysis. ROIs are overlaid on a standardized anatomical template.
Moreover, we calculated a 2nd-level random-effects ANOVA corrected for sphericity violations incorporating each experimental condition contrasted against the Null-baseline for each subject. This analysis is statistically more conservative than the Region-of-interest analysis which was used to specifically assess the assumption of
increased activation in associative visual areas. Since we did not have a comparably strong anatomical hypothesis for other areas, we explored additional activation differences and selective activations in other regions by contrasting thumbs and middle fingers both against the emotionally neutral index finger category (thumbs >
index; middle > index), as well as against each other (middle > thumb) in this 2nd- level model. For these analyses, activations are reported if they exceeded a FDR- corrected (Genovese et al., 2002) threshold of p < .05 at the voxel level and if they reached a size of at least 11 voxels within each cluster (k > 10). Note that in order to exploratively illustrate the anatomical localization of the visual activation differences assessed in the fROI analysis, figures of the according 2nd-level contrast of (thumb >
index) are included using lower thresholds.
