This chapter has presented a computational implementation of PIT. This implementation together with the framework of Chapter 4 serve the pur-pose of making the concepts and ideas described in Chapter 3 more clear.
The computational model also serves as a proof-of-concept showing that PIT is sufficiently complete and consistent as a theory so that implemented instances of it are possible. Lastly, the computational model can run simula-tions of mental imagery tasks which offer concrete mechanistic explanasimula-tions of phenomena as well as specific predictions. In Chapter 6 both the the-ory and the computational model will be applied to show how PIT can account for the different phenomena of mental imagery that were previously presented in Chapter 2.
Figure5.4:Generationofamentalimage.Thefiguredepictshowthemodelsimulatesthegenerationofamentalimageofa scene.Theto-be-imaginedsceneisshownintheupperrightcornerofthefigure.Theworkflowfollowsthearrowsfromleft toright.Detailsaregiveninthetext.
Figure5.5:Inferenceinamentalimage.Thefiguredepictshowthemodelsimulatestheinferenceofnewinformationfrom amentalimage.Theworkflowfollowsthearrowsfromlefttoright.Detailsaregiveninthetext.
Chapter 6
Evaluation
This chapter presents the evaluation of the perceptual instantiation theory (PIT) and its computational model. The previously identified and discussed empirical phenomena of Chapter 2 are picked up in this chapter and the respective explanations and predictions of PIT are elaborated. The phe-nomena of mental scanning and eye movements are explained with support of simulations of the computational model while the explanations for mental reinterpretation and unilateral neglect are based on the theoretical descrip-tion of PIT.
6.1 Mental Scanning
The relevant empirical findings on mental scanning can be summarized as follows: 1) there is a robust mental scanning effect and 2) the specific pa-rameters of that effect, specifically the slope of the linear relation between reaction time and distance, are significantly affected by a number of different factors. I will first discuss the general mental scanning effect and then how it can be influenced.
6.1.1 The General Mental Scanning Effect
The mental scanning effect is the finding that humans show an approxi-mately linear relationship between the time of shifting their attention from one point to another in a mental image and the distance between these two points. The compared distance is the distance in the original stimulus, because distance in a mental image cannot itself be measured. PIT and its model explain this effect with what I will term the equivalence ex-planation1. PIT poses that mental imagery employs the same perceptual processes used during visual perception. Therefore, similar reaction time
1The enactive theory uses the same equivalence explanation to explain the general mental scanning effect (see Section 2.3.1).
Figure 6.1: Two stimuli depicting islands as used in mental scanning exper-iments. The island on the right additionally contains sign posts indicating inconsistent distances (Richman et al., 1979). Compare Figure 2.7.
patterns as observed during visual perception will also be observed during visuospatial mental imagery. During visual perception a shift in attention -whether it is realized as, for example, a saccade or a head movement - shows the property that shifting over a longer distance generally takes longer than shifting over a shorter distance. This property results simply from the phys-ical structure of the human body, e.g., one’s gaze cannot go from A to B without going through the intermediate space (also see Section 3.2.4 for a discussion on this constraint for perception and imagery). Therefore, PIT explains the general mental scanning effect trivially due to the common employment of the same perceptual processes as in visual perception.
The model of PIT reflects this explanation in more detail. A remembered stimulus is encoded in a scene which consists of the conceptual description of the objects and the spatial relations between them. The spatial relations of the scene reflect the remembered distances. For the generation and pro-cessing of the mental image of that scene, the respective mental concepts are instantiated with perceptual information. For this instantiation the mental concepts describing, for example, the spatial relation left-of and close are mapped onto a fitting attention shift implemented in the model as a vector.
The length of the vector corresponds to the distance qualitatively described by the spatial relation. The time for executing an attention shift is linear to the length of the vector that represents that attention shift. It follows that a spatial relation describing a longer distance will be instantiated using an at-tention shift over a longer distance. And that the execution of that atat-tention shift will take proportionally longer the longer the conceptually described distance is. Table 6.1 shows the model’s output for a mental scanning task using the left stimulus of Figure 6.1.
6.1.2 Variations of Mental Scanning
There are several different variations of the mental scanning paradigm which all show the general mental scanning effect. Yet, many variations signifi-cantly affect the slope of the linear relation between distance and time. That
Table 6.1: The model’s reaction times (RT) are averaged over 10 trials and include noise. Correlation: r = 0.94
Scan path RT Model Actual Relative Distance
house → tree 29.61 4.47
house → well 31.85 4.47
house → lake 12.14 2
lake → tree 27.67 4
lake → well 42.34 5.67
tree → well 35.49 4
Table 6.2: Mental scanning (Richman et al., 1979). Reaction times (RT) of the model are averages over ten trials and include noise.
Condition RT Experiment [s] RT Model
20 route 3.118 25.36
80 route 3.496 35.03
is, the speed of mental scanning is affected. I will only focus on one of those variations here, because the explanation that PIT gives for this specific case similarly applies to all other variations as well. The right side of Figure 6.1 shows a stimulus similar to the one used by Richman et al. (1979) for their mental scanning experiment. In contrast to the original mental scanning paradigm, the stimulus additionally contains two sign posts indicating cer-tain distances between entities. These distances are obviously inconsistent with the actual distances in the stimulus. Specifically, the distance between the hut and the tree is of the same length as that between the hut and the well. The sign posts, however, indicate that the distance between the hut and the tree is much longer, i.e., 80 miles, than the distance between the hut and the well, i.e., 20 miles. Participants were asked to mentally scan several routes using their mental image of the stimulus. A reliable effect of the sign posts on the reaction times of scanning was found so that participants took longer to scan along the “80 miles” route than along the “20 miles” route.
This experiment is an example of how cognitive penetration affects mental imagery. That is, the reaction times of the mental scanning task were signif-icantly affected by additional knowledge or belief of the participant; in this case the suggested, albeit incorrect, distances. Other variations of mental scanning can be seen as similar, as they also vary the participants’s belief or knowledge. For example, participants are made belief that mentally scan-ning over certain distances takes a certain time (Goldston et al., 1985) or
more indirectly, the experimenters are made belief that certain mental scan-ning outcomes are to be expected (Intons-Peterson, 1983). In both these cases the induced belief affects the general mental scanning effect in the expected way.
PIT’s explanation for the experiment of Richman et al. (1979) is based on the theory’s assumption that the mental concepts underlying mental im-agery are the result of the integration of multi-modal input. That is, they combine the sensory input of several senses including rather subtle infor-mation such as the suggested distances in the stimulus or different demand characteristics. For this reason, the conceptual description of the stimulus does not only reflect the metrical properties of the stimulus but also inte-grates the semantics of the sign posts suggesting different distances. That is, the spatial relation between, for example, the hut and the tree is not just top-left-of as it might be for the same mental scanning stimulus without sign posts, but rathertop-left-of, far withfar corresponding to the suggested “80 miles” distance. As elaborated above for the general mental scanning effect, such changes in the conceptual description lead to respective changes in re-action time. In this case the scanning time increases for the “80 miles” route and decreases for the “20 miles” route. Table 6.2 shows the resulting reac-tion times of the model for a simulareac-tion of the mental scanning variareac-tion of Richman et al. (1979).
6.1.3 Predictions
PIT makes two clear and testable predictions with respect to mental scan-ning. The first prediction results from the equivalence explanation for mental scanning posed by both the enactive theory and PIT. This prediction has, however, not been mentioned by Thomas (1999) or in other publica-tions on the enactive theory. The second prediction results from the fact that the mental concepts of PIT are the result of the integration of different inputs. These two predictions are discussed in the following.
Both the enactive theory and PIT assume that (visual) perception com-prises of perceptual actions which are also employed during mental imagery.
The linear relation between distance and time in mental scanning is ex-plained by the fact that shifting attention in perception also has this prop-erty. Actually, shifting attention using, for example, saccades does not show a strictly linear relationship between the time and the to-be-shifted-over distances. This is because other effects such as saccades reaching a higher acceleration and velocity over longer distances and, additionally, varying cor-rective movements for “overshooting” of saccadic eye movements also play a role. The prediction, made by both the enactive theory and PIT, is that the observed relationship between time and distance in mental scanning should upon closer observation show a stronger correlation to the (non-linear) one of attention shifts as employed in visual perception rather than to a strictly
linear one.
The second prediction that follows from PIT’s explanation of the vari-ations of mental scanning is that the scanning time is determined not only by the properties of the stimulus but, furthermore, by any other sort of related and even conflicting information obtained through other modalities.
That is, if additional information is systematically varied in both content and mode of communication, PIT predicts that the scanning time will be af-fected according to that additional information. PIT poses that the mental concepts on which mental images are based are multi-modal and integrative so that they combine related input and subsequently the outcome of the instantiation of these mental concepts will change. It should be expected that conflict resolution, i.e., the process of mapping conflicting spatial re-lations onto one set of perceptual actions during mental imagery, requires additional time.
A prediction about eye movements during mental scanning is described in Section 6.3.4 which discusses predictions of PIT with respect to eye move-ments during mental imagery.