• Identification: perceptual information is mapped onto mental concepts using the mapping of the VS-LTM, i.e., mental concepts are identified based on the available perceptual information (see Section 3.2.2).
• Interpretation: in visual perception and mental imagery an interpre-tation is drawn from the set of all identified mental concepts and their respective instances of perceptual information. This interpretation represent the most plausible alternative of all subsets. In visual per-ception the interpretation corresponds to what is consciously perceived (see Section 3.1.6).
• Mental image: in mental imagery this interpretation, i.e., the sub-set drawn from all identified mental concepts together with their in-stance of perceptual information, constitutes the mental image (see Section 3.2.1).
• Spatio-analogical character: the temporal properties of visuo-spatial mental imagery are similar to those of visual perception because the same perceptual actions are employed in both perception and imagery.
The temporal properties of these perceptual actions are determined by the physical properties of the human visual system. (see Section 3.2.4).
Perceptual actions are the basic actions of the human visual sys-tem. For example, eye movements, head movements, adjusting the lens, and covert attention shifts.
Perceptual information and perceptual actions are intimately connected in two ways. First, perceptual actions are used to retrieve perceptual infor-mation from the environment. Second, they are connected through mental concepts. Mental concepts are associative hubs linking perceptual infor-mation and perceptual actions. They are identified when a certain com-bination of perceptual information has been perceived. For example, the mental conceptsquare is identified when the features of a square have been perceived, e.g., four edges in a certain arrangement. The recognition of a square in visual perception corresponds to the full identification of the men-tal conceptsquare. Furthermore, a mental concept links to those perceptual actions which are used to perceive these defining features.
Through this mapping of perceptual information to mental concepts and the mapping of mental concepts to perceptual actions, a top-down guided active perception is realized. For example, the perception of three edges trig-gers the (partial) identification of the mental concept square. The mental concept then provides the respective perceptual action to test for the exis-tence of a further fitting edge to confirm the hypothesis of a square being present.
Figure 4.1 depicts these relationships between perceptual information, perceptual actions, and mental concepts. The figure contains the three functions that realize these relationships: 1) execute, 2) identify, and 3)select. These functions are elaborated in the following.
The functionexecuterepresents the execution of a perceptual action by the visual/motor system. The execution of a perceptual action yields per-ceptual information. The time of executing a perper-ceptual action depends on the physical constraints of the visual/motor system. Both overt and covert perceptual actions can be executed. Overt perceptual actions include specif-ically spontaneous eye movements. The functionexecute has to include a mechanism to decide whether a perceptual action is to be executed overtly or covertly.
The function identify identifies or partially identifies mental concepts based on the newly retrieved and previously retrieved perceptual informa-tion.
The function selectthen selects the next to-be-executed perceptual ac-tion based on the identified mental concepts as well as the available per-ceptual information. For example, consider a saccade towards the location of an anticipated fourth edge of a presumed square. The mental concept square provides the perceptual action to check for a fourth edge while the current perceptual information, e.g., the location and distance between the already perceived edges, is used to adjust the perceptual action so that the edge is checked for in the “correct” location. The selection of a perceptual
Figure 4.1: The cyclic process of select-execute-identify used during visual perception for object and scene recognition and during mental imagery for the instantiation of mental concepts. The cyclic process comprises 1) the selection of a perceptual action based on the identified mental concepts and available perceptual information; 2) the execution of the perceptual action to retrieve further perceptual information; and 3) the identification of mental concepts based on the available perceptual information.
action is subject to the principle of parsimony (see Section 3.2.2).
Now that the functions that govern the relationships between perceptual information, perceptual actions, and mental concepts have been clarified, they can be embedded into a large picture including the components of the framework of PIT. Figure 4.2 depicts this framework which also incorpo-rates the functions and operands of Figure 4.1. The figure contains two additional functions: 1)retrieve, and 2)interpret. These are discussed in the following.
The functionretrieveretrieves a set of mental concepts from conceptual long-term memory. This set of mental concepts is the conceptual descrip-tion of the to-be-imagined scene. These mental concepts are used to start the cyclic process of select-execute-identifyuntil at some point the func-tion interpret selects a subset of mental concepts with their instantiation of perceptual information, i.e., the perceptual information generated by the perceptual actions that were selected based on these mental concepts. This subset is the interpretation of what is perceived in the case of visual per-ception and of what is imagined in the case of mental imagery. That is, the mental imageis constituted by the drawn subset of mental concepts and their instances of perceptual information.
The function interpret selects what is considered the most plausible
Figure 4.2: The formal framework of PIT. The mental imagination of a scene starts with 1) the retrieval of a set of mental concepts from C-LTM which conceptually describe the scene; 2) these mental concepts are suc-cessively instantiated with perceptual information by the cyclic process of select-execute-identify; 3) an interpretation is drawn from all identified men-tal concepts with their instances of perceptual information; 4) this interpre-tation constitutes the mental image of the scene.
subset. For example, a set of perceptual information might correspond to both the mental concepts square and triangle as well as to a (schematic) house. Then the interpretation whether one “sees” a house or two geometric shapes depends on what is most plausible given the current situation and the background knowledge.
The components of the framework are 1) conceptual long-term memory (C-LTM), 2) short-term memory (STM), 3) visuo-spatial long-term memory (VS-LTM), and 4) visual/motor system. TheC-LTMis the long-term mem-ory of conceptual information. It consists of the set of all mental concepts.
The STM temporarily holds perceptual information and mental concepts relevant to the current process of perception or imagery as well as the mental image. The VS-LTM holds the procedural knowledge of how to interact with the environment in order to recognize familiar objects and relations.
This procedural knowledge is represented by 1) a mapping of mental con-cepts onto perceptual actions realized by the functionselectand 2) a map-ping of perceptual information onto mental concepts realized by the function identify. The visual/motor systemrepresents the human visual system which executes perceptual actions to retrieve perceptual information.