A Synthetic Proposal: Images as ”Perceptoid“ Signs

Backgrounded by LOPES’s analysis of the historical situation of theory formation, the main methodological characteristics of the general science of images (visualistics) presently in statu nascendi can be summarized as a proposal to systematically combine the two lines of tradition by conceiving pictures as perceptoid signs.¹⁰ Traditionally, a concept can be determined by giving a superimposed concept, and then adding the specific difference to the other subclasses. Applied to “perceptoid signs”, we distinguish generic characterizations that pictures have in common with all signs from the specific difference »perceptoid«, which allows us to distinguish pictorial signs from other kinds of signs, e.g., verbal signs. There are a couple of important consequences of that conception, which guide us in the following in order to gain an understanding of what precisely is meant by “perceptoid signs”.

3.2.1 »Sign« as Genus Proximum for Pictures

First: choosing »sign« as the superimposed concept clearly connects this position to the semiotic roots of picture theories. As a result, everything conceded about signs (in

8 N^ELSON G^OODMAN is usually conceived to be at present the most prominent “father representative” of the semiotic track; cf. [GOODMAN 1976].

9 ERNST GOMBRICH counts currently as the most influential and relevant “picture theorist” who follows mostly the perceptual track; cf. [G^OMBRICH 1960].

10 It is mainly KLAUS SACHS-HOMBACH who has worked out this synthesis; cf [SACHS-HOMBACH 2001], [SACHS-HOMBACH 2002]. He uses the German expression ‘wahrnehmungsnahes Zeichen’ – which is approximately ‘sign close to perception’.

Figure 14: Moja’s „Bird Picture”

general) must be applicable to pictures, as well. In particular, they are embedded in a specific context of action – the sign act – that involves two participants: one role may be called the “sender”, the other is the “receiver”.¹¹ In a sign act, the sender gives something to understand to the receiver by means of the sign, that is, she acts in this specific way in order to direct the attention of the receiver to something. This “something” must – by logical grounds (cf. [ROS 1990, Vol. III, 125ff] and [ROS 2005, 556f.]) – be primarily an attitude of the sender: an expression of his/her/its readiness to be involved in certain reactions (cf., e.g., a bodily expression equivalent to “I am angry”). This primary attitude may or may not allow us to differentiate the understanding into the sender’s intentionality toward a state of affair (potentially fictitious: a warning cry: “Tiger!!!”; or an assertion “In 1631 Magdeburg was completely destroyed”) or toward an object (possibly not present: “the author of the Philosophical Investigations”).¹² As a consequence, we never investigate single signs but always systems according to the multitude of “things” that the receiver is to be made aware of by means of the sign acts.

Pictures are conceived of as a separate sign system – it may be decomposable in distinct subsystems: naturalistic representations, technical graphs, icons, etc.

Second: since “being a sign” is determined by the use in a particular type of activity it is obviously not a combination of attributes of a physical object that allows us to catego-rize something as a sign. We indeed produce sometimes artifacts with the sole purpose to be used as signs (e.g., name tags for the participants of a conference, street signs).

But more or less any physical object may under certain circumstances be interpreted as a sign, as well. It is always the sign act and all its current (or potential, i.e., anticipated) participants that have to be considered if we understand something as a sign.

Third, if using physical objects as pictures is a communicative act, the various semi-otic aspects are applicable, like the distinction between what is referred to, what is rep-resented, and what is intended with the sign act (or, alternatively in BÜHLER’s terms:

‘representation’, ‘expression’, and ‘appeal’ of a sign; cf. Fig, 15, and [BÜHLER 1933, 28]).

In this context, SACHS-HOMBACH follows the tradition by suggesting to distinguish picture vehicle, picture content, and picture referent [SACHS-HOMBACH 2003]: with the expression ‘picture vehicle’ we restrict our attention to those aspects of a picture that it has as a mere physical object, like a cathode ray screen, with the usual properties of physical objects including the visual ones of shape and color. As such, it may be em-ployed in a sign act, but it also may be used in many other types of activities not related with communication. If we speak of “the referent of a picture” we mean the (factual or fictitious) scenes, events, objects, etc. that the picture is taken to represent. Finally, by considering the “picture content” we focus on those properties of the picture vehicle that are relevant for understanding its significance in the sign act.

11 Alternatively, one sign user may simultaneously take both roles; e.g., somebody wandering through an art gallery contemplating the pictures (“showing them to herself”), or somebody performing soliloquy (“speaking to himself”).

12 Note that the first form is quite common among animals; the second form is restricted to higher verte-brates, the third form is private to human beings (or language users in the close sense of language). cf.

[CLARKE FC, Sect. 5]. It is also important to note that, concerning the more complicated forms, it is still the sender’s attitude toward a state of affairs/object that is denoted or referred to in a complex sign act:

what is denoted or referred to and what is represented are usually not identical in a sign act (cf. [ROS

1989/90, Vol. III, 129ff]).

Fourth: denoting objects or events is not the only communicative purpose pictures are used for. By showing a picture to someone we might want to express our feelings or ask that person to do something (cf. ‘expression’ and ‘appeal’ in the organon model, Fig.

15). It is coherent with this understanding that the development of communications for infants is described as a sequence of repetitions and variations: based on innate emo-tional expressive acts, e.g., smiles, exchanged with (not just send to!) the reference per-son, variations are developed on both sides, and established or ignored by the mutual feedback reactions, so that more and more complicated patterns are formed that can be used to communicate more than just expressing attitudes [DORNES 1993, 152ff].

Communication, then, may be seen as a kind of dance. Each of the diverse types of moves in a language game determines a corresponding interpretation schema for the concrete utterances, which cannot be associated with the mere sign vehicle used but is determined by the position in the overall choreography. For verbal language the sequence of pragmatically possible and plausible communicative moves in various language games is described by the theory of speech acts [AUSTIN 1962, SEARLE 1969].

Some authors have extended this approach to “picture acts” (including mixed communication, as well; [ANDRÉ 2000, Sect. 2.3.1]).

Fifth: using pictures requires mastering a variety of semiotic rules: pragmatic rules that describe the typical functions and use conditions of pictures within the context of the other types of activities the communicating partners are (or may be) involved in;

semantic rules that express the relation between the picture vehicle and its broader meaning as far as this relation can be construed without explicitly dealing with pragmat-ics; and – as the most abstracted level of investigation – syntactic rules that try to iden-tify and formalize the range of attributes the picture vehicles must have in order to be usable as a particular sign of a system. The primary effect of the digitization of pictures necessary to deal with them in computers as mentioned above, is a syntactic effect, and we shall have to deal with that in more detail in the following chapter. But the two other levels have crucial influence on the way images are dealt with in computer science, too.

Therefore the next sections of this chapter summarize some general observations on the particular relation (resemblance) between pictures and their prime referents (spatial ob-jects), and on the relation between pictorial sign acts and the other acts of the picture

Figure 15: Standard Situation of Sign (S) Use (the Organon Model [BÜHLER 1933, 28])

users in that context, in particular propositional communicative acts. We shall see in Chapter 4 that primarily semantic and syntactic features have been investigated in com-putational visualistics for most of its (pre)history; taking explicitly a pragmatic perspec-tive is essentially a rather new development in the field.

Let us now consider the properties that distinguish pictorial signs from other subcategories of »sign«.

3.2.2 »Perceptoid« as Differentia Specifica for Pictorial Signs

What distinguishes pictures from signs like words and sentences is the special role perceptual competences play for constituting the picture’s content, i.e., for interpreting the sign.

Sixth: resemblance theory provides one means of construing the characteristic role of perceptual mechanisms [REHKÄMPER 2002]. According to that approach, a sign is a pic-ture if the perception of essential properties – that constitute the pictorial content – is identical to the perception of the corresponding properties of some other object under a certain perspective. Thus, we may use an object S as a pictorial sign for something else motivated by the observation that S looks similar to that other thing; however, similarity is a secondary condition for being a picture working only within the semiotic context of use. GOODMAN’s critique of resemblance as necessary condition of being a picture is indeed directed only against taking resemblance as a condition constituting the semiotic context of picture use instead of restricting the general scenario of sign acts in a particu-lar manner [FILES 1996].

Seventh: talking about likeness between perceptions instead of resemblance between objects brings the psychological characteristics of (visual) perception into account, and with them the principles of object constitution underlying them. For example, psycho-physical restrictions of receptivity, laws of Gestalt formation, conditions of color invari-ance, and factors of interpretative schemata for 3D-perception must be considered if we want to understand what objects appear as similar for somebody, and in what respect.

As object perception may be distinct for different social groups (different by culture or by age), so may be the motivation to use and understand an object as a pictorial sign.

There is no general restriction to the visual sense: perceptoid signs may be conceived accordingly in any modality. This accounts for the use of the expression ‘picture’ for non-visual perceptoid signs. Whether we can use such “pictures” to focus our attention to things in a way similar of using (visual) pictures depends in these cases on whether there are methods of object constitution associated with the sense modalities involved:

while it is relatively easy for us to employ “sound images”, we usually would need some training in order to use “odor images” for more than evoking very generally a situational context.

Eighth: the shift to psychology also opens an interpretation of pictures with fictitious objects or contradictory scenes by means of resemblance theory. The objects of percep-tion are “intenpercep-tional objects” [HUSSERL 1980], i.e., objects as something in the mind, something one’s attention is directed to, something “in one’s intention”; not something existing independently of any such intention and anybody having the intention. What we perceive in the case of an optical illusion, for example, cannot be – by definition of

‘optical illusion’ – an “objective object.” Correspondingly, there need not be a likeness to any real object or scene for a corresponding picture, only one to intentional objects or imagined scenes.

Ninth: resemblance as a criterion to characterize pictorial signs presupposes that cer-tain properties are excluded that do not contribute to the content of the sign and are relevant to its interpretation. Regarding something as a picture makes it obviously ir-relevant how heavy that object is or what its back looks like. In the case of linguistic expressions, we consider some respects as irrelevant, too, but different ones, e.g., the color of the font. Resemblance comes as a vague criterion; but the fact that it is deter-mined only in certain respects allows us also to accommodate it to quite different picto-rial phenomena. In some cases – mainly naturalistic pictures like photographs – it seems that we immediately and involuntarily regard most respects as relevant that are also relevant in perceiving objects visually. In others, like line drawings, we leave aside many of those respects: the picture is taken to resemble some object only relative to the remaining respects. In the extreme case, a diagram, for example, does not show us any-thing about how any physical object looks like.

Tenth: it is possible to establish different respects of similarity as dominant because the picture vehicle does not in itself determine which properties are relevant for the de-piction. We may develop¹³ different pictorial schemata with respect to some particular communicative functions the pictures are supposed to perform. It is then true to say that all pictures resemble their objects in one way or the other, but this relies completely on the pictorial schema determining in each case the relevant respects. Even more: actually perceiving resemblance depends then completely on the use of the sign vehicle as a sign ruled by a particular pictorial schema. Resemblance can only be established as embed-ded in the sign act (cf. Fig. 16).

Eleventh: in consequence, the distinction between pictures and their content is less clearly marked than is the case for language. The visual impression of a trompe l’œil is – as a crucial feature of this type of image – more or less identical to the impression of the real object depicted in that picture. This closeness to the content gives us the impres-sion of an access that is intuitive: we have to learn with an effort to master words, but to understand pictures seems to be a congenital facility for humans. Compared to verbal language perceptoid signs are less conventional, though the range of conventionality covered may still be rather broad.

13 – within one set of principles of object constitution, see sixth item.

Figure 16: Perceptoid – the Special Connection to Perception for Pictures

Twelfth: This also helps to understand the strange double fact: that we can on the one hand communicate with pictures in a way much more precise and more immediate than we are able to do with verbal language; but that we on the other hand usually need more contextual information to disambiguate what is actually meant by a picture from a lot of possible interpretations. In the words of SACHS-HOMBACH [SACHS-HOMBACH &

SCHIRRA 1999, 35], their high degree of semantic abundance comes along with a sig-nificant lack in semantic precision. Therefore, pictures need to be used in a context of action that determines in the concrete event their meaning, e.g., most explicitly by em-ploying a caption.

3.2.3 A Note on “Natural Images”, “Indices”, and “Icons”

We have mentioned in the beginning of this chapter the images in mirrors. The phenomenon as such is obviously independent from any context of sign use. So the question arises whether – or in what respect – we can classify mirror images as perceptoid signs. Or put inversely: is the definition of pictures as “perceptoid signs” too narrow to include mirror images – and hence probably too narrow in general? How do we have to interprete “natural signs”, as mirror images are sometimes called together with object shadows (cf. Fig. 17), the red spots of measles, and the foot prints on a sandy beach? In fact, in a society of blind nobody would have the idea of associating the discourse about, e.g., the surface of a quite lake with the discussion on perceptoid signs:

considered as a mere object without anybody (even potentially) perceiving it in the right modality of sense (i.e., visually), there is no reason to link a mirror with pictures, at all.

C. S. PEIRCE has introduced the semiotic distinction between “index”, “icon”, and

“symbol” that comes in handy for this discussion [PEIRCE 1931ff, 2.274]. An index is an entity that may be used as a sign for something – the referent – due to its direct physical relation to that referent.Thus, we may use smoke as an indexical sign for fire – we keep

Figure 17: A Shadow in Hiroshima — August 6, 1945, 8:15 a.m.

Burn pattern on the steps of the Sumitomo bank building – the only indication a human life has left in the moment the human race demonstrated its power of ultimate self destruction

our focus of attention on that (assumed) fire by means of “showing to ourselves” the smoke we perceive alone. Analogously, a photograph may be conceived of as an index.

Due to the causal relations – mediated by the light energy and the chemical reactions of the photosensitive emulsion – between a spatial scene and a photo thereof, a person can use the photo in a sign act to move the attention of somebody else to that spatial scene (or rather his/her attitude toward that scene).

Indexical signs cannot be used to refer to fictitious scenes – taking something as an indexical sign implies the reality of the referent. Nevertheless it is possible to lie with a photo [HGBRD 2000]: if the sender is aware of the photo not to be an index (e.g., being a photomontage) but leaves the recipient in believing it to be used as an indexical sign this sign act fulfills all criteria of a lie. That a photo can be used as an indexical sign and as a non-indexical sign for referring to the very same scene leads us to PEIRCE’s second class: icons are objects that may be used as a sign for something motivated by the fact that they bear resemblance with the intended referent. In the case of a photo, such a vis-ual resemblance is usvis-ually assumed even in the case of massive alterations of the origi-nal index: then, a fictitious scene is assumed to look like that, and the photo may be used in a sign act to denote that fictitious scene. Films with naturalistically rendered computer graphics that place believably behaving dinosaurs together with real actors in the background of an exotic forest, which may or may not be a real landscape, give a perfect example of such an iconic sign of a visual fiction.

PEIRCE’s third class, symbols, are characterized by no such immediate relation be-tween sign vehicle and sign object or sign content: it is the semiotic activity of the sign users in general that is responsible for that connection. Hence, symbols are called arbitrary. For indexical signs and iconic signs it is possible to understand them without learning – by spontaneously activating knowledge about causal relations or resemblance within the contextual semiotic activity: “this guy tries to tell me something (anything!) with that thing, which looks similar to / is causally linked to …”. No such spontaneous semiosis may take place for symbols without a prior introduction. The meaning of words, “human life” for example, must be taught; the significance of a date, e.g., “Au-gust 6, 1945”, must be explicitly communicated (as part of an already established com-plex cultural frame) before they can be used as symbols.

In the light of these distinctions, we can interpret mirror images as icons and as iconic indexical signs: the situation for the interpretation as an icon is given when we get a fright because taking erroneously our own mirror image – in the periphery of our field of sight, or when it’s a bit dark – for another person appearing there unexpectedly (PEIRCE’s “genuine icon” [PEIRCE 1931ff, 3.362]). If we realize a moment later what has happened, the mirror image changes its character immediately and becomes for us an iconic indexical sign: we focus our own attention to our own visual appearance by means of something looking similar and being causally linked directly to that appear-ance.

Quite obviously, iconic signs and perceptoid signs are closely related, to say the least.

Note, however, that the explanations of “perceptoid” make an explicit reference to the psychological background of resemblance as something derived in perception according to the principles of object constitution. Speaking of icons does not necessary imply such a complication: if it is possible or favorable to define similarity between objects per se, iconic signs become possible that need not “feel” (“look” etc.) similar to the referent scene, as long as the perception-independent relation of similarity holds as well (and is used as a motivation for preferring that particular vehicle in that sign act). It is however

quite dubious that such a concept of resemblance apart from psychology should not be considered as merely derived and usable only within very limited conditions.

In any case, the use of iconic and indexical signs depends on the sign users’ aware-ness of similarity or causal relation, which can be stabilized inter-individually only by means of symbolic communication. Without anchoring the language-mediated context of Figure 17, its iconic use remains ambiguous (“Is this meant as a human form or not?”), its indexical reference unclear (“where is this?” “who/what made that shadow-like spot?”). In order to provide a better understanding of how iconic (and indexical) sign uses depend on symbolically mediated frames of interpretations, the next section gives a coarse sketch of the complex inner structures of resemblance relations.