Evaluation

Professor: "Mh... I do not believe in that letter, that's certainly a bad joke".

Boy: "But it really looks quite old, it must be authentic!

Professor: "To whom do you think are you talking to? I am expert, I know what I say!"

Boy: "But that's arrogant! (To visitor) Don't you think that's the letter is au-thentic?"

(Here, the visitor is actively urged to contribute. Since the characters lead the conversation, a couple of response patterns is easily foreseen and can be integrated into the scene compound. If the visitor does not answer, the story will continue following a standard trajectory.)

VISITOR: "Yes, I think so"

The affirmative answer of the user was employed to increase emotional proximity parameters of visitor and boy. The recognized pattern is simply "yes", and the so-cial distance is stored as variable of the common data storage. This variable later on influences the choice of appropriate scenes, appearing in the condition of tran-sitions. More on social proximity variables will be explained in the section on “Ask

& Answer”.

volun-teers were divided into two groups, where one group was told beforehand that, after the interaction session with the installation, thematic questions would be posed, and the other group was asked the questions without prior preparation. All volunteers were then presented with a questionnaire on their impressions. The division in two groups was intended to research for the efficiency of the paratelic learning of the installation, i.e. of the playful learning that occurs without any expli-cit intention to learn.

Figure 56 – Quality perception – Diagram by Marc Hassenzahl (cf. [Iurgel05c]).

Not surprisingly, the paratelic learning results were notably inferior, though some learning effects were present. This indicates that visitors learn something through a playful, hedonic interaction with the applications, albeit not as much as they would if they know that they were expected to learn and that the learning results would be evaluated. These results were not surprising because the installation that was evaluated was not fine-tuned to maximize learning results, e.g. historic dates

huma n - technical simple

- com plicated practic

al - im practical straigh

tforwa rd - cum

bersome predictab

le - un predictable clearly

structu red - c

onfusing mana

geable - unruly connec

ting - isola ting profes

siona l - u

nprofe ssional stylish

- ta cky premiu

m - cheap integra

ting - a lienating brings m

e clos er to p

eople - sepa rates m

e from people presen

table - un presentable inventive - co

nventio nal creativ

e - unim aginative bold - ca

utious innova

tive - con servative captiva

ting - dull challen

ging - undem anding novel - o

rdinary 1

2 3 4 5 6 7

Mean item value

that were inquired afterwards were only presented once and shortly, during a ses-sion. But it remains an important research question whether paratelic learning through such applications can in principle achieve similar or even superior results than telic, less hedonic learning. To what extent will applications like Art-e-Fact be able to replace traditional teaching? Up to now, we only have data that indicate that it can complement e.g. school lessons, but not sensibly replace any learning unit.

Then, the other important question refers to whether the visitors are motivated by the installation – in a learning context, this would be an important result, since mo-tivation to learn is often difficult to achieve, and a motivating learning method will often be more appropriate than a method that is maximum efficient, on the short run, but tedious.

Figure 57 – The evaluation of satisfaction produced encouraging results, but also and astonishing tendency to “relaxation”, in contrast to “excitement”. Diagram by Marc Hassenzahl (cf. [Iurgel05c]).

The results in this respect were encouraging, since the overall emotional response to the installation was significantly positive. Interestingly, there is a certain tenden-cy of these positive emotions towards “calm”, “relaxation”, “satisfaction”, and not towards “excitement” and “joy” (Figure 57). This is surprising, if the very technolo-gical environment of the sessions is taken into account. There is no clear explana-tion for this finding, though the presence of rather non-aggressive virtual

charac-positive negative

calm excited

joy excitement

satisfaction relaxation

anger

frustration

boredom sadness

ters, their monotone voice, and the slow pacing of the concrete installation are ex-planatory candidates.

The only clearly negative aspect detected by the evaluation, criticized by more than 60% of all volunteers, was the speech synthesis (cf. Figure 56). This might be partly due to the fact that the installation language was English, and the volunteers were German native speakers. Though they were chosen partly for their good English skills, the synthetic voice might have posed additional difficulties to them.

In sum, the evaluation of Art-e-Fact demonstrates the need for a software frame-work that simplifies the task of building novel applications and of testing applica-tion ideas. The evaluaapplica-tion of this single applicaapplica-tion has pointed to strengths and weaknesses that must be addressed by future research and creative experimental work on interactive storytelling. The problems with speech synthesis were a sur-prise, since the version of the installation built for the evaluation did not contain dialogues that would require an emotional speech and prosody. The unemotional, but comprehendible voices of the TTS employed – ATT NaturalVoice – was ex-pected to be well suited for the task. It is an open question whether a different de-sign of the installation, e.g. with different dialogues and different, more “synthetic”

looking virtual characters, would reduce the dissonance cause by the TTS. Equally important were the findings on unsatisfactory paratelic learning results, which de-mand for a deeper understanding of story concepts that could better foster learn-ing. Further studies and experiments are also required to understand what exactly makes up an “exciting” installation, in contrast to a “calm” enjoyment. Since the concept of “story” is usually linked to “excitement”, the bias found towards “calm-ness” might point to further undetected problems, when non-continuous, “exciting”

narratives are intended.