Object-teaching study 1

Coding

In order to develop the coding schemes, all data from the trials were annotated with the help of ELAN. Annotations were made of

• verbal utterances of the human

• verbal utterances of the robot

• gestures of the human

• objects shown to the robot

The annotations were then analyzed and behaviors (speech and gesture) of the human were identified and grouped. For speech this was done with the help of a linguistic analysis.

Accordingly, utterances with a similar grammatical structure and/or content were identified.

Analyzing the videos, only units of speech and gesture that convey meaning concerning the task were taken into consideration. Thus, utterances like “mhm” or the scratching of the chin were not interpreted as conscious behaviors that were conducted to teach an object to the robot. With the help of the video analysis, eight task-related verbal behaviors were identified:

1. naming object (whole sentence) (“This is a cup.”) 2. naming object (one word, very short utterance) (“Cup”) 3. describing the object (“The cup is blue and has a handle.”)

4. asking for feedback regarding the object (“BIRON, do you know what this is?”)

5. asking for BIRON’s general abilities and knowledge (“BIRON, what can you do at all?”) 6. asking for BIRON’s ability to listen/speak (“BIRON, can you hear me?”)

7. asking for BIRON’s ability to see (“Can you see the object?”) 8. demanding attention for the user/object/task (“BIRON, look at me.”)

While the first four behaviors describe object-related utterances, the last four include utterances about the abilities of the robot and its attentiveness. This shows that also in task-driven interaction, it seems to be important that the repertoire includes behaviors to find out what the interaction partner can do and whether it is attentive. In HHI these behaviors might be subtler than verbal utterances, because based on experience one can more easily estimate what abilities other humans might have and conclude from certain cues if they are attentive or not. The users often have less knowledge about the robot and the robot provides less feedback with this respect.

Next to typical verbal behaviors the data also implied some patterns concerning task-related gestures that the subjects used. Nine types of gestures were proposed:

1. Presenting the object

2. Moving the object once (up, down, to another position, rotate)

3. Moving the object continuously (back and forth, up and down, to different positions, rotate back and forth)

4. Moving the object closer to the robot

5. Manipulating the object (open the book/bottle) 6. Looking at the object

7. Pointing at the object

8. Imitating actions that can be performed with the object (drinking, eating, reading, etc.) 9. Holding the object

It could be questioned whether this categorization is exhaustive for all object-teaching tasks.

This will be tested in part with a second corpus of data that is presented in Section 4.2.4.

Moreover, when this coding scheme was developed, all behaviors of the users were included that seemed important at this point of time. That is why it contains holding the object (behavior 9). Technically, one could argue that this behavior is not a gesture in the sense it was defined in Section 3.2.2 because it has no sharp onset and offset and even more importantly it is not directed at the robot. This was taken into account in the analysis of the second object-teaching study as will be described below. There, the categorization presented here will be used as the basis for an adapted coding scheme.

Results

Even though the coding scheme shall here be presented as one main outcome of the first study, some results that were identified with the help of the coding schemes shall be briefly summarized. These results concern the change of the strategies by the users. Most changes were connected to situations when BIRON said that it had not understood or it could not do something. When this happened, the subjects tried to paraphrase, i.e., they switched between saying a whole sentence (behavior 1) and saying one word or a very short phrase (behavior 2).

Another important reason for changing behavior was the need to verify if BIRON had understood something. This happened when the robot signaled that it understood and the users wanted to be sure if this was true. Hence, the participants asked for feedback (behavior 4), knowledge, and abilities (behavior 5) of the system. Another situation that caused the users to switch between behaviors was a missing reaction by the robot. When BIRON had not done anything for some time, the subjects started naming the object in a detailed manner (behavior 1) or describing the object (behavior 3). Last but not least, the participants changed their behavior when they showed a new object to the robot. In this case, they usually asked BIRON for attention (behavior 8) and named the object in a whole sentence (behavior 1).

When the users started another behavior was also analyzed with respect to gestures. Five typical situations during which the users switched between different gestures were identified. Primarily, the participants applied another behavior when a new object was chosen. Usually the object was then presented to the robot (behavior 1), the object was moved in front of the robot (behavior 2, 3), or the subjects pointed at the object (behavior 7). All these behaviors seemed to be applied to attain the robot’s attention. Thus, in this situation the gestures seemed to have the same function as the speech, where asking BIRON for attention (behavior 8) was found to be most common.

Similar behaviors were evident when the users tried to present the same object one more time because BIRON had not recognized it or had not done anything for quite some time. As described above, when BIRON had not understood something the users paraphrased. While

doing this, they also tried two different types of gestures. They held the objects (behavior 9), which often seemed to be a sign of disappointment. Some chose the opposite behavior though and tried to regain BIRON’s attention by moving the object to another position (behavior 2).

This might be due to the fact that the users felt that BIRON might not have seen the object at the previous location. The same new behaviors were chosen when BIRON had not done anything for quite some time. The last situation that typically caused a change in behavior was the description of an action (for example, “This is a pencil. It is used for writing.”). In this case, a very close coherence of speech and gestures could be seen because the actions were described verbally and at the same time imitated in the gestures. The most common switches of gestures took place between presenting the object and moving it to another position. Thus, there was a constant change between holding the object still for the robot to recognize and trying to obtain the robot’s attention.

All these switches in behavior showed that the participants conducted them in reaction to the robot’s behavior. However, changes in user behavior seemed to be carried out consciously only when robot feedback for a certain channel was available. Thus, mainly changes in speech were reported by the participants when they were asked after the interaction how they adapted their behavior to the robot (see Table 4-1). All but two subjects only mentioned conscious linguistic adaptations. The participants did not consciously adapt their gestures to the robot, even though a notable number of gesture behaviors were found and changes between them occurred as frequently as linguistic changes. These changes of the users’ gesture behavior seemed to be an unconscious variation instead of an adaptation because there was no way for the users to find out which behavior was beneficial for the robot.

Table 4-1. Adaptation behaviors reported by the participants in the first object-teaching study (# = number of participants that mentioned the adaptation behavior)

Adaptation behavior #

Manner of speaking 8

• speak more clearly

• vary intonation

• vary loudness

• speak more slowly

2 1 2 3

Sentence structure 17

• verification questions

• switch between different sentence structures

• simple sentences

• imperative sentences

• one-word sentences

• special sentence structure (“This is a…”)

4 1 2 2 6 2

Content of utterances 7

• repetitions

• paraphrasing

• descriptions

3 1 3

Change between gestures 2

• hold object into the camera focus

• change between moving object and holding it still

1 1

# = number of participants that mentioned the adaptation behavior

The most common conscious adaptation was to use one-word sentences (sentences that only contain the name of the object taught to the robot). This finding implies that the users thought that BIRON only understood very simple sentences. Another common behavior was to ask verification questions. These questions show the users’ need for more feedback.

Moreover, it was found that feedback influenced the expectations of the users during the interaction insofar as their views of the speech in- and output of the robot were rather consistent after the interaction, whereas they were not sure about BIRON's abilities to recognize people, mimic, and gestures. As the feedback seems to be such an important factor, in the following it shall be evaluated whether the coding scheme holds for interaction with more sophisticated robot behaviors and if there are differences when the robot behavior is varied systematically.

Conclusion

The analysis of the first object-teaching study leads to some implications with respect to expectation theory and the model presented above. The consistent view on speech in- and output across the participants after the interaction points to the fact that they developed target-based expectations during the interaction with the robot. Since the judgment was similar across subjects, it can be assumed that it was mainly influenced by the situation, in particular by the skills of the robot, and not by the personality of the users. In contrast, the participants did not agree on BIRON’s abilities concerning recognizing people, mimic, and gestures. They perceived the situation differently with these respects because the robot did not provide them with explicit feedback. Hence, they did not form similar expectations. Also changes of user behaviors as a result of the robot’s behavior were only conscious for speech but not for gestures.

Speech behaviors changed when the robots’ behavior disconfirmed the users’ expectations.

Thus, the robots’ behavior directly influenced the users’ behavior. Since the robot did not gesture, this relation was not established for the users’ gesture behaviors which users changed unconsciously. Accordingly, the users’ expectations strongly depend on the behavior of the robot with respect to the modalities.