Participants were given 45 minutes to create a small corpus of five rules covering different situations belonging to a scenario in which a robot is playing a quiz game with a famil-iar user. We placed no restrictions on the order in which rules had to be created and worked on. Rule names and descriptions were provided in the assignment instructions, but subjects had to create the rules themselves and fill both LHS and RHS60. We chose to have participants work with predefined rule names and descriptions for several reasons:
First of all, we did not want subjects to dedicate valuable time to coming up with rule names and descriptions themselves. Secondly, in order to obtain comparable results, giving participants a completely open-ended task61 was not an option. As a result, assignment instructions had to include descriptions of specific rules subjects were supposed to create.
With these descriptions in place, however, it did not seem sensible to ask participants to create original descriptions for the rules. Lastly, by providing rule names and descriptions we were honoring the assumption that in a real world setting, novice users would most likely continue to receive fairly concrete instructions about the kinds of rules they are expected to create after completing their very first training session. The decision to
never-60As mentioned above, each task participants had to complete during training included specific instructions about content to add, modify, or remove from LHS and RHS of rules. No such information was provided for any of the rules subjects had to create to complete the assignment.
61For instance, we could have specified a general scenario and left the task of coming up with appropriate rules completely up to participants.
16.2 Assignment
Figure 23: Text describing the concept of tasks. Displayed on the same HTML page as the text shown in Figure 22.
theless have subjectscreate the rules themselves instead of making them fill in predefined rule stubs was made for two reasons: Most importantly, we wanted to avoid any thematic overlap between training and assignment. Aside from not having participants recreate the same rule they built during training, we also had to ensure subjects did not have access to any information about the rules they were supposed to create during the assignment in order to achieve this goal. Secondly, providing a set of predefined rules to fill inand keep participants from seeing these rules during training would have required us to exchange the database between training and assignment. This was undesirable because it most likely would have further increased the overall run-time of individual experiments.
In contrast to the training phase, subjects were not allowed to ask questions while com-pleting the assignment. Instead, they were told to ask any questions they might have about the assignment before starting to work on it, and to only break the aforementioned rule if they encountered any problems that made it impossible for them to proceed without assistance. However, we did provide a printed reference manual describing relevant func-tionality that participants were allowed to consult in case they needed a reminder of how to perform a certain action. This manual listed and described system features in the same order in which they were introduced during training. We reused introductory texts from corresponding task pages to create individual entries of the manual. Task descriptions were omitted, and no additional information was provided about how to perform different types of actions. A copy of the reference manual can be found in Appendix E. We chose to provide a reference manual because as mentioned in the previous section, our goal was to create a scenario that was similar to how actual users might be introduced to the system.
While we do expect that users will stop receiving human assistance once they are suffi-ciently familiar with the system, we have no reason to assume that they will not have any kind of reference material at their disposal when working on a rule base by themselves.
Assignment instructions (shown in Figure 24) and rule descriptions stayed visible for the whole duration of the assignment. Participants were made aware of the time limit for completing the assignment because we wanted to give them the opportunity to plan their time, so as to avoid situations in which subjects failed to finish the assignment because they spent too much time focusing on a subset of the rules we asked them to create. In order to make it easy for subjects to check how much time they had left to work on rules, we added a countdown timer (in minutes) to the HTML page displaying the assignment instructions. The HTML page also included functionality for stopping the timer in case of serious problems requiring intervention of the experimenter. Participants were told to start the timer after they had finished reading assignment instructions and asking questions
16 Experimental Setup
Figure 24: Assignment instructions
about them. If subjects were in the middle of carrying out an operation (such as dragging content to a placeholder or typing in a part or an output string) when the time limit was reached, they were allowed to finish that operation, but they were not permitted to carry out another operation afterwards.
As mentioned above, one aspect we wanted to address with the evaluation experiments was how much variation subjects would be able to generate within a certain time limit.
To ensure that results produced by individual subjects would be comparable, we asked them to continue working on rule output even if they managed to create five rules with appropriate features and some output in less than 45 minutes.
Aside from a single example rule and the rule created during training, no other rules were present at the beginning of the assignment. The set of features and values from which LHS of individual rules had to be built was carried over from the training phase as well. Note that this set did not contain any features or values that were irrelevant with respect to completing the assignment. To provide further assistance for the task of choosing appropriate features, participants were told to check rule descriptions for information that would prohibit the use of a given feature, and to add the feature if no such information existed. Appendix D lists the rules subjects were asked to create.