Mapping Expert Behavior onto Task-Level Frameworks:
Hypothesis 3: If appropriate psychological methods were available they would be used by knowledge engineers. What is needed are methods which are at the same
E: Well, it is difficult to say because the finds lateral resistance normal
forces are extremely small and it
does not really seem sticky finds that this manual test is if 1 move it back and forth sideways too insensitive
1 feel not real lateral resistance
Figure 4: Output of Phase I: Re-narration
Instructions Phase III: Problem solving analysis phase (mapping the "edited" text onto the framework)
Try to map the "edited" protocol onto the task-level framework. There will be no need to go back to the original text at any point (except to revise the whole analysis). The goal of Phase HI is a task-structure which describes the concrete task at hand.
0. It is a good strategy to do two tasks in parallel: (a) inserting the problem solving terminology into the protocol, and (b) developing the graphical task structure1. 1. Try to identify the top-level task and decompose it into subtask as indicated by the
"components" framework.
2. A d d the generic labels to the subtasks and to the problem solving methods.
Whenever possible, use labels from the taxonomy that is provided with the framework.
3. Whenever there is no appropriate label in the taxonomy, invent a new meaningful name.
4. Organize each problem solving method with a control regime.
5. For each problem solving method, identify the domain models, that is, the knowledge needed to perform the task. Again, in labeling use the predefined terms whenever possible.
6. For each task work out the epistemological problems and its associated pragmatic features and try to integrate them into the selection criteria of the identified problem solving methods.
7. Try to account for as many statements in the "edited" protocol as possible.
The output of phase III (see Figures 5 & 6) thus achieved is subject to less variation according to our preliminary experiments i f this method of protocol analysis is applied. But more systematic tests are needed. The next step will be to follow the second phase up with a second consensus session. It would be of interest to see i f consensus can also be achieved as easily as after the first phase. Moreover, the whole procedure would have to be done with knowledge engineers from different theoretical backgrounds ( K A D S , Componential Framework, Generic Tasks, Problem Solving Methods etc.). If it turns out that the inter-group reliability is high, this method would provide a good vehicle to be used in expert system construction.
Since methods of protocol analysis vary with the nature of the protocol (e.g., expertise-in-action, structured or a focussed interview, expert-guided problem solving, unstructured conversation), the protocol needs to be classified first. Our method w i l l only work for expertise-in-action or behavior observation protocols. The construction o f a causal story largely relies on this fact. It also restricts the applicability of the method. However, expertise-in-action is among the most valuable data for gaining more insights into the ways humans deal with the pragmatic features of a task.
1 Some of the steps and products mentioned in the instructions are o f course not required for the use o f task-level frameworks, but serve purposes of documentation and enable discussion with other knoweldge engineers and experts.
Output of Phase III: adding problem solving terminology (protocol) Domain: Fault diagnosis of a record player
Restates initial complaint: arm jumps get-initial-symptoms toward periphery
Has suspicion: sticky bearing cover-with-hypothesis:
causes arm to jump sticky bearing
moves arm back and forth test-hypothesis: sticky bearing
sideways
finds lateral resistance normal evaluate-test-result:
negative,
but test insufficient
finds that this manual test is too hypothesis not rejected but
insensitive given less weight
Figure 5: Output of Phase III: Problem solving analysis phase
diagnose-and-repair
diagnose
get-symptoms
ask-user generate- new-observation
find- covering-hypo theses
find-by-associadon {association}
repair
sequential- manipulation-and-observation {primitve actions}
test-hypotheses {functional
model}
find-by- consultadons-of-causal-model {causal-model}
find-explanations
Figure 6: Task structure (including the problem solving process) of the fault diagnosis task
Validation
It is clear that we have not said much about validity yet. In our case validity means to what extent the analysis reflects the actual thinking processes of the expert. Validity could be assessed by showing the final analysis back to the expert and getting his reactions. The expert could also contribute to the consensus sessions. However, it has to be kept in mind that it is not our goal to adequately describe the expert's reasoning, but that we aim at building expert systems, which is a very different story. For the present purposes we are therefore not worried too much about validity. What is more important is that the method is ecological (i.e.
applicable to real-life problem solving) and pragmatic (i.e. leads to systems).
Summary and conclusions
We first introduced a distinction between two extreme positions, the "engineering" and the
"expert" one. W e argued that both suffer from significant problems and suggested that the use of task-level frameworks, i f applied appropriately, provides a means for getting the best of both worlds. W e identified a largely neglected problem, namely that of mapping expertise onto task-level frameworks. The goal of our current efforts is to support precisely this process. A s a starting point we studied how expertise-in-action protocols can be mapped onto task-level frameworks. We developed a methodology to make protocol analysis more effective and more suitable for supporting this mapping activity. Given the high complexity of this activity, support w i l l consist of a set of guidelines, mainly in the form of verbal instructions, plus a set of computerized tools. There will be little in terms of automated systems.
It is concluded (a) that protocol analysis is a good starting point for developing tools to support the entire knowledge engineering process — //appropriate methods are available, and (b) that methods are only appropriate i f they are both ecological and pragmatic.
The methods developed on the basis of the approach outlined in this paper (using protocol analysis) should not be viewed as final or complete. Rather they serve as a starting point for further development: knowledge engineers must be studied carefully over extended periods of time in how they use the tools. What parts do they use most, what parts not at all, and—very importantly—how does the nature of their task change as they are using the tool. So, protocol analysis is only a starting point, but because of its ecological nature we expect that the tools developed on this basis can be naturally extended.
Acknowledgement
This work was partly supported by the University of Zurich, Tecan A G Hombrechtikon, and the "Swift A l Chair" of the Free University of Brussels.
References
Abelson, R . P . (1981). Whatever became of consistency theory. Proceedings of the 3rd International Conference of the Cognitive Science Society.
Bouchet, C , Brunet, E . , & Anjewierden, A . (1989). Shelley: A n integrated workbench for K B S development. Proceedings of the 9th International Workshop on Expert Systems and Their Applications, Avignon, 303-315.
Breuker, J . A . , & Wielinga, B . J . (1989). Models of expertise in knowledge acquisition. In G . Guida & C . Tasso (Eds.). Topics in expert system design, 265-295. Amsterdam:
Elsevier.
Chandrasekaran, B . (1986). Generic tasks in knowledge based reasoning: High level building blocks for expert system design. IEEE Expert, Fall, 1986, 23-30.
C h i , M . T . H . , Feltovich, P.J., & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5, 121-152.
Clancey, W . (1985). Heuristic classification. Artificial Intelligence, 27, 298-350.
Davis, R . (1984). Diagnostic reasoning based on structure and behavior. Artificial Intelligence, 24 347-411.
de Kleer, J. (1984). H o w circuits work. Artificial Intelligence, 24, 205-281.
Dennett, D . (1971). Intentional systems. The Journal of Philosophy, 68, 87-106. (reprinted in J. Haugeland (Ed.) (1981). Mind design (pp.220-242). Montgomery, V T : Bradford Books.)
Diaper, D . (Ed.) (1989). Knowledge elicitation. Chichester: Ellis Horwood.
Dorner, D . (1976). Problemlosen als Informationsverarbeitung (Problem solving as information processing). Stuttgart: Kohlhammer.
D y m , C . L . & Mittal, S. (1985). Knowledge acquisition from multiple experts. Al Magazine, 6 ( 2 ) .
Elstein, A . S . , Shulman, L . S . , & Sprafka, S . A . (1978). Medical problem solving: An analysis of clinical reasoning. Cambridge, M A : Harvard University Press.
Ericsson, A . , & Simon, H . A . (1984). Protocol analysis: Verbal reports as data. Cambridge, M A : M I T Press.
Holsti, O.R. (1969). Content analysis for the social sciences and humanities. Menlo Park, C A : Addison-Wesley.
Holyoak, K . (1990). Symbolic connectionism: toward third-generation theories of expertise. Techreport U C L A - C S R O - 9 0 - 1 4 . (to appear in I.A. Ericsson & J. Smith (Eds.).
Toward a general theory of expertise: prospects and limits. Cambridge, M A : Cambridge University Press.)
Mayring, P. (1990). Qualitative Inhaltsanalyse: Grundlagen und Techniken (2., durchges.
Auflage). (Qualitative content analysis: Foundations and techniques). W e i n h e i m : Deutscher Studien Verlag.
McDermott, J. (1988). Preliminary steps toward a taxonomy of problem-solving methods. In S. Marcus (Ed.). Automating knowledge acquisition for expert systems (pp. 225-256).
Boston, M A : Kluwer.
Neisser, U . (1982). Memory observed. San Francisco, C A : Freeman.
Newell, A . (1982). The knowledge level. Artificial Intelligence, 18, 87-127.
Newell, A . , & Simon, H . A . (1972). Human problem solving. Englewood Cliffs, N J : Prentice H a l l .
Nisbett, R . E . , & Wilson, T . D . (1977). Telling more than we can know: Verbal reports on mental data. Psychological Review, 84. 231-259.
Norman, D . A . (1988). The psychology of everyday things. New York: Basic Books.
Patel, V . F . , & Groen, G J . (1986). Knowledge-based solution strategies in medical reasoning.
Cognitive Science, 10, 91-116.
Shadbolt, N . , & Burton, A . M . (1989). The empirical study of knowledge elicitation techniques. SIG ART Newsletter, 108, 15-18.
Shaw, M . L . G . , & Gaines, B . (1989). Knowledge acquisition: Some foundations, manual methods, and future trends. Proceedings ofEKAW-89, Paris, France. 3-19.
Simon, H . A . (1969). The sciences of the artificial. Cambridge, M A : M I T Press.
Sol, H . G . , Takkenberg, C A . , & DeVries Robb€, P . F . (Eds.) (1987). Expert systems and artificial intelligence in decision support systems. Dordrecht: Reidel.
Steels, L . (1990). Components of expertise. Al Magazine, 11(2), 28-49.
Vanwelkenhuysen, J. (1989). TroTelc: An expert system troubleshooting printed ciruit boards. V U B A l Memo 89-17. Free University of Brussels, Belgium.
Vanwelkenhuysen, J., & Rademakers, P. (1990). Mapping a knowledge level analysis onto a computational framework. Proceedings ECAI (pp. 661-666). Stockholm, Sweden.
Vogel, C . (1990). K O D : A method for knowledge acquisition and modeling. Tutorial at the Tenth International Workshop on Expert Systems and Their Applications. A v i g n o n , France.
Waldmann, M . R . , & Weinert, F . E . (1990). Intelligenz und Denken (Intelligence and thinking). Gottingen: Hogrefe.