T Disadvantages solving with 90

90 Computer-based Testing and Training

with Scenarios from Complex Problem- solving Research: Advantages and

Disadvantages

Joachim Funke*

T h e u s e o f P C - b a s e d s i m u l a t i o n s f o r s e l e c t i o n a n d t r a i n i n g f o r j o b s w h i c h r e q u i r e c o m p l e x p r o b l e m s o l v i n g skills is e v e r - i n c r e a s i n g . T h i s p a p e r g i v e s a s h o r t r e v i e w o f s u c h s i m u l a t i o n s a l o n g w i t h a list o f a d v a n t a g e s a n d d i s a d v a n t a g e s o f t h e i r u s e . P o s s i b l e f u t u r e d e v e l o p m e n t s a r e s k e t c h e d .

K e y w o r d s : C o m p l e x p r o b l e m - s o l v i n g , c o m p u t e r - b a s e d t e s t i n g .

tically rich domains {e.g.. A n z a i and S i m o n 1979;

Bhaskar and Simon 1977), researchers b e g a n to investigate p r o b l e m s o l v i n g in specific

k n o w l e d g e domains (e.g., physics, writing, chess playing). Frequently, such research focused on trying t o understand the d e v e l o p m e n t of problem s o l v i n g within a certain domain, that is, o n the d e v e l o p m e n t of specialized expertise (e.g., A n d e r s o n et ai 1985; C h a s e and S i m o n 1973; C h i et ai 1981). D o m a i n s that h a v e attracted e x t e n s i v e attention in N o r t h A m e r i c a include such diverse fields as reading, writing, calculation, political decision making, managerial problem solving, l a w y e r s ' reasoning, mechanical problem solving, problem s o l v i n g in electronics, computer skills, and g a m e playing.

In Europe, t w o main approaches h a v e surfaced during the past t w o decades, o n e initiated b y D o n a l d Broadbent (1977; see Berry and Broadbent 1995) in Great Britain and the other b y Dietrich D o m e r (1975; see D o r n e r and W e a r i n g 1995) in G e r m a n y . T h e t w o approaches h a v e in c o m m o n an emphasis o n relatively c o m p l e x , semantically rich, computerized laboratory tasks that are constructed to b e similar t o real-life problems. P C - b a s e d

simulations h a v e been seen b o t h b y Broadbent and D o m e r as being central to our understand­ ing o f C P S .

Their t w o approaches, h o w e v e r , differ s o m e w h a t regarding theoretical goals and m e t h o d o l o g y . T h e tradition initiated b y Broadbent emphasizes the distinction b e t w e e n c o g n i t i v e problem s o l v i n g processes that operate under awareness versus outside of awareness, and typically e m p l o y s m a t h e -

T

the analysis of h o w people deal with c o m p l e x computer-simulated systems. This approach has spread quickly from the research lab t o applica­

tions in personnel selection and training. T h i s trend is discussed within four major perspectives.

First, a short historical sketch of different d e v e l o p m e n t s in N o r t h America and in Europe regarding research paradigms in this area will be g i v e n . Second, e x a m p l e s of computer

simulations used in selection as well as in training will be discussed. Third, a d v a n t a g e s and disadvantages o f using P C - b a s e d simulations for b o t h selection and training will b e

enumerated. Last, emerging d e v e l o p m e n t s in this R & D area will be presented.

1 Emergence of complex problem solving (CPS) research

'Address lot correspondence Jojchun Funke. Department of Psychology. Heidelberg Uni- venity. Hauptstr. 47-51. D- 09117 Heidelberg. Germany Email; loadiimFunketeununl- hcldelberg.de

T h e emergence o f c o m p l e x problem s o l v i n g (CPS) research has its roots in a d e e p disillusion with theories and results o n simple problem solving, as represented b y tasks like anagrams, concept identification, puzzles, etc. (for a r e v i e w see Bourne and D o m i n o w s k i 1972). T h e main reason for this can b e f o u n d in the assumed l o w validity of simple laboratory tasks with respect to the c o m p l e x i t y o f problems s o l v e d in e v e r y d a y life.

In N o r t h A m e r i c a , initiated b y the w o r k of Herbert S i m o n o n learning b y d o i n g in seman-

' &!*:lwd Publisher. Ud IWI 106 Cowtoj II I I llrfofd I «M iff, Ul I

JSO Mim Slteet Mdden. MA 011*6. USA. Volume 6 Number 2 April 1998

matically well-defined computerized systems.

The tradition initiated b y D o m e r . o n the other hand, is interested in the interplay of the

cognitive, motivational, and social c o m p o n e n t s of problem solving, and utilizes v e r y c o m p l e x computerized scenarios that contain up to 2,000 highly interconnected variables.

Buchner (1995) has classified the Dorner approach as a search for individual differences, which has needed semantically rich scenarios w i t h o u t g i v i n g m u c h notice to the mathematical properties of the systems. T h e tradition founded b y Broadbent w a s labeled b y him as o n e which is more interested in basic research on the

psychological processes o f learning to control c o m p l e x systems and which needed simple and mathematically well-defined systems. T h e detection o f a dissociation b e t w e e n verbalizable k n o w l e d g e about a s y s t e m and the ability to control that s y s t e m has led to an impressive b o d y o f research o n w h a t w e call n o w 'implicit learning and m e m o r y ' (for an o v e r v i e w , see Berry and Broadbent 1995).

N o r t h A m e r i c a n research o n C P S has typically concentrated o n e x a m i n i n g the d e v e l o p m e n t of expertise in separate, natural k n o w l e d g e domains. M o s t of the European research, in contrast, has focused o n novel, c o m p l e x problems, and has been performed with c o m - puterized and sometimes highly artificial tasks.

M u c h o f the N o r t h A m e r i c a n w o r k has been summarized in a v o l u m e edited b y Sternberg and Frensch (1991). T h e European research has been systematically and e x t e n s i v e l y summarized in a recent v o l u m e edited b y Frensch and Funke (1995).

W h e r e a s N o r t h - A m e r i c a n p s y c h o l o g i s t s seem to be reluctant in using c o m p l e x scenarios in assessment situations, European psychologists are not. In training situations, especially within the area o f flight training for pilots,

p s y c h o l o g i s t s from b o t h continents are equally interested in using these n e w instruments.

O f the t w o European approaches mentioned a b o v e the o n e initiated b y D o m e r has mainly influenced the European practitioner's w o r k in assessment and training. T h e reason for this m a y b e the surface or face validity of the scenarios w h i c h , naturally, is higher if not restricted for purely methodological reasons.

2 Prevalence of computer-based testing with CPS scenarios

Especially in the German-speaking countries, m a n y C P S scenarios are offered for use within the area of personnel selection and training (for a r e v i e w , see U. Funke 1995). T w o recently published G e r m a n editions (Geilhardt and M u h l b r a d t 1995; Straufi and Kleinmann 1995)

reveal this increased interest in the topic from the practitioner's v i e w .

T h e distinction of the f o l l o w i n g t w o sections b e t w e e n PC-based scenarios used for selection and those used for training is s o m e w h a t arbitrary because s o m e s y s t e m s can b e used for both purposes. T h e m a j o r reason for making such a distinction lies in respect t o the different goals of the systems.

2.1 Scenarios used for assessment purposes

W i t h i n the context o f the A s s e s s m e n t Center p h i l o s o p h y in G e r m a n y , a set o f quality stan- dards has been established (Arbeitskreis A s - sessment Center 1996). O n e o f the nine principles is called the "Simulation principle"

(see also G u i o n 1991). It requires the

psychologist ( w h o sets u p the instruments used within an assessment center) to simulate situations in as m a n y and as different w a y s as they will be found in the target position. It is explicitly not recommended t o o n l y think of these situations (instructions like "please imagine ..."); instead o n e should try to simulate a

situation as realistically as possible.

T h e r e is o n e simple reason w h y PC-based simulations are s o attractive t o use in the assessment context: t h e y allow for the design of highly c o m p l e x scenarios w h i c h b e h a v e dynamically o v e r time and respond directly t o subjects' decisions.

Space restrictions preclude an exhaustive listing of all simulations used for assessment purposes. This w o u l d be a v e r y difficult task because m a n y systems are applied for which n o documentation or references exist. Instead t w o examples which illustrate the approach will be offered.

T E X T I L F A B R I K ("textile factory";

Hasselmann 1993) is the name of a scenario d e v e l o p e d from basic research in the D o r n e r school. It has been subject to m a n y evaluation studies and represents the t y p e of scenario in which subjects h a v e to m a x i m i z e the profits of the small enterprise. T h e s y s t e m operates o n 3 0 variables and behaves dynamically d e p e n d i n g on subjects' interventions. O n half of these

variables, subjects can determine their values according to their decisions. A l s o , background information about the importance of the vari- ables and their role in the scenario can be accessed b y the subject. For a total period of 20 simulated months, subjects receive m o n t h l y

information about the actual state o f the factory.

D e p e n d e n t variables in T E X T I L F A B R I K are the total value of the c o m p a n y at the end of the simulation p e r i o d the relative amount of months

with an increase in raw capital, or an index s u m m i n g up the increase f r o m four selected

important areas o f the system. A n a l y s e s

91

:i.*kwell Publohcn Lid I W 8 Volume 6 Number 2 April 1998

» 2 INTERNATIONAL JOURNAL OF SELECTION ANU AX>txuvicni

concerning reliability and validity of these indices are presented in Hasselmann (1993).

Test-retest reliability as well as prediction of external criteria of job performance (salary; job position 0, I, or 5 years after testing, etc.) with the indices yield acceptable values in the range b e t w e e n 0 . 4 0 and 0.60.

A I R P O R T ( O b e r m a n n 1991, 1995) requires management of an airport for eight periods distributed o v e r a restricted amount of total time (about 2 hours). A l l information a b o u t the four different departments in this simulated airport has t o be collected b y the subject. A l s o , subjects are free to decide o n m a n y of topics related to that business. B y this procedure, o n e can see according to the author h o w people gather information, set priorities, and make a decision.

In the current version, about 3 0 variables can be manipulated directly b y the subject, about

150 dependent variables play a role in this d y n a m i c scenario. A t the end of the game, an automatic evaluation occurs s h o w i n g about 20 measures which can be condensed t o three main result variables reflecting analytic behavior, decision making, and logical and deductive abilities. A c c o r d i n g t o O b e r m a n n (1995). m o r e than 1000 subjects h a v e w o r k e d with this scenario. Published data on reliability and validity ( O b e r m a n n 1991) refer o n l y t o 150 subjects. A dear indication of the reliability and

validity for each of the 20 measures is missing.

2.2 Scenarios used for training purposes

A slightly different perspective is i n v o l v e d if scenarios are not used for selection but for training purposes. In this case, the question of reliability and validity of the measurement instrument is not as important as in the selection case because the impact of a n y measurement error upon a selectee is normally greater than u p o n a trainee w h o a l w a y s gets a second chance'. Nevertheless, reliable and valid indicators are also needed in the training situation. M o s t important regarding the training perspective is the question of transfer and generalizability t o real-world situations: for g o o d scenarios, o n e should b e able to demonstrate the effectiveness of the training with respect to a list of training goals.

A s mentioned, there also exist m a n y P C - simulations for training purposes, most of w h i c h w e r e not subject to psychometric o r other scientific analyses. In an o v e r v i e w about P C simulations for practical applications published b y G r a f (1992), 139 scenarios are listed - most of them d e v e l o p e d for training purposes.

A s an e x a m p l e of PC-simulations used for training w e look at the "Strategic Flight M a n a g e m e n t Simulators" d e v e l o p e d b y the M I T g r o u p o f systems thinking. Morecroft

(1988), Bakken et al (1992) as well as G r a h a m el ai (1992) g i v e reports on this approach w h i c h requires participants to d e v e l o p their o w n models about certain domains of reality.

Training is d o n e in this approach not w i t h a r e a d y - m a d e simulation but with the construction of the simulation model itself. T h e rationale behind this strategy is the detection that learning effects w i t h simulation scenarios are maximal for the persons w h o construct these systems and not as high for the o n e s w h o play w i t h them.

Bakken et al (1992) report the results of a transfer evaluation s t u d y comparing 17 M B A students (participating individually) with 3 2 professionals f r o m m a j o r corporations (participating in teams of t w o ) . T h e players w e r e confronted w i t h o n e of t w o structurally identical decision making g a m e s (oil tanker transportation and office real estate), for t w o trials consisting of 4 0 periods each w i t h the task o f maximizing profits. A f t e r a break, t h e y

received information about the other g a m e and had t o play it for o n e trial. It turned o u t that during the first game, students w e n t bankrupt twice as often as professionals. But with respect t o transfer, students s h o w markedly better performance than professionals. Training success in professionals w a s l o w because t h e y felt much m o r e restricted than students w h o learned a lot from their experienced bankruptcies. T h e con- sequence for designing effective training settings is according to the authors t o produce non- threatening learning laboratories which foster experimentation a n d a g a m e p l a y i n g strategy w h i c h uses "the information available in the g a m e as a springboard for investigation into causal d y n a m i c s " (Bakken el ai 1992. p. 180).

3 Advantages and disadvantages of computer-based testing and training with CPS scenarios

In what follows, s o m e of the m a j o r advantages as well as disadvantages of using P C - b a s e d simulations in the context of assessment a n d / o r training are discussed. T h e summarized evalua- tions rely largely u p o n trends discerned from the relevant literature.

3.1 Advantages

In general, the a d v a n t a g e s of P C - b a s e d simula- tions can b e summarized as follows:

(1) P C - b a s e d simulations allow the construction o f highly c o m p l e x scenarios w h i c h b e h a v e dynamically o v e r time and, thus, pose n e w requirements t o subjects.

(2) P C - b a s e d simulations allow for an e c o n o m i c presentation of c o m p l e x scenarios as well as

Volume 6 Number 1 April 1998 O BlKkwril PuWi*h»fi Lid 1 W

for a process-oriented data registration.

(3) P C - b a s e d simulations a l l o w for a quick c o m p u t i n g o f results - and at least in part

— for an 'automatic' interpretation of these results.

(4) P C - b a s e d simulations make it possible t o h a v e c o m p l e x scenarios presented in a standardized manner,

(5) P C - b a s e d simulations h a v e high acceptance f r o m the testee's point of v i e w .

In a training context. P C - b a s e d m e t h o d s h a v e s o m e specific advantages. A c c o r d i n g to Funke (1995, p. 228), these advantages are:

(6) T h e o p p o r t u n i t y for practice: trainees can experiment w i t h a c o m p l e x s y s t e m and thus, leam b y d o i n g , without risk and cost. T h e y can actively discover and e x p l o r e an u n k n o w n scenario and acquire declarative as well as procedural k n o w l e d g e about a certain d o m a i n of reality.

(7) A u g m e n t e d feedback: due to the time compression w h i c h is inherent in most P C - based simulations, feedback about the long- term consequences and side-effects of decisions appears quickly and can be related t o o n e ' s o w n behavior.

(8) Increased m o t i v a t i o n : m o s t P C - b a s e d

simulations p r o v o k e challenge and curiosity.

This is related t o affective processes which enable n o t o n l y acquisition of k n o w l e d g e but also the c h a n g e of attitudes.

(9) A d a p t a b i l i t y t o training objectives: de- p e n d i n g on the actual training goals, the c o m p l e x i t y of m a n y simulations can be changed and adjusted to the trainee's needs.

T h e s e a d v a n t a g e s are also reported in the a b o v e - m e n t i o n e d A m e r i c a n literature {e.g., Bakken et al.

1992; G r a h a m et al. 1992).

T h e realization of certain training goals (e.g., increased understanding of interdependent systems, intensified reflections about o w n decision strategies) d e p e n d s not o n l y o n the learner but also m u c h o n the activities of the trainer: t o present an environment in which learning can occur is not e n o u g h - t o increase the learning potential o f such situations, the controlled "explication o f implicit information"

seems t o b e a v e r y important step (see Leutner, 1995). T h i s can be d o n e b y adaptive hints and requests w h i c h require the active discussion of a topic related t o the learner's decision o r t o the state of the system, presented either b y a h u m a n trainer o r b y a s y s t e m - i m p l e m e n t e d intelligent tutor.

3.2 Disadvantages

Besides their a d v a n t a g e s , c o m p u t e r - b a s e d scenarios h a v e serious d i s a d v a n t a g e s (see also

Funke a n d Geilhardt 1996). S o m e are presented here t o g e t h e r w i t h p r o p o s a l s for their solution.

(1) P C - b a s e d simulations are often s o c o m p l e x that e v e n the developer of the system d o e s not k n o w what the correct or best solution will be for a g i v e n problem constellation.

This situation poses a problem for the

comparison o f individual results. If the m a x i m u m of a scale cannot b e determined (as is often not

possible in scenarios w i t h non-linear equations), o n l y relative rankings b e t w e e n scores are

possible.

Possible solution: use of scenarios w i t h k n o w n features and w i t h the existence o f a best solution.

(2) P C - b a s e d simulations produce a lot of behavioral data for most of which the p s y - chological interpretation m a y b e unclear.

S o m e simulations produce log-files containing all of the subjects' interventions and decisions.

Even if this huge amount of data are compressed to s o m e measures and indices this reduction d o e s not guarantee useful and sensible indicators.

Possible solution: use o n l y those measures and indices in a selection situation for w h i c h validity data are available. This proposal d o e s not prohibit the inclusion of measures and indices which h a v e not yet been validated t o evaluate their eventual validity. But t h e y should h a v e no impact o n the selection decision.

(3) PC-based simulations cannot easily be evaluated with respect t o their simulated domain validity.

Simulations often promise t o reflect the structure of a certain d o m a i n of reality w i t h a high degree o f fidelity. M o s t often, such promises cannot be verified because details about the internal structure of the system will not b e published. For systems with published equations (e.g.. T A I L O R S H O P ; see Funke 1983) it turns out that there exist relations which don't make much sense (e.g.. increase in number of vans d o e s increase requests and orders in an unlimited w a y ) as well as that reasonable and realistic factors (e.g.. market competitors) are not realized at all.

Possible solution: use o n l y those simulations for which domain validity has been p r o v e n . This proof can be d o n e best b y looking into details of the simulation equations. Because structural equation models for larger domains of reality are not k n o w n exactly, a plausibility check m a y be the o n l y possible evaluation.

(4) Results from P C - b a s e d simulations cannot easily be compared from o n e subject to another because the d y n a m i c situations differ b e t w e e n subjects due t o their different interventions.

T h i s argument reflects the f a d that shortly after start of a simulation, the further course of action in a simulated e n v i r o n m e n t for o n e person m a y differ drastically from that in the same

O Bl*rtw*P Publisher. Ltd 1998 Volume 6 Number 2 April 1998

9*

participated in a free simulation for s o m e amount of time can be difficult or impossible."

Possible solution: use o f systems which d o not differ w i t h respect to the simulated course of action (see Streufert et aL 1988, for this quasi- experimental simulation technique). T h e d i s a d v a n t a g e of this solution is the renounce- ment of the simulation principle; thus, this proposal w o u l d not be compatible with the scenario concept which is realized in nearly all of the systems.

(5) PC-based simulations are l o w on the social dimension.

This argument points t o the fact that most simulations require decisions m a d e b y a single subject faced with the system. In real life, c o m p l e x situations often i n v o l v e other people.

T h e social dimension o f an interaction b e t w e e n different people cannot be easily reconstructed in a simulation.

Possible solution: use of simulations which allow for g r o u p decision making. This could be realized, for example, b y use of interconnected computers. Especially in the area of distributed decision making, these interconnections b e t w e e n different problem solvers h a v e been analyzed successfully (see, e.g., Rasmussen, Brehmer and Leplat 1991; Rogalski and Samurcay 1993). Even formal models for communication in these situations already exist (Billard and Pasquale

1995). 1

(6) PC-based training applications need more data t o demonstrate their usefulness in the practice of management.

U p t o n o w , there is w e a k empirical e v i d e n c e o n the usefulness o f P C - b a s e d scenario training w i t h respect t o the practice o f m a n a g e m e n t . A s in the case o f scenarios for assessment w h e r e empirical data a b o u t reliability a n d v a l i d i t y o f the indices are i m p o r t a n t , it is also necessary in the case o f scenarios f o r training p u r p o s e s t o h a v e empirical indicators for the quality o f the training. S t r o n g e v a l u a t i o n o f such training u n d e r practical aspects is u r g e n t l y required.

4 Future developments

S o m e remarks about potential future d e v e l o p - ments in this area will be offered:

(1) P C - b a s e d simulations will n o t replace other diagnostic instruments. Rather, t h e y enrich the existing instruments with n e w aspects.

In the beginning of this research it looked as if

o n e had t o t h r o w a w a y conventional selectio and training instruments. N o w the picture look much m o r e realistic. Booth (1991) points out the the problem is not whether o r not o n e uses P(

t e c h n o l o g y but rather, w h e n , and t o w h a degree. This is also true for the use o f P C - b a s e , simulations in selection and training.

W h a t has b e c o m e clear in the meantime is th fact that n o t all simulation programs on thi market are useful for selection. S i m p l y U program a nice scenario which p o s e s a higl d e m a n d on the players' abilities for informatiot integration and decision making is not e n o u g h Psychometric data on reliability and validity oi derived quality measures must b e availabk before scenarios are presented t o subjects - and, most important, selection should not rel\

o n scenario data alone. For a full impression of z candidate's qualities, data from P C - b a s e c simulations will not be enough because such data touch o n l y s o m e aspects.

(2) Purely behavioral data which can be easily recorded f r o m the P C (e.g., number o f questions, frequency o f decision making) are, per se.

meaningless. If o n e neglects the c o n t e x t o f their occurence, validity o f such information decreases and an 'information loss' occurs.

Especially in P C - b a s e d simulations purely behavioral data seem t o s o l v e the problem of assessing the quality of g a m i n g performance because at the end of the assessment these data can be collected automatically. But think, for example, o f a variable such as number o f posed questions'. T h i s number should b e o p t i m a l l y high in the beginning of the simulation and l o w at the end - if y o u r program c o m p u t e s

something like total number o f questions p o s e d ' then y o u cannot evaluate that figure because y o u lost the important time context. But e v e n if y o u r program c o m p u t e s this figure differentially for the beginning and the end section, the quality o f questions is not reflected in these figures.

Similar arguments are g i v e n b y Strohschneider and Schaub (1995, p. 20lf.).

(3) P C - b a s e d simulations h a v e t o realize a high degree of 'fidelity' if generalization and transfer t o real life situations are expected.

In training c o n t e x t s , m o s t s u b j e c t s like P C - b a s e d s i m u l a t i o n s because t h e y are o f t e n fun.

But w h a t ensures that - besides the subject's e n j o y m e n t - a transfer f r o m the g a m e situa- tion t o real life occurs? It can b e stated as a general rule: the m o r e realistic the s i m u l a t i o n (i.e., the m o r e the s i m u l a t i o n b e c o m e s a simulator) the b e t t e r the e x p e c t e d transfer!

U s i n g a flight simulation p r o g r a m o n a P C leads t o different e x p e r i e n c e s t h a n b e i n g in o n e o f the b i g flight simulators u s e d f o r p r o f e s s i o n a l pilots.

Simulations in the N o r t h A m e r i c a n tradition s h o w high fidelity with respect t o the simulated

Volume 6 Number 2 April 1998

O BUckwftl Publisher! Lid 1*98

domain and, thus, allow for the training of context-specific skills. In contrast, simulations in ihe European tradition s h o w high fidelity with respect t o the required use of domain

independent rules and heuristics b y the manager to be trained.

(4) T h e use o f video-clips could enhance the fidelity of the presented scenario b y giving more context information.

Even more realistic, virtual reality ( V R ) could become an important tool for simulating do- mains of reality within assessment and training by means of interactive role play. V R is one o f the most powerful interfaces between computers and humans. It is an interactive, three-

dimensional, multisensory experience that immerses the individual in a computer-simulated world. V R promises to minimize the barriers currently inherent in prosocial skill training programs via its real life, immersion simulation.

Limitations in the technology currently exist and applications that attempt to generate accurate simulations that m o d e l human behavior may prove to b e the most difficult challenge o f all.

(5) The simulation approach could b e used to estimate training requirements.

Before one starts extensive training one needs to k n o w the training requirements of specific persons — otherwise training time will be wasted.

Simulations can present 'critical incidents' in order to assess h o w the trainee reacts to such situations. W i t h properly designed simulations and reliable indicators, one can provide the trainer with a list of weaknesses and strength of the person's decision making and problem solving qualities.

5 Concluding remarks

T h e influence of PC-based simulations o n selection, placement, and training will increase due to the ever-increasing and ubiquitous use of

PCs. In light of the fact that such simulations not o n l y bring advantages but disadvantages into

the R & D field a careful decision about when, w h y , and h o w to use them seems most

important. This is w h e r e basic research can help with carefully designed evaluation studies which s h o w the advantages as well as the

disadvantages of the n e w scenarios in more detail. It is the task and responsibility of the practitioners of PC-based simulation not to hastily rely o n such fascinating assessment instrumentation without first carefully analyzing their many caveats.

Author note

Thanks to John Booth and t w o a n o n y m o u s

reviewers w h o all g a v e valuable comments to earlier versions of this paper.

References

Anderson, J.H, Boyle. C.B. and Reiser. B.J. (1985) Intelligent tutoring systems. Science, 228. 456- 462.

Anzai, K and Simon. H.A. (1979) The theory of learning by doing. Psychological Review, 86, 124- 140.

Arbeitskreis Assessment Center (1996) Standards der Assessment Center-Technik [Standards of the Assessment Center technique). In Arbeitskreis Assessment Center, (ed.). Assessment Center als Instrument der Personalentwicklung 485-493.

Hamburg, Windmuhle.

Bakken, B., Gould. J. and Kim, D. (1992) Experimentation in learning organizations: A management flight simulator approach. European journal of Operational Research. 59, 167-182.

Berry, D . C and Broadbent, D.E. (1995) Implicit learning in the control of complex systems. In Frensch. P.A. and Funke. J. (eds.). Complex problem solving: The European Perspective. 131-150.

Hillsdale, NJ. Lawrence Erlbaum Associates.

Bhaskar. R. and Simon. H.A. (1977) Problem solving in semantically rich domains: A n example from engineering thermodynamics. Cognitive Science. 1, 193-215.

Billard, E.A. and Pasquale. J . C (1995) Adaptive coordination in distributed systems with delayed communication. IEEE Transactions on Systems.

Man. and Cybernetics. 25, 546-554.

Booth. ). (1991) The key to valid computer-based testing: The user interface. European Review of Applied Psychology. 41, 281-293.

Bourne, L.E. and Dominowski, R.L. (1972) Thinking.

Annual Review of Psychology, 23. 105-130.

Broadbent. D.E. (1977) Levels, hierarchies, and the locus of control. Quarterly Journal of Experimental Psychology, 29, 181-201.

Buchner, A . (1995) Basic topics and approaches to the study of complex problem solving. In Frensch, P.A. and Funke, J. (eds.), Complex problem solving:

The European Perspective. 27-63. Hillsdale, NJ, Lawrence Erlbaum Associates.

Chase. W . C . and Simon. H.A. (1973) Perception in chess. Cognitive Psychology. 4. 55-81.

Chi. M.T.H.. Feltovich, P.J. and Glaser, R. (1981) Categorization and representation of physics problems by experts and novices. Cognitive Science.

5, 121-152.

Dorner, D. (1975) Wie Mcnschen eine Welt venbessem wollten I H o w people wanted to improve a world). Bildder Wissenschafl, 12. 48-53.

Dorner, D. and Wearing A . (1995) Complex problem solving: Toward a (computersimulated) theory. In Frensch, P.A. and Funke, J. (eds.), Complex problem solving: The European Perspective. 65-99. Hillsdale, NJ, Lawrence Erlbaum Associates.

Frensch, P.A. and Funke, J. (eds.), (1995) Complex problem solving: The European perspective. Hillsdale.

NJ, Lawrence Erlbaum Associates.

Funke. J. (1983) Einige Bemerkungen zu Problcmen

35

CBI^Kk«rll Pddnhrr* Lid 19*8 Volume 6 Number 2 April 1998

INTERNATIONAL JOURNAL OF SELECTION AND ASSESSMENT

der Problemloseforschung oder: 1st Testinlelligenz d o c h ein Pradiktor? [Some remarks o n the

problems of p r o b l e m solving research or: D o e s test intelligence predict control performance?!

Diagnostica, 29. 2 8 3 - 3 0 2 .

Funke. J. and Ceilhardt, T . (1996) Diagnostik mit Hilfe v o n PC-Simulationen (Diagnosis based o n PC-simulations|. In Arbeitskreis Assessment Center (ed.), Assessment Center als Instrument Her Personalentwicklung. 2 0 1 - 2 0 9 . Hamburg, W i n d - muhle.

Funke, U. {1995) Using c o m p l e x problem solving tasks in personnel selection and training. In Frensch. P.A. and Funke. J. (eds.), Complex problem solving: The European Perspective, 2 1 9 - 2 4 0 . Hillsdale, NJ, Lawrence Erlbaum Associates.

Ceilhardt. T . and Muhlbradt. T. (cds.). (1995) Planspiele im Personal- una" Organisaiionsmanagement [Scenarios for personnel and organization management]. G o t l i n g e n , Verlag fur A n g e w a n d t e P s y c h o l o g i c

Graf. j . (ed.), (1992) Planspiele - simulierte Realilaten fur den Chef von morgen. Mit Planspiel-Marktubtrsicht (Scenarios - simulated realities for t o m o r r o w s ' manager]. Speyer, G A B A L - V e r l a g .

G r a h a m , A.K.. Morecroft. J . D . W . , Senge, P.M. and Sterman, J.D. (1992) Model-supported case studies (or management education. European Journal of Operational Research, 59. 1 5 1 - 1 6 6 .

C u i o n , R . M . (1991) Personnel assessment, selection, and placement. In Dunnctte, M . D . and Hough.

L.M. (eds.), Handbook of industrial and organizational psychology, 2 (2nd edition) 3 2 7 - 397. Palo A l t o , C A . Consulting Psychologists Press.

Hassclmann, D . (1993) Computersimulierte komplexe Problemstellungen in der Management-Diagnostik [Computcrsimulated c o m p l e x problems for diagnosis o f management skills). Hamburg, W i n d m u h l e .

Leulner. D . (1995) Computerunterstutzte Planspiele als Instruments der Personalentwicklung [Computer-based scenarios as instruments in personnel development). In Ceilhardt. T . and Muhlbradt, T . (eds.), Planspiele im Personal- und

Organisationsmanagement, 1 0 5 - 1 1 6 . G o t l i n g e n , Verlag fur A n g e w a n d t e P s y c h o l o g i c Morecroft, J. (1988) System d y n a m i c s and

microworlds for policymakers. European Journal of Operational Research, 3 5 , 3 0 1 - 3 2 0 .

O b e r m a n n , C (1991) AIRPORT. Problemldsesimulation 2.2 ( A I R P O R T problem solving simulation!.

G d t t i n g e n . H o g r e f c

O b e r m a n n . C (1995) C o m p u t e r g e s t u t z t e Planspiele in der Mitarbeiterauswahl - A n w e n d u n g s b c i s p i e l Airport [Computer-based scenarios in personnel selection — Airport scenario as an application example). In Ceilhardt, T . and Muhlbradt. T. (eds.), Planspiele im Personal- und Organisations-

management. 4 0 1 - 4 0 9 . G o t l i n g e n , V e r l a g fur A n g e w a n d t e Psychologie.

Rasmussen, | , Brehmer. B. and Leplat.). (eds.). (1991) Distributed decision making: Cognitive models for cooperative work. N e w Y o r k , W i l e y .

Rogalski, J. and Samurcay, R. (1993) A n a l y s i n g communication in c o m p l e x distributed decison- making. Special Issue: C o g n i t i v e processes in c o m p l e x tasks. Ergonomics, 36, 1329—1343.

Sternberg, R.J. and Frensch, P.A. (eds.). (1991) Complex problem soloing: Principles and mechanisms.

Hillsdale. NJ, Lawrence Erlbaum.

Straufi, B. a n d K l e i n m a n a M . (eds.), (1995) Computersimulierte Szenarien in der Personalarbeit [Computer-simulated scenarios in personnel selection and trainingl. Gfittingen, V e r l a g fur A n g e w a n d t e Psychologic.

Streufert. S., Pogash. R. and Piasecki, M . (1988) Simulation-based assessment of managerial competence: Reliability and validity. Personnel

Psychology, 4 1 . 5 3 7 - 5 5 7 .

Strohschneider, S. and Schaub, H . (1995) Erforschung komplexen Problemlosens durch computer- unterstutzte Planspiele: Kritische A n m e r k u n g e n zur Forschungsmethodologie [Research o n c o m p l e x problem solving b y means of simulation games: Critical c o m m e n t s o n the research m e t h o d o l o g y l . In Ceilhardt, T. and Muhlbradt.

T. (cds.), Planspiele im Personal- und Organisations- management, 1 8 7 - 2 0 3 . G d t t i n g e n , V e r l a g fur A n g e w a n d t e Psychologie.

Volume 6 Number 2 April 1998 O BUrk-fll PubtiUim Ud I * *