W O R K I N G P A P E R
A PROTOTYPE SELECTIOZJ C O M M I m DEC1SIO:J 1 AIGLYSIS PJD SUPPORT SYSTEN, SC3lL?iS :
'IIIEORETICAL EACI<GROTJlJ;)
A!
COQUTER IMPIXFECJTATIOiJA. & w a n d o w s k i S
.
Johnson A. T d i e r z b i z l ~ iJune 1935 LIP-36-337
I n t e r n a t i o n a l I n s t i t u t e for Applied Systems Analysis
NOT FOR QUOTATION WITHOUT THE PERMISSION OF THE AUTHORS
A PROTOTYPE SEXZXXION COMkUlTEZ DECISION ANALZS'E AND SUPPORT SETEX. SCDAS:
THEORETICAL BACKGROUND AND COMPUTER -ATION
A n d r z e j L e w a d m s k i S a r a h Johnson A n d r z e j W i e n b t c k i
June 1986 WP-86-27
Working P a p e r s a r e interim r e p o r t s on work of the International Institute f o r Applied Systems Analysis and have received only limited review. Views o r opinions expressed herein do not necessarily r e p r e s e n t those of t h e Institute o r of its National Member Organizations.
INTERNATIONAL INSTITUTE FOR APPLID SYSEMS ANALYSIS 2361 Laxenburg. Austria
Foreword
One of t h e important problems in decision analysis r e l a t e d t o t h e situation, where t h e committee (group of decision makers) h a s t o s e l e c t t h e b e s t a l t e r n a t i v e from a given, finite set. In t h e most cases, t h e a l t e r n a t i v e s are evaluated on t h e basis of s e v e r a l quality factors. In t h e p a p e r , t h e a u t h o r s p r e s e n t t h e new ap- p r o a c h , based on t h e principle of s a t i s f a c t o r y decision making. This a p p r o a c h en- s u r e s p r o p e r structuralization of t h e decision p r o c e s s and allows p r o p e r balance of opinion of t h e group member. The experimental decision s u p p o r t system SCDAS w a s developed t o test t h i s a p p r o a c h .
The r e s e a r c h i s a r e s u l t of cooperative work between t h e System and Decision Sciences Program and t h e Institute of Automatic Control in Warsaw, done within t h e scientific agreement between IIASA and t h e Polish Academy of Sciences. S a r a h Johnson took p a r t in t h e p r o j e c t during h e r participation in t h e YSSP program at IIASA.
Alexander B. Kurzhanski Chairman
System and Decision Sciences Program
A PROTOTYPE SELECXION COMMI'ITEE DECISION ANALYSIS AND SUPPORT SICSTEM.
SCDAS:
THEORETICAL BACKGROUND
AND
COMPUTER JMPLEMENTATION* * *
A n d r z e j L e w a n d o w s k i , S a r a h ~ o h n s o n * a n d A n d r z e j W i e r z b i c k i
*
International Institute f o r Applied Systems Analysis, Laxenburg ,Austria**
Institute of Automatic Control, University of Warsaw, Warsaw, Poland1. MTRODUCTION
Many major decisions in public and p r i v a t e a r e n a s are delegated to commit- tees. The institution of a committee, though i t h a s many shortcomings, remains a n important a s p e c t of many decision p r o c e s s e s ; t h e p r o c e s s of committee decision- making must t h e r e f o r e b e improved. A s a r e s u l t of p e r s o n a l e x p e r i e n c e s with com- mittees, t h e a u t h o r s have developed a p r o c e d u r a l concept and a n automated aid f o r decision-making by committee, aimed in p a r t i c u l a r at a committee c h a r g e d with t h e t a s k of selecting from a finite set of alternatives.
The t h e o r e t i c a l framework f o r t h e automated system called "SCDAS" ( f o r Selection Committee Decision Analysis and S u p p o r t system) follows t h e concept developed by Johnson (1984). The multi-person decision s u p p o r t system i s based on t h e construction of a n order-consistent achievement function (Wierzbicki, 1985) which i s used as a multivariable c a r d i n a l utility function and depends explicitly on t h e contextual information supplied by t h e u s e r s . The system d e s c r i b e d c a n b e ap- plied to a wide spectrum of decision problems and s e r v e s as a p r o c e s s o r of infor- mation a b o u t p r e f e r e n c e s and a l t e r n a t i v e s t h a t guides t h e committee. The comput- er implementation i s non-procedural in t h a t a menu format allows e n t r y a n d re- e n t r y into many s t a g e s of t h e p r o c e s s , t h u s allowing a g r e a t deal of p r o c e d u r a l flexibility. Additionally, a r i c h g r a p h i c r e p r e s e n t a t i o n h a s proven quite u s e r - friendly on t h e basis of s e v e r a l empirical tests.
The organization of t h e p a p e r i s as follows. F i r s t , t h e t h e o r e t i c a l background and t e c h n i c a l a s p e c t s of t h e system are discussed. A section devoted to a discus- sion of t h e p r o c e d u r a l framework follows. A t u t o r i a l example of t h e selection of a candidate by a r e c r u i t i n g committee i s used throughout f o r i l l u s t r a t i v e purposes.
The final section p r e s e n t s in brief t h e computer implementation of SCDAS and t h e limitations and f u r t h e r extensions of t h e system, t h e primary o n e being t h e explicit inclusion of uncertainty in t h e evaluation of a l t e r n a t i v e s .
2. THEORETICAL BACKGROUND
The problem of selecting one a l t e r n a t i v e from a finite set of a l t e r n a t i v e s p r e s e n t e d t o a committee i s one of t h e most basic and classical decision problems and h a s received much attention in t h e decision-theoretical l i t e r a t u r e . T h e r e are many detailed v a r i a n t s of such a problem; h e r e , w e consider t h e following a b s t r a c t variant:
A committee consists of s e v e r a l members (denoted h e r e by k
=
1....,
K); e a c hmember c a n have e i t h e r equal o r different v o t i n g power (denoted h e r e by a voting power coefficient v ( k ) ) , specified a p r i o r i by t h e committee c h a r t e r . In addition t o t h e committee s t r u c t u r e , t h e committee c h a r t e r might specify t h e p u r p o s e of t h e committee's work, f u r t h e r p r o c e d u r a l details, e t c .
The problem faced by t h e committee i s t o jointly r a n k o r s e l e c t one o r a f e w from a set of available d e c i s i o n a l t e r n a t i v e s (these might b e candidates f o r a job, proposals f o r R&D p r o j e c t s , a l t e r n a t i v e t r a n s p o r t a t i o n r o u t e s , proposed s i t e s of a n industrial facility, a l t e r n a t i v e computer systems, etc.). The list of a l t e r n a t i v e s need not b e complete at t h e beginning of t h e committee's work; during t h e decision-making p r o c e s s , new a l t e r n a t i v e s may b e g e n e r a t e d and subsequently evaluated.
Evaluation of a l t e r n a t i v e s i s performed by t h e committee by f i r s t specifying d e c i s i o n a t t r i b u t e s (such as a candidate's age, e x p e r i e n c e , professional r e p u t a - tion, e t c . ) and t h e n assessing e a c h a l t e r n a t i v e with r e s p e c t t o e a c h of t h e s e a t t r i - butes. The l i s t of decision a t t r i b u t e s (denoted by j
=
I,.. ., J) might b e specified in t h e committee's c h a r t e r o r decided upon by t h e committee. In any c a s e , decision a t t r i b u t e s must b e specified b e f o r e a l t e r n a t i v e s c a n be evaluated and compared.Each a l t e r n a t i v e (denoted by i
=
1 , . . . , I ) must b e evaluated by t h e committee o r i t s individual members. The problem consists of proposing a d e c i s i o n process which t o g e t h e r with assessment of various a t t r i b u t e s of t h e a l t e r n a t i v e s and aggregation of evaluations a c r o s s both a t t r i b u t e s and committee members, leads t o a final ranking o r selection of a n alternative@) in a way t h a t i s rational, under- standable and a c c e p t a b l e t o t h e committee members.S e v e r a l a p p r o a c h e s t o t h i s problem have been developed; most of them a r e based on t h e classical multi-attribute utility t h e o r y ( s e e e.g. Keeney and Raiffa, 1976), but t h e r e are a l s o a l t e r n a t i v e a p p r o a c h e s , such as t h e analytical h i e r a r c h y of Saaty (1982) o r t h e o r d e r i n g s of Roy (1971). Some of t h e s e a p p r o a c h e s have been a l s o implemented as microcomputer-based decision s u p p o r t systems: a n in- t e r e s t i n g implementation i s t h a t of analytical h i e r a r c h y (EXPERT CHOICE, 1983) o r t h e non-procedural package DEMOS (1982) used f o r probabilistic evaluation of al- t e r n a t i v e s . Another commercially available implementation (LIGHTYEAR. 1984), based on utility t h e o r y and weighting coefficients specified by t h e u s e r , employs a r a t h e r primitive decision p r o c e s s and i s r e s t r i c t e d t o only one u s e r , hence i t i s not applicable in committee decisions.
Most of t h e s e a p p r o a c h e s r e l y on e i t h e r user-supplied rankings of a t t r i b u t e s and a l t e r n a t i v e s f o r e a c h a t t r i b u t e , pairwise comparisons of alternatives, o r some uncertainty equivalence principle (e.g. comparisons t o a lottery). The available assembly of a l t e r n a t i v e s plays a n important r o l e when establishing t h e principles of t h e decision. Such decision p r o c e s s e s will b e called a l t e r n a t i v e - l e d . An attempt t o establish decision principles independently of available a l t e r n a t i v e s is possible when specifying weighting coefficients by t h e u s e r ; but in addition t o t h e problem of having t o specify utility functions o r explicit weighting functions f o r t h e multi- ple a t t r i b u t e s , weighting coefficients c a n b e reasonably i n t e r p r e t e d only locally, when t h e available a l t e r n a t i v e s do not d i f f e r much in all of t h e a t t r i b u t e s . When t h e available a l t e r n a t i v e s d i f f e r significantly in some a t t r i b u t e s . t h e approximate
linearity of t h e u s e r ' s utility function i s a questionable assumption.
An easily i n t e r p r e t a b l e outline of decision principles t h a t are independent of available a l t e r n a t i v e s i s possible when requiring e a c h member t o specify a s p i r a - tion and ( o r ) r e s e r v a t i o n levels f o r t h e evaluation of e a c h a t t r i b u t e . Such a pro- c e s s will b e called aspiration-Led. The concept of a n aspiration level i s essential f o r t h e s a t i s f i c i n g f r a m e w o r k of decision-making (Simon, 1958). where i t i s as- sumed t h a t as soon as a n a l t e r n a t i v e i s discovered t h a t meets aspiration levels f o r all a t t r i b u t e s , t h e s e a r c h f o r a l t e r n a t i v e s is terminated and t h e choice i s made.
However, w e do not a d h e r e h e r e t o t h e s t r i c t l y satisficing framework: aspiration levels are used r a t h e r in t h e construction of a n approximate multivariable cardi- nal utility function t h a t i s f u r t h e r a v e r a g e d and maximized in t h e system. This ap- p r o a c h i s called q u a s i s a t i s f i c i n g ( s e e Wierzbicki, 1985).
The r e s e r v a t i o n level r e p r e s e n t s a minimum a c c e p t a b l e level f o r e a c h a t t r i - bute (e.g. minimum 5 y e a r s ' e x p e r i e n c e f o r t h e position), whereas a n a s p i r a t i o n level r e f l e c t s a h i g h e r d e s i r e d level of e x p e r t i s e . If a n a l t e r n a t i v e i s evaluated below t h e r e s e r v a t i o n level on even one a t t r i b u t e , i t i s considered unacceptable, and if i t i s evaluated at l e a s t equal t o aspiration levels f o r a l l a t t r i b u t e s , i t i s con- sidered highly desirable. Nonlinear approximations of utility functions based on aspiration (reservation) levels supplied by t h e u s e r a r e called (order-consistent, o r o r d e r - p r e s e r v i n g and r e p r e s e n t i n g ) achievement f u n c t i o n s and have been stu- died in detail by Wierzbicki (1982, 1985). Johnson (1984) h a s worked out a concept f o r a selection committee decision analysis and s u p p o r t system based on committee-supplied aspiration levels and t h e use of achievement functions f o r both alternative-led and aspiration-led v a r i a n t s of t h e decision p r o c e s s ; however, only t h e latter i s chosen h e r e f o r implementation.
2-1. Setting and discussing aspirations
An aspiration-led decision p r o c e s s h a s s e v e r a l advantages. Most judgmental decision p r o c e s s e s r e q u i r e a choice of (and, in a committee, agreement upon) s c a l e s of evaluation f o r e a c h decision a t t r i b u t e
.
The s c a l e s are often qualitative, such as unacceptable, bad, a c c e p t a b l e , good. v e r y good, excellent, though they can b e transformed into quantitative s c a l e s f o r computational purposes. When asked t o specify a n c h o r p o i n t s (aspiration and r e s e r v a t i o n levels) on t h e s e s c a l e s at a n e a r l y s t a g e of t h e decision p r o c e s s , t h e decision-maker i s b e t t e r p r e p a r e d t o make consistent evaluations a c r o s s alternatives. However, w e cannot e x p e c t and should not r e q u i r e full consistency in any judgmental decision p r o c e s s , s i n c e not all r e l e v a n t a t t r i b u t e s might b e evaluated and t h e r e l e v a n t information on a l t e r n a - tives i s n e v e r completely s h a r e d by all committee members. If e a c h committee member i s asked independently t o specify his o r h e r a s p i r a t i o n and ( o r ) r e s e r v a - tion levels f o r e a c h a t t r i b u t e , a comparison of such r e s u l t s a c r o s s t h e committee and a c r o s s a t t r i b u t e s s e r v e s s e v e r a l purposes:(a) t h e r e l a t i v e importance of e a c h a t t r i b u t e f o r e a c h committee member and a c r o s s t h e committee, as implied by t h e more o r less attainable levels, becomes ap- p a r e n t , as discussed below.
(b) t h e division of opinions among t h e committee members c a n b e discussed: if a significant s u b s e t of t h e committee h a s high aspirations (reservations) f o r a n at- t r i b u t e and a n o t h e r subset h a s low aspirations (reservations), i t i s a c a s e of a c l e a r disagreement on decision principles. The committee might then discuss t h i s disagreement and come t o a consensus; o r a g r e e t o d i s a g r e e by allowing t h e forma- tion of coalitions t h a t rally f o r t h e importance of various a t t r i b u t e s ( f o r example, when deciding on siting a n industrial facility, a p a r t of t h e t h e committee might b e more concerned with environmental impacts, a n o t h e r more concerned with econom-
ic impacts).
(c) if t h e discussion shows t h a t t h e r e a s o n f o r disagreement s t e m s from dif- f e r e n t perceptions by various committee members about t h e e x a c t meaning of a p a r t i c u l a r a t t r i b u t e and i t s s c a l e of evaluation, t h e r e s u l t might b e a b e t t e r specification of, o r at l e a s t c o r r e c t i o n s in, t h e list of a t t r i b u t e s .
(d) if t h e committee ( o r a coalition inside t h e committee) a g r e e s t o use a v e r - aged aspiration and ( o r ) r e s e r v a t i o n levels, e a c h committee member has a b e t t e r perception of t h e a n c h o r points t o b e used when evaluating alternatives.
In o r d e r t o s u p p o r t t h e s e discussions, a number of indicators c a n b e comput- ed. Denote t h e individually specified aspiration levels f o r a t t r i b u t e j by t h e com- mittee member k by p ( j , k ) and t h e corresponding r e s e r v a t i o n levels by r ( j , k ).
Then t h e committee "voting" p r o c e d u r e might specify a n averaging of individual in- puts, weighted by t h e voting power coefficients as follows:
Such a n a v e r a g e i s s u b j e c t t o manipulations by committee members who have a n incentive t o d i s t o r t t h e i r t r u e aspirations in o r d e r t o influence t h e e n t i r e com- mittee. A classical remedy, successfully used in subjective evaluations of c e r t a i n s p o r t performances (e.g. ice-skating o r ski-jumping) i s t o exclude outlying opin- ions, in t h i s case deleting t h e highest and t h e lowest p ( j , k ) o r r ( j , k ) a c r o s s a l l k b e f o r e aggregating. This p r o c e d u r a l option motivates committee members t o s t a t e t h e i r p r e f e r e n c e s carefully since they will have n o impact if t h e y voice t h e outly- ing opinions. If t h e committee adopts this option ( o r if i t i s imposed by t h e commit- t e e c h a r t e r ) , t h e n a n aggregation of opinions c a n b e c h a r a c t e r i z e d by:
where
denote t h e committee members with outlying aspiration levels who are t h e r e f o r e excluded from t h e averaging. The calculations a r e similar f o r aggregation of r e s e r v a t i o n levels r ( j ) and k ( r , j ).
2.2. Assessing disagreement
The disagreement about aspiration (reservation) levels f o r a n a t t r i b u t e among t h e committee c a n be measured in various ways. Clustering algorithms c a n b e used in t h e case of v e r y l a r g e numbers of committee members t o identify t h e positional s t r u c t u r e of t h e committee. Or, one could evaluate various statistical moments of t h e distributions of p ( j ,k ) and r ( j ,k ) a c r o s s k , although moments of a distribu- tion do not typically indicate t h e configuration of dissent. A good indicator of disagreement should distinguish between t h e case when t h e r e a r e two o r more siz- a b l e dissenting g r o u p s of committee members, each r e p r e s e n t i n g a uniform opin- ion, and t h e case when t h e differences of opinion a r e distributed uniformly o r at- t r i b u t e d mainly t o outlying opinions. To identify t h e s e differences, a d i s a g r e e - ment i n d i c a t o r c a n b e defined in t h e following way.
F i r s t l e t us consider t h e absolute change of aspirations:
where committee members are renumbered such t h a t
N o w A P ( j , k ) c a n b e split into t h e distribution of individual changes of opinion:
In t h e s e equations, k c a n b e i n t e r p r e t e d as t h e index of t h e pairwise comparison between t w o r a n k e d committee members. If l a r g e d i f f e r e n c e s o c c u r only at t h e ends of t h e r a n g e of k , corresponding to outlying opinions or small minority groups, t h e y are not as significant as when t h e y o c c u r in t h e middle of t h e r a n g e . To c o r r e c t f o r this, w e introduce a coefficient c (k ):
Other formulae c a n also b e used for t h i s coefficient; t h e a b o v e h a s been selected a f t e r empirical tests. The maximum value of c ( k ) f o r any ( K , k ) i s one.
A l s o , for a l l K , c ( k )
=
0 for both k=
1 and k =K-1 s i n c e outlying opinions are not counted in t h e aggregation. I t i s useful to define t h e disagreement indicator a s :This disagreement indicator i s bounded by t h e absolute d i f f e r e n c e of aspirations, A P ( j , k ) ; but D l @ , j) = A P ( j , k ) only if t h e committee i s split into two equal f r a c - tions of equal a s p i r a t i o n s in e a c h fraction. Note t h a t t h e disagreement indicator (5) h a s a peculiar p r o p e r t y : i t i s always equal t o z e r o if K S 3. Clearly t h i s is be- c a u s e a committee of t h r e e always h a s two outlying opinions and only o n e will t h e r e f o r e b e counted in t h e aggregation.
Similarly, disagreement indicators D l ( r , j ) f o r t h e distribution of r e s e r v a t i o n levels A r ( j ,k ) c a n b e computed. If both a s p i r a t i o n and r e s e r v a t i o n levels are used, t h e committee might b e i n t e r e s t e d in disagreement indicators for a v e r a g e s , D I ( p r , j ) , computed for t h e distribution of pr ( j ,k ), defined as:
I t should b e s t r e s s e d t h a t t h e a b o v e indicators s e r v e only t o draw t h e a t t e n - tion of t h e committee to t h e a t t r i b u t e s and a s p i r a t i o n s t h a t c a u s e dissent, f o r which a discussion of d i f f e r e n c e s of opinion might b e useful. Similar disagreement indicators c a n b e used when comparing t h e d i f f e r e n c e s between individual assess- ments of specific a l t e r n a t i v e s .
Another t y p e of indicator relates to t h e r e l a t i v e importance of v a r i o u s a t t r i - b u t e s as implied by specified a s p i r a t i o n s (reservations). Various t y p e s of indica- t o r s can a l s o b e used h e r e . W e choose d o m i n a n t w e i g h t i n g f a c t o r s i m p l i e d by a s p i r a t i o n s as r e l e v a n t indicators because t h e y are consistent with t h e function used later for t h e evaluation of a l t e r n a t i v e s .
To b e consistent with o u r t h e o r e t i c a l decision model, t h e weighting f a c t o r s f o r a t t r i b u t e s are c o n s t r u c t e d as follows: If a committee member specifies a s p i r a t i o n s f o r one a t t r i b u t e t h a t are "closer" to t h e u p p e r end of i t s evaluation scale than a n o t h e r , t h e n t h i s implies t h a t t h i s a t t r i b u t e i s more important to him or h e r than t h e o t h e r . More specifically, a n indicator should b e inversely p r o p o r t i o n a l t o such a distance a n d , if t h e indicators are i n t e r p r e t e d as weighting coefficients, t h e y should b e normalized s o t h a t t h e y sum up t o one across all a t t r i b u t e s . To
avoid computational e r r o r s , t h e indicators should b e calculable even in such a n unreasonable c a s e t h a t a committee member specifies aspirations equal t o t h e u p p e r end of t h e scale. Hence, w e extend t h e u p p e r bound slightly, denoting i t by u b ( j ) , and f o r simplicity normalize all s c a l e s s o t h a t t h e lower bounds of t h e scales of a l l a t t r i b u t e s are zero. Then t h e dominant weighting f a c t o r s implied by aspiration levels p of a t t r i b u t e s j f o r committee member k are computed as fol- lows:
J
w (P , j
=
( u b ( j ) / ( u b ( j )-
P ( j .kI))/
( u b(?I/
( u b(7)
- P ( j . k ) ) ) ( 7 ) J = lWeighting f a c t o r s implied by s t a t e d r e s e r v a t i o n levels w ( r , j ,k ) a r e calculated similarly.
These weighting f a c t o r s can a l s o b e calculated f o r t h e committee's aggregated p r e f e r e n c e s . In a l l c a s e s , t h e indicators s e r v e only as feedback signals t o indivi- duals o r t o t h e committee t o check whether t h e i r a s p i r a t i o n s c o r r e c t l y r e f l e c t t h e i r p e r c e p t i o n of t h e r e l a t i v e importance of various a t t r i b u t e s . If t h e r e a r e in- consistencies, they c a n easily b e c o r r e c t e d .
2.3. E v a l u a t i n g alternatives by individual c o m m i t t e e members
An essential p a r t of t h e decision p r o c e s s i s a n individual assessment and analysis of a l l a l t e r n a t i v e s by e a c h committee member. In t h e a p p r o a c h followed in t h i s p a p e r , i t i s assumed t h a t t h e assessment i s performed not by rankings o r pair- wise comparisons but simply by assigning s c o r e s f o r e a c h a t t r i b u t e t o e a c h alter- native ( a s a t e a c h e r would assign g r a d e s f o r e a c h s u b j e c t of learning t o e a c h pu- pil). Uncertainty in each assessment could b e e x p r e s s e d by supplying a r a n g e of s c o r e s o r a probability distribution f o r t h e s c o r e s ; however, we consider only t h e simpler c a s e without individual assessment of uncertainty. The s c o r e s of t h e k-th committee member f o r t h e j-th a t t r i b u t e of t h e i-th a l t e r n a t i v e a r e denoted h e r e by q ( i . j , k ) .
In o r d e r f o r e a c h committee member t o s e e what t h e s c o r e s imply and check f o r any scoring e r r o r s , rankings of a l t e r n a t i v e s by various a t t r i b u t e s c a n b e pro- duced in t h e system by listing t h e alternatives. s t a r t i n g with t h e b e s t s c o r e on a given a t t r i b u t e and ending with t h e worst s c o r e . However, t h e committee member i s a l s o i n t e r e s t e d in a n a g g r e g a t e ranking which t a k e s into account s c o r e s on a l l at- t r i b u t e s t o test whether his o r h e r intuitive opinion about which a l t e r n a t i v e s are b e s t i s consistent with t h e r e s u l t s of t h e scoring p r o c e d u r e .
A special approximation of a utility function implied by aspiration levels i s ap- plied in o r d e r t o produce such a n a g g r e g a t e ranking; t h i s approximation i s called a n (order-consistent) achievement function.
Consider t h e following question (Wierzbicki, 1986). Suppose t h e u s e r knows t h e u p p e r and lower bounds of a n assessment s c a l e and h a s specified a r e s e r v a t i o n and a n aspiration level f o r e a c h decision a t t r i b u t e ; t h e s e f o u r points w e denote respectively by Lb(j), u b ( j ) , r ( j ) and p ( j ) , where Lb(j)
<
r ( j ) < p ( j )<
u b ( j ) . Suppose a satisfaction (utility) value of z e r o i s assigned t o a n a l t e r n a t i v e whose at- t r i b u t e assignments are a l l equal t o r e s e r v a t i o n levels, and a satisfaction (utility) value of one t o a n a l t e r n a t i v e whose a t t r i b u t e s a r e a l l equal t o a s p i r a t i o n levels.We assume f u r t h e r t h a t a l t e r n a t i v e s which have s c o r e s satisfying a l l t h e i r r e s e r - vation levels are p r e f e r r e d t o any a l t e r n a t i v e which h a s at l e a s t one s c o r e not satisfying t h e corresponding r e s e r v a t i o n level. And similarly, a l t e r n a t i v e s which have s c o r e s satisfying all t h e i r aspiration levels a r e p r e f e r r e d t o any a l t e r n a t i v e which h a s at l e a s t one s c o r e not satisfying t h e corresponding aspiration level. Fi- nally, l e t a n (unlikely) a l t e r n a t i v e with s c o r e s all equal t o t h e lower bounds of t h e
scales have t h e value of d (a negative number) and a n (unlikely) a l t e r n a t i v e with s c o r e s all equal t o t h e u p p e r bounds have t h e value of 1
+
a (a number g r e a t e r than one). What i s t h e simplest cardinal utility function (i.e. a function t h a t i s in- dependent of all l i n e a r transformations of t h e assessment scales) t h a t i s consistent with all of t h e s e assumptions?The simplest function t h a t meets t h e s e requirements can b e constructed by us- ing l i n e a r approximations between t h e points f o r which i t s values are known ( 4 , 0 , 1 and 1
+
a ) . Such a function, called a l s o a n o r d e r - r e p r e s e n t i n g achievement function, h a s t h e following form:where
u j ( q ( i , j , k ) , p ( j ) , f ( j ) )
=
and q ( i , k )
=
( q ( i , l , k ),...,
q ( i . j , k ),...,
q ( i , J , k ) ) i s t h e v e c t o r of s c o r e s given by t h e k-th committee member t o t h e i-th alternative. Thus t h e achievement function maps a v e c t o r of a t t r i b u t e s into a scalar value f o r e a c h alternative. Additionally, p=
(P ( I ) ,. . .
, p ( j ),. . .
, p ( J ) ) and r = ( r ( I ) ,. . .
, r ( j ),. . .
.r ( J ) ) a r e v e c t o r s of a s p i r a t i o n and r e s e r v a t i o n levels aggregated a c r o s s t h e committee in a way t h a t i s a c c e p t - a b l e t o all members. In i t s middle r a n g e , t h e function (8) c a n also b e i n t e r p r e t e d by t h e difference between a s p i r a t i o n and r e s e r v a t i o n levels f o r e a c h a t t r i b u t e .However, t h e above achievement function h a s some disadvantages. Suppose t h e s c a l e s of assessments f o r all a t t r i b u t e s a r e from 0 t o 1 0 , and t h e r e s e r v a t i o n levels are all 3 while t h e aspiration levels a r e all 7. Compare two alternatives:
one with all s c o r e s equal t o 5 s o t h a t t h e value of t h e achievement function (8) equals 0.5, while t h e second a l t e r n a t i v e h a s s c o r e s of 7 f o r all a t t r i b u t e s but one.
which h a s t h e s c o r e 4 s o t h a t s (q , p , r )
=
0.25. But t h e second a l t e r n a t i v e might b e considered b e t t e r : t h e b e t t e r achievements on many a t t r i b u t e s could compensate f o r a worse achievement on one a t t r i b u t e . In o r d e r t o c o r r e c t f o r t h i s considera- tion, we propose a modified form of t h e function (8), called a n order-approximating achievement function:J
min u j ( q ( i , j , k ) , p ( j ) , f ( j ) )
+
( E / J ) u j ( q ( i . j . k ) , p ( j ) , f ( j ) ) + E )=
I1*Y j =lI
where u j (q ( i , j , k ) , p ( j ) , r ( j ) ) are defined as in (9). The p a r a m e t e r E in t h i s func- tion r e p r e s e n t s t h e intensity of c o r r e c t i o n of t h e worst (under-)achievement by t h e a v e r a g e (over-)achievement. In t h e example considered above, if E
=
1 and t h e r e are 5 a t t r i b u t e s , t h e n t h e f i r s t a l t e r n a t i v e h a s a value of t h e achievement function (10) equal t o 0.5 (due t o t h e subdivision by 1+
E in ( l o ) , t h i s does not depend on E if all u j are equal) but t h e second a l t e r n a t i v e has t h e corresponding value of 0.55. S o t h e second a l t e r n a t i v e is p r e f e r r e d . If, however, E=
0.5, then t h e f i r s t a l t e r n a t i v e h a s a n achievement value equal t o 0.5 but t h e second a l t e r n a - tive has a n achievement value of 0.45, s o t h e f i r s t a l t e r n a t i v e i s now p r e f e r r e d .The choice of t h e p a r a m e t e r E i s l e f t t o t h e committee: if i t s members f e e l t h a t t h e worst achievement matters most, they should choose slight c o r r e c t i o n (say, E
=
0.1); if t h e y f e e l t h a t t h e a v e r a g e achievement m a t t e r s most, they shouldchoose v e r y s t r o n g c o r r e c t i o n (say, E
=
2), indicating t h a t a v e r a g e achievement i s twice as important as worst achievement. A good i n t e r p r e t a b i l i t y of t h e values of t h e achievement function (10) by t h e u s e r s is obtained if a =b =1 and t h e values of s ( q ( i , k ) , p , r ) are multiplied by 10. Then t h e achievement r a n g e i s from -10 (corresponding t o a l l s c o r e s equal t o 0) through 0 (all s c o r e s on r e s e r v a t i o n lev- els), through 1 0 (all s c o r e s on aspiration levels) t o 20 (all s c o r e s maximal, equal t o 10).W e should a l s o mention h e r e some mathematical i n t e r p r e t a t i o n s of t h e dom- inant weighting f a c t o r s implied by aspiration o r r e s e r v a t i o n levels in connection with achievement functions in t h e forms (8) and (10). These achievement functions are nonlinear, hence t h e i r derivatives (corresponding t o t h e classical concept of a weighting f a c t o r in a l i n e a r utility function) depend on q ( i , k ) . In f a c t , t h e s e achievement functions are nondifferentiable, hence they do not possess deriva- tives in t h e classical s e n s e at some points
-
and, in p a r t i c u l a r , at t h e a n c h o r points, t h a t is, if q ( i , k )=
r o r q ( i , k )=
p. The dominant weighting f a c t o r s indi- c a t e directions in t h e J-dimensional s p a c e of t h e assessment v e c t o r s q ( i .k ), on which t h e points of nondifferentiability are located. While t h e s e p r o p e r t i e s of t h e dominant weighting f a c t o r s are important mathematically, t h e r e a d e r should remember two points: t h e dominant weighting f a c t o r s are not specified a p r i o r i o r supplied explicitly, r a t h e r they are implied by t h e choice of a s p i r a t i o n and/or of various a t t r i b u t e s as implied by aspiration and/or r e s e r v a t i o n levels.The achievement function (10) is used t o a g g r e g a t e s c o r e s given by a commit- tee member t o various a t t r i b u t e s of a n a l t e r n a t i v e and then t o r a n k various a l t e r - natives according t o t h e i r achievement values. This c a n b e done when using e i t h e r individual a s p i r a t i o n s (reservations) of a committee member o r aggregated a s p i r a - tions (reservations). In t h e former c a s e , t h e ranking proposed by t h e system s e r v e s as a feedback t o t h e committee member: h e o r s h e should compare i t with his o r h e r intuitive perception of ranking of alternatives. If t h e ranking does not match his o r h e r intuitive perception, he o r s h e should check whether h e did not make any e r r o r s in scoring; a n o t h e r r e a s o n f o r such mismatch might b e his disagreement with t h e c o r r e c t i o n coefficient E adopted by t h e committee. If t h e ranking does match his o r h e r intuitive perception, h e o r s h e should b e p r e p a r e d t o a c c e p t t h e f a c t t h a t t h e ranking based on aggregated aspirations (reservations) might b e different; but t h e committee member cannot p r o t e s t if h e o r s h e a c c e p t s t h e r i g h t of t h e committee t o impose aggregated decision principles on t h e collec- tive group.
2.4. Aggregating individual assessments across the committee
There are various i n t e r p r e t a t i o n s of t h e p r o c e s s of aggregating p r e f e r e n c e s a c r o s s a group of decision-makers. Typically, t h e i n t e r p r e t a t i o n i s r e l a t e d t o t h e concept of fairness; however, various paradoxes in decision t h e o r y (Saari, 1982) show t h a t t h e r e is no absolute meaning in t h i s concept. In this p a p e r , w e simply re- q u i r e t h a t t h e committee specify a set of p r o c e d u r e s t h a t is a c c e p t e d as f a i r by t h e group. For example, if t h e c h a r t e r of t h e committee specifies t h e voting power of e a c h member, t h e procedurally "fair" aggregation is t o t a k e t h e weighted a v e r - a g e of evaluations. The members with g r e a t e r voting power are supposedly e i t h e r more responsible (consider, s a y , t h e r o l e of t h e chairman of t h e committee), more concerned with t h e outcome of t h e decision p r o c e s s , o r more knowledgeable in a c e r t a i n substantive area.
Hence, a final ranking of a l t e r n a t i v e s f o r t h e e n t i r e committee c a n b e pro- posed by t h e decision s u p p o r t system by computing t h e (weighted) a v e r a g e d achievement values f o r e a c h alternative:
with s ( q ( i , k ) , p , r ) defined a s in (8) or (10).
This aggregation p r o c e d u r e gives r e l i a b l e r e s u l t s under c e r t a i n assumptions, of which t w o are m o s t important. F i r s t , w e assume t h a t committee members d o not bias t h e i r opinions in o r d e r to manipulate t h e outcome of t h e decision p r o c e s s . In o r d e r to discourage such manipulations, it is advisable to exclude outlying opinions f r o m t h e averaging p r o c e s s , as w a s done in (2) f o r t h e aggregation of a s p i r a t i o n levels:
where
& ( i )
=
a r min s ( q ( i , k ) , p , r ) ; c ( i )=
argmax s ( q ( i , k ) , p , r )l3rK lee
Second, w e assume t h a t committee members possess t h e same information about a l t e r n a t i v e s . This v e r y demanding assumption is n e v e r fully satisfied in p r a c t i c e . The decision p r o c e s s e n c o u r a g e s discussion and exchange of information a b o u t a l t e r n a t i v e s between committee members in p a r t by including concise descriptions of a l t e r n a t i v e s and r e q u i r i n g agreement a t c e r t a i n s t a g e s . When disagreement is indicated by major d i f f e r e n c e s in individual rankings of a l t e r n a - tives or by l a r g e values of t h e disagreement indicators, t h i s should tell t h e com- mittee to s t o p and s e a r c h for s o u r c e s of disagreement. If t h e disagreement i s d u e to a d i f f e r e n c e in t h e information b a s e between individuals, t h e n t h e problem c a n b e resolved by s h a r i n g and exchanging information. A g r a p h i c r e p r e s e n t a t i o n of t h e diverging scores f o r a n a t t r i b u t e of a n a l t e r n a t i v e helps g r e a t l y in s u c h dis- cussions; a committee member with a dissenting opinion c a n e i t h e r convince t h e committee t h a t h e or s h e h a s specific valuable information to s h a r e , or b e con- vinced t h a t h i s or h e r opinion cannot b e substantiated. This s e r v e s as a n additional disincentive for attempting to manipulate t h e outcome of t h e decision p r o c e s s by biasing assessments. The i n t e r e s t e d r e a d e r should a l s o consult Tversky et a l . (1983) for discussions about b i a s e s in decision-making.
After such discussion, t h e committee c a n e i t h e r decide to r e t u r n t o some ear- l i e r s t a g e of t h e decision p r o c e s s ( f o r example, to correct t h e s c o r e s ) or conclude t h e p r o c e s s . When adopting t h e final decision (a ranking or a selection of a l t e r n a - tives) t h e committee i s by n o means constrained by t h e a g g r e g a t e ranking pro- posed by t h e decision s u p p o r t system, but merely guided by t h e r e s u l t s .
3. PROCEDURAL FRAldEWORK
The a b o v e t h e o r e t i c a l background of a n aspiration-led decision p r o c e s s sug- g e s t s a g e n e r a l p r o c e d u r a l framework for t h e committee; however, t h i s framework i s r a t h e r e l a s t i c a n d c a n b e modified variously for any specific application.
The f i r s t p o i n t on t h e agenda i s to define t h e p r o c e d u r e s by which t h e com- mittee will o p e r a t e . The questions a d d r e s s e d h e r e should include t h e following:
(a) What i s t h e e x p e c t e d p r o d u c t of t h e committee's work and how does i t in- fluence t h e a p p r o p r i a t e p r o c e d u r e ? The answer to t h i s question depends on t h e committee's c h a r t e r and i t s p e r c e i v e d r o l e . For example, if t h e e x p e c t e d p r o d u c t i s a s h o r t list of significantly d i f f e r e n t a l t e r n a t i v e s , p r o c e d u r a l r u l e s will b e dif- f e r e n t f r o m t h e case when t h e e x p e c t e d p r o d u c t i s a consensus opinion on t h e
"best" a l t e r n a t i v e .
(b) What aggregation r u l e s should b e adopted, and in p a r t i c u l a r , should outly- ing opinions be included in o r excluded from aggregation?
(c) Should t h e committee b e allowed t o divide and form coalitions t h a t might p r e s e n t s e p a r a t e assessments of aspirations, a t t r i b u t e s c o r e s and t h u s final rank- ings of a l t e r n a t i v e s ?
The committee should a l s o become familiar with basic concepts concerning t h e use of t h e decision s u p p o r t system. A s e c r e t a r y o r a designated committee member whose duties include working with t h e decision s u p p o r t system should study thoroughly t h e description of t h e system in t h e u s e r ' s manual (1985), and p r e s e n t t h e basic concepts t o t h e e n t i r e committee in t h e f i r s t meeting.
The s e c o n d p o i n t on t h e agenda i s problem specification. Neither t h e list of a l t e r n a t i v e s , n o r t h e i r descriptions need b e complete a t this stage. The most im- p o r t a n t p a r t of t h i s p r o c e s s t h a t r e q u i r e s discussion and specification by t h e en- t i r e committee is t h e definition of t h e a t t r i b u t e s of t h e decision and t h e i r s c a l e s of assessment.
Various studies in decision t h e o r y suggest t h a t a reasonable number of a t t r i - butes should not e x c e e d seven t o nine ( s e e e.g. Dinkelbach, 1982); if more a t t r i - butes are suggested, t h e y should b e aggregated. For example, t h e r e might b e a l a r g e number of qualitative indicators t h a t are a l l r e l a t e d t o professional r e p u t a - tion; instead of using all t h e s e indicators, i t is b e t t e r t o a s k committee members t o evaluate subjectively t h e a t t r i b u t e "reputation", t h a t i s , t o t r a n s l a t e t h e informa- tion about all t h e s e indicators into one assessment, given originally on a v e r b a l s c a l e from "unacceptable" t o "excellent", into a quantitative s c a l e , say from 0 t o 10.
A c l e a r definition of r e l e v a n t a t t r i b u t e s i s a v e r y important p a r t of t h e deci- sion process. One possible a p p r o a c h is t o f i r s t list a l a r g e number of a t t r i b u t e s , t h e n o r d e r them into groups in a h i e r a r c h i c a l s t r u c t u r e , and finally decide on a s h o r t list of aggregated a t t r i b u t e s satisfying two requirements:
(a) t h e y should h a v e t h e same h i e r a r c h i c a l importance
-
which does not mean t h a t they should b e equally important, b u t they should not obviously d i f f e r in im- p o r t a n c e n o r b e h i e r a r c h i c a l l y dependent;(b) t h e y should not b e highly c o r r e l a t e d
-
t h a t is, two d i f f e r e n t a t t r i b u t e s should not e x p r e s s , under different names, t h e same essential a s p e c t of t h e deci- sion.Aggregated a t t r i b u t e s t h a t satisfy t h e s e requirements often h a v e a qualitative c h a r a c t e r . A committee should avoid t h e t r a p of selecting some a t t r i b u t e s only be- cause t h e y might b e quantitatively measurable (such as t h e number of publications of candidates f o r a scientific position). Typically, such a t t r i b u t e s are inadequate and are more r e l e v a n t when e x p r e s s e d in a g g r e g a t e terms.
During t h e t h i r d p o i n t on t h e agenda, aspiration a n d / o r r e s e r v a t i o n levels f o r a l l a t t r i b u t e s a r e determined s e p a r a t e l y by each committee member. After t h e s e values are e n t e r e d into t h e system, all necessary a v e r a g e s and o t h e r indica- t o r s (disagreement indicators, dominant weighting f a c t o r s ) c a n b e computed.
The f o u r t h p o i n t i s t h e analysis and discussion of aspirations by t h e e n t i r e committee. These discussions a r e supported by t h e computed indicators and t h e i r g r a p h i c interpretations.
In t h e s e discussions, t h e committee might a d d r e s s t h e following questions:
(a) Do t h e dominant weighting f a c t o r s a c c u r a t e l y r e f l e c t t h e p e r c e p t i o n s of individual committee members about t h e r e l a t i v e importance of various a t t r i b u t e s (if not, should t h e a s p i r a t i o n s o r r e s e r v a t i o n s b e c o r r e c t e d ) ?
(b) What a r e t h e r e l e v a n t differences and do they r e p r e s e n t a n essential disagreement about decision principles?
(c) Does t h e e n t i r e committee a g r e e t o use joint, aggregated aspirations (reservations), o r will t h e r e b e s e v e r a l s e p a r a t e sub-group aggregations?
The fiJYh p o i n t on t h e agenda i s a survey of alternatives. Discussions might c e n t e r on t h e following:
(a) Are t h e available descriptions of a l t e r n a t i v e s adequate f o r judging them according t o t h e a c c e p t e d list of a t t r i b u t e s ? If t h e answer i s negative, additional information should b e g a t h e r e d by sending out questionnaires, consulting re- viewers, e t c .
(b) Which of t h e available a l t e r n a t i v e s are i r r e l e v a n t and should b e deleted from t h e list? This kind of c u r s o r y screening c a n b e done in various ways. The com- mittee might define some screening a t t r i b u t e s and r e s e r v a t i o n levels f o r them (of a quantitative o r simple logical s t r u c t u r e ) : f o r example, w e d o not a c c e p t candidates t h a t do not have at least f o u r y e a r s of teaching experience. The s e c r e t a r y c a n b e empowered t o p r e p a r e t h e l i s t of i r r e l e v a n t a l t e r n a t i v e s t o b e deleted; t h i s list should b e presented t o t h e e n t i r e committee f o r approval. I t i s easy t o overlook special opportunities r e l a t e d t o seemingly i r r e l e v a n t a l t e r n a t i v e s .
(c) I s t h e list of relevant a l t e r n a t i v e s promising f o r a reasonable choice? O r should t h e committee look f o r new a l t e r n a t i v e s ? What a r e t h e a t t r i b u t e s t h a t have not been sufficiently w e l l addressed by t h e existing set of a l t e r n a t i v e s ?
Most of t h e s e questions a r e analyzed subjectively without much s u p p o r t from t h e system. However, t h e list of r e l e v a n t a l t e r n a t i v e s must b e sufficiently s h o r t b e f o r e going t o t h e next point on t h e agenda.
Z'he sizth p o i n t on t h e agenda i s t h e individual assessment of alternatives.
The assignment of s c o r e s f o r e a c h a t t r i b u t e t o e a c h a l t e r n a t i v e i s t h e main input of committee members into t h e system. Each member specifies s c o r e s ; t h e system s u p p o r t s this by displaying those assignments a l r e a d y made and t h o s e still t o b e entered.
The s e v e n t h p o i n t on t h e agenda i s individual analysis of a l t e r n a t i v e s , based on calculations of t h e achievement function which lead t o a ranking of all alterna- tives f o r e a c h committee member. This ranking i s t h e main s o u r c e of learning about t h e distribution of a l t e r n a t i v e s r e l a t i v e t o aspirations.
The questions addressed by e a c h member at t h i s point might b e as follows:
(a) Do t h e rankings along e a c h a t t r i b u t e c o r r e c t l y r e p r e s e n t t h e individual's evaluations of a l t e r n a t i v e s ; does t h e achievement ranking, based on individual as- pirations, c o r r e c t l y r e p r e s e n t t h e a g g r e g a t e evaluation (if not, should t h e s c o r e s b e modified)?
(b) If t h e r e remains disagreement about t h e member's individual achievement ranking of a l t e r n a t i v e s suggested by t h e system, should h e o r s h e propose at t h e next committee meeting t o modify t h e p a r a m e t e r E t h a t e x p r e s s e s t h e importance of a v e r a g e achievements as compared t o t h e worst achievement?
(c) If h e o r s h e a g r e e s with t h e individual achievement ranking proposed by t h e system, what are t h e differences between t h i s ranking and t h a t based on indivi- dual s c o r e s but r e l a t e d t o committee-aggregated aspirations? A r e t h e s e differen- c i e s significant, o r c a n h e o r s h e a c c e p t them as t h e r e s u l t of agreement on joint decision principles?
Z'he e i g h t h p o i n t on t h e agenda is a committee discussion of t h e essential differences in scoring and disagreements about t h e preliminary ranking of a l t e r - natives aggregated a c r o s s t h e committee. These discussions a r e s u p p o r t e d by t h e system; t h e system computes indicators of differences of opinion and p r e p a r e s a
preliminary aggregated ranking.
The questions addressed by t h e committee at this point might b e t h e following:
(a) On which a t t r i b u t e s are t h e l a r g e s t differences in scoring between commit- tee members observed? Do t h e s e differences r e p r e s e n t essential differences in information about t h e same a l t e r n a t i v e ?
(b) What i s t h e essential information ( o r uncertainty about such information) t h a t c a u s e s t h e s e differences? Should additional information b e g a t h e r e d , o r can c e r t a i n committee members supply t h i s information?
(c) Would t h e r e s u l t s of t h e s e discussions and possible changes of scoring in- fluence t h e preliminary aggregated ranking list proposed by t h e system? This c a n b e tested by applying simple sensitivity analyses.
(d) Does t h e preliminary ranking proposed by t h e system c o r r e c t l y r e p r e s e n t prevalent committee p r e f e r e n c e s ? If not, should t h e p a r a m e t e r E b e modified?
A f t e r t h e s e discussions, a r e t u r n to any previous points on t h e agenda i s pos- sible. If t h e committee decides t h a t t h e decision problem h a s been sufficiently clarified, i t can p r o c e e d t o t h e final,
ninth point
on t h e agenda: agreement on t h e aggregated ranking o r selection of one o r more alternatives. I t i s important t o stress again t h a t t h e committee need not stick t o t h e ranking proposed by t h e sys- tem, since t h e p u r p o s e of t h i s ranking-
as well as of a l l information p r e s e n t e d by t h e system-
i s t o clarify t h e decision situation r a t h e r t h a n t o p r e s c r i b e t h e action t h a t should b e taken by t h e committee.4. IMPLEMENTATION ASSUMPTIONS BND EXTENSIONS
A p r o t o t y p e implementation of SCDAS on t h e IBM-PC ( o r compatible computer) i l l u s t r a t e s t h e possibilities of t h e system and s t r e s s e s g r a p h i c presentations t o en- s u r e user-friendliness.
This implementation in BASIC s e r v e s actually only as a n illustration but con- tains a well-documented t u t o r i a l example (of scientific recruitment committee work) and h a s a u s e r ' s manual t h a t allows a n a v e r a g e u s e r t o work with t h e system on his o r h e r own problems. The implementation i s limited in s e v e r a l aspects:
-
only aspiration levels, not aspiration and r e s e r v a t i o n levels, a r e used in t h e de- cision p r o c e s s and in t h e aggregating achievement function;-
disagreement indicators a r e computed only f o r aspirations, not f o r s c o r e s ;-
g r a p h i c illustrations, though quite r i c h , d o not y e t completely r e p r e s e n t informa- tion t h a t might b e useful at various s t a g e s of t h e decision process.This implementation is available from A. Lewandowski at IIASA. Work on t h e next implementation
-
with much more professionally t r e a t e d system details-
i s inp r o g r e s s . The new implementation is designed not only to overcome t h e shortcom- ings listed above, but will a l s o a d d r e s s some new issues, such as r e p r e s e n t a t i o n s of uncertainty in scoring, a joint d a t a b a s e of information r e l e v a n t t o t h e decision p r o c e s s and r e s e r v e d fields of t h e d a t a b a s e f o r use by individual members.
REFERENCES
Dinkelbach. W. (1982). Entscheidungsmodelle, Walter d e Gruyter. Berlin, N e w York.
EXPERT CHOICE, Decision S u p p o r t Software Inc., 1983.
Johnson, S . (1984). Decision s u p p o r t f o r committee selection, I n t e r n a l R e p o r t , International Institute f o r Applied Systems Analysis, Laxenburg, Austria.
Henrion, M. (1981). An Overview of DEMOS. Dept. of Engineering and Dept. of Public Policy. Carnegie-Mellon University, Pittsburgh.
Keeney, R.L. and Raiffa, H. (1976). Decisions with Multiple Objectives: P r e f e r e n c e s and Value Tradeoffs. Wiley, N e w York, 1976.
LIGHTYEAR
-
The Decision S u p p o r t Software, Lightyear, Inc., 1984.Roy, B. (1971). Problems and methods with multiple objective functions, Math. P r o - gramming, Vol. 1, pp. 233-236.
S a a r i , D. (1982). Inconsistencies of weighted Voting Systems. Math. of Operations Res. Vol. 7.
S a a t y , T.L. (1982). Decision Making f o r Leaders: The Analytical H i e r a r c h y P r o c e s s f o r Decisions in a Complex World, Lifetime Learning Publ., Belmont.
Simon, H. (1958). Administrative Behavior, Mcmillan, New York.
Tversky, A., Kaheman, D. and P . Slovic (1983). Judgement Under Uncertainty:
Heuristic a n d Biases, Cambridge University P r e s s .
Wierzbicki, A.P. (1982). A mathematical basis f o r satisfying decision making, Math.
Modelling, Vol. 3, pp. 391-405.
Wierzbicki, A.P. (1985). Negotiation and mediation in conflicts: p l u r a l rationality and i n t e r a c t i v e decision p r o c e s s e s . In: M. G r a u e r . M. Thompson, A.P.
Wierzbicki, eds: P l u r a l Rationality and I n t e r a c t i v e Decision P r o c e s s e s , Proceedings Sopron, Springer-Verlag, Berlin, Heidelberg, New York, Tokyo.
Wierzbicki, A.P. (1986). On t h e completeness and constructiveness of p a r a m e t r i c c h a r a c t e r i z a t i o n s t o v e c t o r optimization problems. To a p p e a r in OR-Spektrum.
