Issues in Model Validation

(1)

NOT FOR QUOTATION WITHOUT PERMISSION OF THE AUTHOR

ISSUES IN MODEL VALIDATION

Andrzej Lewandowski March 1 9 8 1

WP-81-32

Working Papers are interim reports on work of the International Institute for Applied Systems Analysis and have received only limited review. Views or opinions expressed herein do not necessarily represent those of the Institute or of its National Member Organizations.

INTERNATIONAL INSTITUTE FOR APPLIED SYSTEMS ANALYSIS A-2361 Laxenburg, Austria

(2)

AKNOWLEDGEMENTS

The a u t h o r would l i k e t o t h a n k P r o f e s s o r A n d r z e j W i e r z b i c k i f o r many u s e f u l d i s c u s s i o n s , comments a n d s t i m u l a t i . o n t o b e g i n r e s e a r c h i n t h e f i e l d o f model v a l i d a t i o n .

(3)

CONTENTS

INTRODUCTION

2 . VALIDATION: DEFINITIONS

3. MODEL ATTRIBUTES

4. SYSTEM ATTRIBUTES 5. VALIDATION ATTRIBUTES

Modeling for Understanding Modeling for Forecasting

Modeling for Scenario Analysis Optimization Models

6. VALIDATION PROCESS

7 . CONCLUSIONS

APPENDIX A: BIBLIOGRAPHY ON MODEL VALIDATION APPENDIX B: TERMINOLOGY FOR MODEL CREDIBILITY REFERENCES

(4)

ISSUES IN MODEL VALIDATION Andrzej Lewandowski

1 . INTRODUCTION

It is commonly agreed between modeling methodologists that model validation is one of the most important stages in the model building process. Many papers addressing this subject have been published and an SCS Technical Committee on Model Credibility has been established in order to generalize and summarize the experi- ences in this field (see Appendix)

.

However, at the present

stage of research there are almost no suggestions concerning concrete methods of validation. Practically all authors only discuss definition of validation - not methods. The number of papers

dealing with methods of model validation is also rather limited.

The reason for this gap between methodological consciousness and the practice of model building seem to be obvious

-

^{the dis-}

cussion stays at too h.igh a level of abstraction. In general, all authors consider "model" as a description of reality, and on this level of concretization it is only possible to generate

rath.er general statements, frequently true but without operational meaning. The author of this paper believes that, in order to ex- amine validation methods, it is necessary to specify more precisely the model under consideration, the properties of the model, the modeling techniques, and, most importantly, the purpose of the model.

The aim of this paper, threfore, is to present a classification of models and an analysis of the modeling process from the point of view of model validation. At this stage of the investigation, however, it is not yet possible to design, nor to analyze, methods of validation. Our goal is to design a framework for model validation as a first and important step in

establishing a model validation methodology.

(5)

2. VALIDATION: DEFINITIONS

There are variousdefinitions for model validation, but all are very similar and have been summarized by SCS Technical

Committee on Model Credibility (1979). This set of modeling methodology definitions and concepts is quite precise and clear.

...

(model validation is) substantiation that a computerized model within its domain of applicability possesses a satisfactory range of accuracy consistent with the intended application of the model.

This definition also coincides very well with the definitions given by, for example, Naylor (1972) and Mihram (1974). The most interesting consideration of validation methodology, however, can be found in Mankin et al. (1975), where a more formal definition is given.

...

model is valid if its behaviour corresponds to system behaviour under all conditions of interest.

A model is considered invalid if we can devise an experiment in which the model outputs disagree with system measurements within the specified area of interest.

..

Similar notions have also been investigated by Beck (1980) : A

somewhat broader notion I s that of usefulness "...a model is useful if it accurately represents some of the system behavior and useless if it does not." (Mankin et al.)

Model validity can be related to model reliability and adequacy:

--

reliability is defined as the fraction of the model outputs which correspond correctly to system outputs;

--

adequacy is the fraction of system outputs which can be modeled correctly.

In the definitions formulated above, "model output" should be understood in a rather general sense and by "output" is meant the result of the modeling experiment.

Since the last two concepts have more definite operational meaning and can be relatively easily measured and computed, they can be treated as more practical tools for model testing and choosing between alternative models. These more qualitative model validity measures imply application possibilities of more advanced techniques, for example, statistical hypothesis testing

(Greig 1979). Hence, there is now a good terminological background for model validation in the sense that we know generally what model validation means. There remains open, however, the problem of how to validate a given model.

3. MODEL ATTRIBUTES

A large number of mode2 a t t r i b u t e s can be listed, but only three of them seem to be interesting for model validation pur-

(6)

poses. The first attribute can be called m o d e l b a c k g r o u n d which gives information on the natural and behavioral background of the model. This attribute determines to what extent basic consideration and natural laws have been applied when building the model. Hard models with a natural background are built on the basis of well established natural laws, for example, such precise and well-

defined concepts as mass or energy balances, variational mechanical principles, etc. In other words, the validity of these models

can be judged on the basis of well-known and accepted theories.

This type of validity consideration can be called i n t e r n a l

v a l i d a t i o n , and consists of checking the preservation of the

basic laws which have been used when building the model. Models of electrical circuits, technological processes, and selected

environmental problems (water quality) are examples of hard models with natural backgrounds.

At the other end of the spectrum we have soft models with behavioral background. They are formulated on the basis of more inductive analysis of system behavior

-

without such a p r i o r i

knowledge of natural laws governing the system under consideration.

In many important practical cases we must hypothesize when dealing with system behavior, either because of the complexity of the

system, large numbers of factors, or because of an insufficient level of basic knowledge dealing with the phenomena being modeled.

This situation frequently arises in the modeling of social,

environmental, or economic systems. Similar considerations have been performed by Kalman (1979):

...

the usual procedure of making a model of a system is obvious. A catalog of known facts,and data is com- piled and equations are written down by taking into ac- count all available quantitative information

...

^An

absolutely essential assumption for this process to work is that the "laws" governing physical phenomena are independent of the system context... Oversimplifying a bit, no matter what system is built, who builds it, how it is built, and why it is built, Ohm's law is immutable. The essential feature of economics is that this is simply not so... There are no "laws" in economics as this term is understood in physics, because economics is a system- determined science

...

Similar concepts of hard and soft models have also been introduced by Beck (1980)-, but his definition is a little bit broader.

The second model attribute relates to the logical t y p e o f

t h e model. One can consider two types of models

-

c a u s a l and

d e s c r i p t i v e . Causal models can be built if one can distinguish

between cause and effect and the input and output variables in

the system consideration. According to Zadeh's (1963) terminology, these models should be called "oriented models." Descriptive

(or nonoriented) models are built on the basis of correlation analysis, without distinguishing between inputs and outputs.

Correlation analysis makes it possible to test the dependence between various variables, but cannot give conclusive evidence about cause and effect. Independent information on natural laws and logical relations governing the system under consideration

(7)

is needed to establish a causal relationship. Most of the econometric and regression-based models belong to this class. A

typical example is a model of dependence between the weight and height of individuals in a population. There is a strong correlation between these variables, but what is cause and .effect, what is input and output?

This second attribute is rather important from the point of view of validation methodology: causal models can be subjected to simulation experiments, while such experiments are not possible in the case of descriptive models. In other words we can

experiment with modeling to answer what will happen with a specific input signal. This kind of experiment cannot be performed for the model mentioned above. It is possible, however, to use a formally obtained relationship between height and weight (usu- ally inthe form of a linear equation) but such an experiment is not very sensible.

The third attribute, called the interpretative type of model, is related to the way in which the modeling results are

interpreted. Here we can distinguish between probabilistic and nonprobabilistic (or deterministic) approaches to model interpre- taticn, although there are also other ways of including uncertainty in model interpretation (e.g., the fuzzy approach). It is necessary to stress here that:

--

the same model can be interpreted in both.!ways. For example, we can use a linear model estimated on the basis of least squares analysis, and interpret the results in terms of a probabilistic analysis, or compare only judgementally the numbers obtained from measurements and from the model. Thus, the interpreta-

tive type of model depends on the methods of analysis rather than on the form of the model.

--

the interpretative type of the model does not depend on the nature of the real world. The assumptions about the deterministic or indeterministic nature of the real

world is a purely philosophical hypothesis and has nothing to do with the type of models we use: we can describe a deterministic world using probabilistic models and vice versa.

The interpretative type of models automatically determine the possible tools for model validation. The only difficulty relates to the necessity of specifying assumptions about the model environment. In fact, when using probabilistic models it is also necessary to build models of the environment of the base model, for example, statistical properties of measurement errors. It is then necessary to validate these additional models, which, of course, causes further technical difficulties.

In the case of deterministic models, the situation is even more difficult: there are no formal methods of model validity analysis. The only possibilities here are sensitivity analysis and heuristic methods (visual inspection of the results, judge- mental estimation, etc.). Model adequacy can then be tested only

in a qualitative way. We are now able to characterize the model in terms of the attributes formulated above, and hopefully can

(8)

suggest tools for model validation connected with every attribute.

Possible situations are presented in Figure I . Let us briefly consider the existing combinations (eight possibilities). Some of these combinations seem to be empty, for example, it does not seem possible to build a natural and descriptive model, or to build a descriptive and deterministic one. The suggestions dealing with possible validation tools, however, can be formulated rather automatically, on the basis of previous considerations.

These suggestions have been collected in Figure 2. It can be seen, for example, that for a natural, causal, and deterministic model one can use an internal validity approach based on a simulation approach supported by sensitivity analysis and judgmental evaluation. If the last attribute is "probabilistic" we can also use internal validity based on simulation techniques but using probabilistic methods to interpret the results (Klejinen 1974)

.

These statements seem to be rather general and, of course, do not constitute a solution to the problem, but provide instead guidelines for the solution of a concrete problem. Moreover, for some combinations of model attributes there are no existing tools for model validation. Thus, on the basis of these investigations, we can see what kind of methods should be used in future and what classes of validation techniques are interesting from the practical point of view. It is necessary to point out here the model attributes listed above are incomplete. It is, of course, possible to formulate many other attributes but they are not so important from the point of view of model validation; however they do have influence on the validation process, and for this reason we shall call them "secondary attributes." In this way we obtain two model classification levels. It is also necessary to point out that these attributes can be essential at the early model building stage to determine possible technical tools for the modeling. These secondary attributes consist of the following:

--

^linearity

-

nonlinearity

--

time constant

-

time dependent

--

continuous time

-

discrete time

--

^dynamic

-

^static

4. SYSTEM ATTRIBUTES

The model is only the first component in the validation process. The second component is the system or the real world.

Clearly, system attributes and their relationship to model attributes will influence the validation methodology.

The first attribute we shall consider is the e x p e r i m e n t a l

t y p e of t h e s y s t e m . This attribute determines which kind of ex-

periments can be performed with the system. Three possible situations may occur:

1 . The system is a design abstraction, not yet existing in the real world and there is no experimental basis for modeling.

This kind of situation arises very frequently in engineering problems when determining new systems: modeling is then used to test complicated projects. As the real system does not exist,

(9)

Figure 1. Model Attributes and Classes

(10)

; BEHAVIORAL ? ^!

!

I I

1

ⁱ

_I

1 ^I

INTERNAL

t

VALIDITY

!

_I

NATURAL I ( b a s i c l a w s of p h y s i c s

I e t c . )

i _-. I

/ S 1MULATIO:J TECHNIQUES

i

I

D E S C R I P T I ?

1

:

/HYPOTHESIS

j I S T I C

! ~TESTING

F i g u r e 2 . Model C l a s s e s a n d E x i s t i n g V a l i d a t i o n Methodologies

(11)

there is no "reality" which can correspond tothemodel. In every realistic situation system being modele-d, however, there is a correspondence with reality; practically every new system under construction consists of components already applied in other existing systems. This means that the model consists of s u b m o d e l s which have previously been tested. A good example is chemical engineering modeling where new technology connects a series of apparatus (reactors, distillation columns, mixers, etc.). Models of these apparatus are well known and in this case we are able to

e x t r a p o l a t e our knowledge. Models consisting of well-validated

submodels will probably be valid, and this kind of approach can be called c o m p o n e n t v a l i d a t i o n .

2. The system exists in the real world, but it is not possible to make active experiments. This is the situation which arises most frequently. It occurs in economic and social system modeling, and environmental and technological problems. The

"reality" in this case is a data record which in most instances is too short and of too low a quality. This situation makes things rather difficult from the point of view of model validation. Because of a small data base, typical statistical methods frequently cannot be applied. A possible solution is to apply the extended model concept developed by Wierzbicki (1977). The extended model is built starting with the basic model in question and supplementing it by models of possible differences between the basic model and reality from a p r i o r i knowledge of system properties and partially validated by existing measurements. The extended model is then treated as the "real world" for evaluation and verification of the simplified model. This concept has been applied with success in the modeling of technological processes

(in chemical engineering, gas and water transmission systems).

The author also believes it is possible to apply this concept to environmental systems modeling (e.g., water quality problems) or even economic systems.

3. The system exists in the real world and it is possible to make a series of active experiments. This is the best situation, of course, but it occurs very rarely. In this case we

have good support for model validation; it is possible to generate as much. data as necessary, to apply experiment design techniques, and so on. Statistical methods can be applied as well as those described in the literature (for examples of Turing test and extensions, see Schruben, 1980; for hypothesis testing, see Greig, 1979). Possible situations in the model validation process are shown in Figure 3.

5. VALIDATION ATTRIBUTES

Let us now consider the validation process. It is obvious that this process depends both on the model and system attributes and that it is necessary to combine them; some combinations, however, limit the number of possible validation approaches. It is not possible, for example, to use statistical methods for analyzing the validity of a deterministic model. Model type, however, is only one of the important attributes of the validation procedure. Two other important aspects are the model purpose and the relationship between the model and the real world.

(12)

P O S S I B L E EXPERIMENTS

- L

PLANNING

APPROACH

FOR UNDER- FOR PRE- FOR SCEN- FOR O P T I - FIODEL

STANDING D I C T I O N A R I O ANAL. M I ZATION P U R P O S E

F i g u r e 3 . P o s s i b l e s i t u a t i o n s i n t h e model v a l i d a t i o n process.

(13)

Many authors point out that the model validation process should be goal-oriented, however, it is not an easy task explain- ing what this statement means. Let us consider possible situations:

Modeling for Understanding

In many instances, the only modeling goal is to understand the system structure and its behavior better. The modeler can perform simulation experiment, he can "play" with the model in order to observe what will happen in certain situations. One of the most important advantages of such experiments is the fact that it is then possible .to view the internal structure of the model and see the processes "inside" the investigated phenomena.

This kind of investigation is especially popular in physics and astrophysic research, and has also been utilized in ecological research (Mankin, et al., 1975)

.

The main problem that arises with validation is the relationship between the s t r u c t u r e of t h e p r o c e s s and t h e s t r u c t u r e o f

t h e model. According to the terminology introduced above, the

internal validity (or model testing "part-by-part" 1 should be performed in this case. One other factor can also be important:

that the model should pose a level of "internal stability" with respect to data. Sensitivity analysis is then recommended for checking this property. "Sensitivity" should be understood here

in a rather broad sense. During the modeling process we make a number of assumptions dealing with the external world (system neighborhood), model structure and model parameters, and one

of the goals should be the exploration of the influence that these assumptions have on model behavior. It is necessary to mention here that a single simulation run without more exact analysis is of little practical value from the point of view of understanding the system. The importance of sensitivity testing has been described well by Quade (1968) :

Ordinarily there is no unique, "best" set of assumptions in modeling, but a variety of possibilities, each of which has some basis for support. A good system study will include sensitivity tests on the assumptions in order to find out which ones really affect the outcome and to what extent. This enables the analyst to determine where further investigation of assumptions is needed and to call attention to the decisionmaker to possible danger that might be present

...

Similar ideas are also considered in Quade and Findeisen (1980).

There are many formal tools for sensitivity analysis and basic concepts have been considered by Tomovic (1970) and Wierzbicki

(1977). Especially interesting is the general framework for senstlvity analysis developed by Wierzbicki and his concept of basic and extended models. There are also a number of good examples of model sensitivity analysis, especially in ecosystem modeling (see, for example, Rose and Harmsen 1978). A lot of research in this direction has also been performed at IIASA:

(14)

sensivity analysis for energy models (Konno and Srinivasan 1974;

Suzuki and Schrattenholzer 1974), for demograph.ic models (Arthur 1980; Willekens 1976) as well as some more general investigations

(Stehfest 1975). There are, of course, many other excellent works available in the literature (see, for example, Thornton, et al.,

1979) but because of lack ofspace these will not be considered in detail here.

It is necessary to point out here, however, that the existing methods of sensitivlty analysis are only local and parametric.

This means that it is rather difficult to investigate large devi- ations of parameters and structural changes in the model. All methods are also only applicable to models continuously depending on parameters

-

there is no way to analyze sensitivity in a dis- continuous case. In the non-differentiable case, for large

parameter variations,estimation of Lipschitz constant might be a help; however, there are only a few theoretical papers on model sensitivity that deal with this question and the theoretical basis is as yet not fully advanced.

Modeling for Forecasting

This is one of the most frequent situations, and probably the most difficult one from the point of view of validation approach. This particular situation has been considered by Beck

(1980) and Mankin et al. (1975). The main difficulty arises from the fact that a well-validated model, in the sense that the

model responses correspond very well to the system outputs, does not necessarily reflect the future behavior of the system well.

The reasons seem to be rather obvious in that there can be an essentialnonstationarity in the system environment, or that there

are some additional input variables which are not considered in the model. In both cases the model is evidently inadequate although it may happen that factors not considered in the model manifest their presence only during the forecasting (model utilization) period. Mankin et al. (1975) have therefore

introduced a concept of model u s e f u l n e s s and model r e l i a b i l i t y . According to their terminology, a v a l i d model has no behavior which does not correspond to system behavior, and a u s e f u l

model predicts some system behavior correctly. It is of course obvious, however, that although generally a valid model is useful this may not always be the case. There still remains the problem, however, of how to determine the usefulness of the model, and, of course, it is not possible to do it a p r i o r i . In the case of statistical model interpretation, validation of forecasting models is understood better, and we can use these tools to

determine the model usefulness. Moreover, by applying the Bayes approach it is possible to determine the confidence intervals

for predicted system behavior. Pioneering work has been performed by Box and Jenkins (1970) andtheirmethodology is a good example of general modeling methodology. As a final test for the usefulness of the model they consider the statistical properties of the prediction error. Another criterion for model validation has been considered by Kashyap and Rao (1976) and in every case

they assume that the quality of prediction is the main criteria for model quality analysis. In this case, however, it is

(15)

necessary to assume t b t prediction will be performed many times, and only in this case can we apply probabilistic methods to

analyze th.e quality of the prediction; and consequently the quality of the model.

A different situation arises frequently in the case of economic forecasting where we have a very short data series and a prediction is only made once. This is complicated and only a few rather heuristic methods have been developed. Introductory work on this subject has been made by Waszkiewicz (1976) where

some new validation criteria for forecasting methods have been formulated and analyzed.

Modeling for Scenario Analysis

Scenario analysis model simulate the future behavior of a system on the basis of a judgementally chosen set of assumptions called scenarios and where the time horizon here may be rather long, say, 100 years. The World Global Models and the IIASA Energy Models are good examples of this type of model, and in this case there is no accepted methodology for model validation.

An additional difficulty connected with scenarios is the fact that they are also models, models of the neighborhood of

the system being modeled, andthesemodels should also be validated.

As yet, there are only a few works dealing with this problem, and much more researchinthis direction is needed. A critical analysis of the existing modeling approaches for scenario- analysis has recently been made by Kalman (1979) where he

analyzes the world models of Forrester and Meadows from a system theorist point of view. In his opinion:

...

the model consists of a system of nonlinear dif- ference equations which are analyzed by simulation.

It is a well-known fact that in such a system almost anything can happen

...

U n l e s s there is an "organizing principle" for writing down these equations and thereby

a p r i o r i controlling their properties, rather compli-

cated and erratic behavior may be expected on general theoretical grounds. Such an organizing principle is not available from theoretical economics and the naive faith that the equations (might) "represent" reality is certainly not good enough

...

Kalman also stresses the role of sensitivity analysis as a validation tool in scenario model analysis:

...

⁽they observed) that small variations in the assumed parameters and initial conditions result in gross changes in observed behavior. Since these parameter variations of the order of 2

-

10 percent are much smaller than the reasonable uncertainties

in their values on economic grounds (of the order of 30

-

100 percent), the value.of-the Meadow exercise is utterly destroyed. Any general conclusion from the model must be rejected because the behavior of the model is just not robust enough under parameter uncertainty

...

(16)

A c r i t i q u e of t h e e x i s t i n g methodology o f s c e n a r i o a n a l y s i s has a l s o been performed by S c o l n i k (19781, and Dubovsky and P i r o g o v

(19791.. P r a c t i c a l l y , s e n s i t i v i t y a n a l y s i s i s t h e o n l y method f o r v a l i d a t i n g t h e s e models. I n a c a s e mentioned by Kalman, t h i s a n a l y s i s h a s shown n o n a d e q u a c y of t h e model. However, t h e r e a r e a number o f o t h e r works a v a i l a b l e where s e n s i t i v i t y a n a l y s i s a p p l i e d t o s c e n a r i o models d o e s n o t g i v e s u c h a p e s s i - m i s t i c c o n c l u s i o n ( f o r example, Konno and S r i n i v a s e n 1974,

Suzuki and S c h r a t t e n h o l z e r 1974, and S c h r o e d e r e t a l . , 19701.

D e s p i t e these e f f o r t s and t h e u n d e r s t a n d i n g p a r t i a l l y g i v e n by them, w e must c o n c l u d e t h a t t h e methodology f o r v a l i d a t i o n of s c e n a r i o models d o e s , a s y e t , n o t e x i s t .

O p t i m i z a t i o n Models

T h e r e are three b a s i c t y p e s o f model where o p t i m i z a t i o n methods c a n be a p p l i e d , and i n e v e r y c a s e the r o l e o f o p t i m i z a -

t i o n i s quite d i f f e r e n t ; t h u s , d i f f e r e n t methods f o r model v a l i - d a t i o n s h o u l d b e a p p l i e d .

T h e f i r e t s i t u a t i o n o c c u r s when t h e p h e n o m e n a b e i n g modeled can be d e s c r i b e d i n t e r m s o f v a r i a t i o n a l p r i n c i p l e

-

^{h e r e}

m i n i m i z a t i o n ( o r m a x i m i z a t i o n ) o f something i s a b a s i c p r i n c i p l e o f s y s t e m b e h a v i o r . A t y p i c a l example i s t h e m i n i m i z a t i o n o f e n e r g y i n m e c h a n i c a l o r e l e c t r i c a l s y s t e m ; e v e r y s y s t e m o p e r a t e s i n s u c h a way a s t o minimize t h e t o t a l e n e r g y accumulated. I n this s i t u a t i o n i n s t e a d of w r i t i n g down a l l t h e e q u a t i o n s and t h e n s o l v i n g them, w e c a n f o r m u l a t e t h e f u n c t i o n by d e s c r i b i n g t h e t o t a l e n e r g y which d e p e n d s on t h e s y s t e m v a r i a b l e s . Then,

m i n i m i z a t i o n o f t h i s f u n c t j o n a l s o l v e s t h e problem and we o b t a i n t h e v a r i a b l e s a t the p o i n t o f e q u i l i b r i u m . T h i s a p p r o a c h has been i n v e s t i g a t e d by many a u t h o r s ( f o r example, Kurman 1 9 7 5 ) .

The r o l e o f o p t i m i z a t i o n i s e v i d e n t : it i s o n l y a t o o l f o r s o l v i n g the model, w h i l e t h e model i t s e l f b e l o n g s t o o n e o f t h e p r e v i o u s l y mentioned c l a s s e s .

T h e s e c o n d s i t u a t i o n o c c u r s when we want t o make some e x p e r i - m e n t s w i t h the model t o d e t e r m i n e t h e p o s s i b l e model r e s p o n s e s . I n many si!tuatsons,optimization _methodsare good t o o l s w i t h which t o p e r f o r m this t a s k . U s u a l l y w e c a n f o r m u l a t e an o b j e c - t i v e f u n c t i o n (sometimes a l s o c a l l e d t h e p e r f o r m a n c e i n d e x ) . While u s i n g a p p r o p r i a t e p a r a m e t e r i z a t i o n and o p t i m i z a t i o n p r o c e - d u r e s i t is p o s s i b l e t o i n v e s t i g a t e system r e s p o n s e s . I t i s

n e c e s s a r y t o p o i n t o u t however, t h a t v e r y o f t e n a s i n g l e o b j e c t i v e f u n c t i o n h a s n o economic o r o t h e r p r a c t i c a l meaning and s h o u l d be c o n s i d e r e d more as a t e c h n i c a l t o o l f o r d i m i n i s h i n g t h e

number of i n v e s t i g a t e d p a r a m e t e r s . C l e a r l y , i t i s more c o n v e n i e n t t o o p e r a t e w i t h low numbers o f o b j e c t i v e f u n c t i o n p a r a m e t e r s t h a n w i t h a l a r g e n u m b e r o f m o d e l s o l u t i o n s o r t r a j e c t o r i e s . I n t h i s

s i t u a t i o n , a more s t r a i g h t f o r w a r d a p p r o a c h i s t o s p e c i f y many o b j e c t i v e f u n c t i o n s w i t h good economic, o r o t h e r p r a c t i c a l , i n - t e r p r e t a t i o n s and a p p l y a n e of t h e e x i s t i n g m u l t i p l e - o b j e c t i v e o p t i m i z a t i o n methods. A s r e f e r e n c e p o i n t , a n o p t i m i z a t i o n

method d e v e l o p e d by W i e r z b i c k i (1977) i s a v e r y u s e f u l t o o l f o r a n a l y z i n g p o s s i b l e s o l u t i o n s t o o p t i m i z a t i o n models w i t h many o b j e c t i v e f u n c t i o n s . T h i s a p p r o a c h h a s r e c e n t l y been a p p l i e d

(17)

t o s e v e r a l IIASA models, s e e , f o r example, t h e i n v e s t i g a t i o n of t h e F i n n i s h f o r e s t and wood i n d u s t r y s e c t o r s ( K a l l i o 1 9 8 0 ) . I n t h i s c a s e , t h e u s e o f t h e o p t i m i z a t i o n a p p r o a c h a l s o d o e s n o t re- f l e c t d i r e c t l y on v a l i d a t i o n methodology b e c a u s e o p t i m i z a t i o n i s o n l y u s e d h e r e a s a t o o l f o r model a n a l y s i s .

T h e t h i r d s i t u a t i o n i s e s s e n t i a l l y d i f f e r e n t from t h e p r e - v i o u s o n e s i n t h a t a model i s used t o d e t e r m i n e a n o p t i m a l system o p e r a t i o n a n d t h e r e s u l t i n g d e c i s i o n s a r e t h e n a p p l i e d t o t h e

r e a l system. These k i n d o f models a r e c a l l e d d e c i s i o n and c o n t r o Z models. I t I s n e c e s s a r y t o s t r e s s from t h e b e g i n n i n g one

i m p o r t a n t f a c t which v e r y o f t e n i s o n l y i m p l i c i t l y u n d e r s t o o d : i n t h e c a s e o f d e c i s i o n and c o n t r o l models, we d e a l , i n f a c t , w i t h two models

-

t h e model o f t h e s y s t e m b e i n g o p t i m i z e d and t h e o b j e c t i v e f u n c t i o n model. T h i s d i s t i n c t i o n i s i m p o r t a n t a s it i s r e l a t e d t o t h e f o l l o w i n g o b s e r v a t i o n s :

--

s o l u t i o n s o b t a i n e d i n a d e c i s i o n and c o n t r o l model a r e o f t e n v e r y s e n s i t i v e t o t h e form o f t h e o b j e c t i v e func- t i o n s ; p r a c t i c a l l y , t h e o b j e c t i v e f u n c t i o n d e t e r m i n e s t h e s o l u t i o n of t h e problem.

--

t h e o b j e c t i v e f u n c t i o n model i s o n l y a n a p p r o x i m a t i o n o f t h e r e a l c o s t s i n many c a s e s ( e s p e c i a l l y i n s o c i a l s c i e n c e s and e c o l o g y ) and I t i s n o t p o s s i b l e t o e x p r e s s a l l t h e a s p e c t s o f the system o p e r a t i o n i n t h e same

(monetary) u n i t s .

I t i s a l s o n e c e s s a r y , t h e r e f o r e , t o v a l i d a t e t h e o b j e c t i v e f u n c - t i o n model. E s s e n t i a l m e t h o d o l o g i c a l d i f f i c u l t i e s a r i s e when c o n s i d e r i n g t h e r e l a t i o n s h i p between a d e c i s i o n and c o n t r o l model and a r e a l s y s t e m . P r a c t i c a l l y , t h e f i r s t g o a l of d e c i s i o n and c o n t r o l modeling i s t o improve t h e system o p e r a t i o n , t h a t i s , t o o p t i m i z e t h e v a l u e o f t h e r e a l o b j e c t i v e f u n c t i o n , measured on t h e r e a l system. T h i s c a u s e s s e v e r a l problems, one o f them b e i n g t h a t i t i s n o t a l w a y s p o s s i b l e t o measure r e a l v a l u e s o f o b j e c - t i v e f u n c t i o n s . A s e c o n d , and i m p o r t a n t , problem i s t h a t t h e p r o p e r t i e s o f t h e p a i r

-

model

-

r e a l s y s t e m

-

depend on t h e s t r u - t u r a l p r o p e r t i e s o f t h e c o n n e c t i o n between t h e model and t h e

system, and on t h e method o f a p p l y i n g computed d e c i s i o n s t o t h e r e a l system.

one of t h e p o s s i b l e ways o f v a l i d a t i n g d e c i s i o n and c o n t r o l models i s t o u t i l i z e t h e knowledge of an e x p e r i e n c e d s y s t e m op- e r a t o r ( a manager, a d i s p a t c h e r , o r a s i m i l a r e x p e r t f a m i l i a r w i t h system b e h a v i o r ) . I n p r a c t i c e , t h i s knowledge i s q u i t e

s u b s t a n t i a l and such e x p e r t s u s u a l l y have n o d i f f i c u l t y i n e v a l u - a t i n g computed s o l u t i o n s . T h e r e a r e a l s o more f o r m a l a p p r o a c h e s of t a k i n g e x p e r t o p i n i o n i n t o a c c o u n t , i . e . , m u l t i o b j e c t i v e

methods, d e v e l o p e d , f o r example, by R a i f f a and Keeney (1976) and t h e methods proposed by Eremin and Mazurov (1979) among o t h e r s . A v a l i d d e c i s i o n model c a n be d e f i n e d i n t h i s c a s e a s a model whose s o l u t i o n s do n o t c o n t r a d i c t w i t h t h e e x p e r t ' s o p i n i o n s . An e x t e n s i v e a n a l y s i s of t h e r e l a t i o n s h i p between the "model

-

r e a l system" p a i r c a n be found i n W i e r z b i c k i (1977) b u t s o f a r h i s r e s u l t s have o n l y been a p p l i e d t o c o n t r o l e n g i n e e r i n g

problems. However h i s methodology i s u n i v e r s a l and c o u l d a l s o

(18)

he a p p l i e d i n o t h e r f i e l d s . T h e fundamental concept i n t h i s

methodology i s t h e d i s t i n c t i o n between b a s i c l a n d extended models, mentioned e a r l i e r , and supplemented w i t h a r a t h e r e x t e n s i v e sen-

s i t i v i t y a n a l y s i s .

6 . VALIDATION PROCESS

V a l i d a t i o n i s n o t a s i n g l e a c t , it i s a p r o c e s s . I t f o l - lows from t h e f a c t t h a t model b u i l d i n g i s an i t e r a t i v e procedure.

I t i s p o s s i b l e , however, t o s e p a r a t e t h i s p r o c e s s i n t o s t a g e s , connected s t r i c t l y w i t h t h e s t a g e s o f model b u i l d i n g . I n t h e f i r s t s t a g e of model b u i l d i n g it i s n e c e s s a r y t o d e t e r m i n e t h e model t y p e , what i t s b a s i c a t t r i b u t e s a r e and what i t s r e l a t i o n t o the system being modeled i s . T h i s s t a g e of modeling and con- s e q u e n t l y t h e d e t a i l a n a l y s i s of t h e assumptions made (which can be c a l l e d " i n i t i a l v e r i f i c a t i o n n o r " h y p o t h e s i s v e r i f i c a t i o n " ) i s e s p e c i a l l y i m p o r t a n t a s any m i s t a k e s a r e c o s t l y and time con- suming. For example, a t t h i s s t a g e i m p o r t a n t a s p e c t s s u c h a s t h e p o s s i b l e a p p l i c a t i o n o f t h e d i s c r e t e t i m e model t o the con- t i n u o u s time system, s t a t i c models f o r dynamic system, e t c . , a r e d i s c u s s e d . I n any c a s e , however, t h e i n i t i a l assumption should be v e r y c a r e f u l l y analyzed t a k i n g i n t o a c c o u n t t h e purpose and p o s s i b l e f u t u r e a p p l i c a t i o n s of t h e model being developed.

I n t h e second s t a g e o f model b u i l d i n g , when t h e model i s being f o r m u l a t e d and computerized, it i s n e c e s s a r y t o v a l i d a t e t h e "model i t s e l f , " t h a t i s , w i t h o u t t a k i n g i n t o a c c o u n t t h e modeling purpose. One of t h e q u e s t i o n s a t t h i s s t a g e i s t h e r e l a t i o n s h i p between the computerized model and the c o n c e p t u a l model obtai'ned i n t h e f i r s t s t a g e . I n o t h e r words, the c o r r e s - pondence between t h e model, the i n i t i a l knowledge of t h e modeled phenomena and t h e e x p e c t e d model b e h a v i o r should be checked.

According t o Hermann terminology t h i s s t a g e of model v e r i f i c a t i o n can be c a l l e d " f a c e v a l i d i t y n " . . . f a c e v a l i d i t y i s a s u r f a c e o r i n i t i a l i m p r e s s i o n . o f a s i m u l a t i o n o r game's r e a l i s m " (Hermann

1 9 6 7 1 . From the methodological p o i n t o f vfew, however, this i s n o t r e a l l y v a l i d a t i o n : this s t a g e should r a t h e r be c a l l e d a test of r e a s o n a b l e c r e d i b i l i t y o f t h e model. I n many c a s e s , i n - formation can be o b t a i n e d from e x p e r t s ( o r managers) t h a t c o u l d judge w h e t h e r t h e model is r e a s o n a b l e . I n o t h e r c a s e s more f o r - mal methods can a l s o be used.

T h e t h i r d s t a g e of v a l i d a t i o n depends s t r i c t l y on t h e purpose of modeling, and f o r t h i s r e a s o n t h i s s t a g e c a n be c a l l e d

" e s s e n t i a l v a l i d a t i o n . " P o s s i b l e q u e s t i o n s a r i s i n g from t h i s s t a g e have a l r e a d y been c o n s i d e r e d i n a p r e v i o u s s e c t i o n and w i l l n o t be r e p e a t e d h e r e . I t i s u s e f u l , however, t o stress t h e d i f - f e r e n c e between " f a c e v a l i d i t y " and n e s s e n t i a l v a l i d i t y . " Con- s i d e r f o r example, a model f o r p r e d i c t i n g f u t u r e system o u t p u t s . Face v a l i d a t i o n i s concerned w i t h t h e correspondence of model o u t p u t s t o p a s t h i s t o r i c a l d a t a , where e s s e n t i a l v a l i d a t i o n i s concerned w i t h t h e q u a l i t y of p r e d i c t i o n . I t i s obvious t h a t we cannot e x p e c t good p r e d i c t i o n s from a model which h a s been rejec- t e d a t t h e f a c e v a l i d a t i o n s t a g e ; however, a p o s i t i v e f a c e v a l i - d a t i o n cannot g u a r a n t e e good q u a l i t y p r e d i c t i o n s . Face v a l i d a -

(19)

tion can be interpreted as a sieve for the selection of models before further, more complicated stages of validation are performed.

7. CONCLUSIONS

In this work, a framework for model validations has been proposed. The main conclusion is that the problem of model validation can be more strictly defined by analyzing in more detail the model itself and the purpose of modeling. On the basis of this analysis it is possible, in many specific cases, to propose appropriate tools for model validation. The problem still remains, however, of putting these tools to the best use. Moreover, in

many important cases such tools do not exist, or are insufficiently developed. In the author's opinion, a more detailed analysis of possible situations, appropriate tools, and their use is an interesting and important direction to take in model validation research.

There is evidence, of course, that it is not possible -to develop a l l validation methodologies at IIASA (all in the sense of all possible combinations of model attributes, system attributes etc.). It is possible, however, to propose some directions for the research to take which can be interesting from IIASA's point of view and these should be extensively developed.

The first research direction deals with validation methodology of m o d e l s f o r s c e n a r i o a n a l y s i s . There are a number of models developed at IIASA for scenario analysis

-

energy models, economic models

-

and some introductory work in sensitivity analysis has been performed already (Konnon and Srinivesan 1974, Suzuki and Schrattenholzer 1974), but alarger effort in this direction

should be made. New methods for sensitivity analysis especially should be developed, or existing methods should be adapted for this purpose. The main difficulties arise because of the large complexity of these models and the large number of uncertain parameters, and for these reasons the standard methods cannot be applied in a straightforward way.

The second research direction deals with validation methodology of e c o n o m i c f o r e c a s t i n g m o d e l s . A good example of this type of modeling are the models developed in the Fbod and Agri- culture Program. The especially interesting problems in this field deal with the influence of data quality on the modeling results, parameter estimation on the basis of very short data series, stationarity of the model parameters, etc.

The third research direction deals with the d e c i s i o n a n d

c o n t r o l m o d e l s . Further development of the Wierzbicki approach to

multiobjective optimization and sensitivity analysis seems to be very promising. There are also a number of areas for possible application of thse methods, for example, sensitivity analysis of optimal control economic models. The role of the decisionmaker (or expert) in decision and control models should also be

investigated.

The fourth and last direction deals with the e c o l o g i c a l

m o d e l s . In this case application of an extended model concept

also seems to be very promising, especially for the analysis and simplification of distributed-parameter models.

(20)

APPENDIX A : BIBLIOGRAPHY ON MODEL VALIDATION

The b i b l i o g r a p h y p r e s e n t e d h e r e i s , o f c o u r s e , n o t c o m p l e t e however i n t h e a u t h o r s o p i n i o n r e f l e c t s q u i t e w e l l the c u r r e n t s t a t e - o f - t h e - a r t i n t h i s f i e l d .

J o u r n a l P u b l i c a t i o n s

1967 C.F. Hermann

-

V a l i d a t i o n P r o b l e m s i n G a m e s and S i m u l a t i o n s w i t h . S p e c i a l R e f e r e n c e t o Models o f I n t e r n a t i o n a l P o l i t i c s . B e h a v i o r a l S c i e n c e , 1 2 .

1968. R . D . Smallwood

-

^A D e c i s i o n A n a l y s i s o f Model S e l e c t i o n . IEE T r a n s a c t i o n s o n System S c i e n c e a n d C y b e r n e t i c s , SSC-4

(-3 1

.

1972 C . A . M i h r a m

-

The M o d e l l i n g P r o c e s s . IEEE T r a n s a c t i o n s o n S y s t e m , Man, a n d C y b e r n e t i c s , SMC-2(5).

C.A. Mirham

-

Some P r a c t i c a l A s p e c t s o f t h e V e r i f i c a t i o n a n d V a l i d a t i o n o f S T m u l a t i o n Models. O p e r a t i o n a l Research Q u a r t e r l y , 2 3 ( 1 1 .

1974 D. Mirham, G . A . Mirham

-

Human Knowledge, The R o l e o f M o d e l s , M e t a p h o r s a n d Analogy. I n t e r n a t i o n a l J o u r n a l o f G e n e r a l S y s t e m s , 1:41-60.

G . A . Mirham

-

S i m u l a t i o n M o d e l i n g . I n t e r n a t i o n a l J o u r n a l o f G e n e r a l S y s t e m s , 1:155-163.

J . F . Barman

-

W e l l - P o s e d e n e s s o f Dynamical S y s t e m Models.

I n t e r n a t i o n a l J o u r n a l o f S y s t e m S c i e n c e s , 5 ( 4 ) .

1975 T . J . T e o r e y

-

V a l i d a t i o n C r i t e r i a f o r Computer System Sim- u l a t i o n s . S i m u l e t t e r , 6 ( 4 ) : 9 - 2 0 .

J . R . B u r n s

-

E r r o r A n a l y s i s o f N o n l i n e a r S i m u l a t i o n s

-

A p p l i c a t i o n o f World Dynamics. IEEE T r a n s a c t i o n o n S y s t e m , Man, a n d C y b e r n e t i c s , SMC-5(3)

(21)

P. Delatte

-

On Methodology for the Elaboration of a Theoretical Mode. International Journal of General Sys- tems, 2.

R.W. Hemming

-

How do you Know the Simulation is Relevant?

Simulation, November.

G.A. Mihram

-

The Scientific Method and Statistics in Simu- lation. Simulation, January.

M.G. Reggiani, F.E. Marchetti

-

On Assessing Model Adequacy.

IEEE Transaction on System, Man and Cybernetics, SMC-5(3).

1976 S. Kahne

-

Model Credibility for Large-Scale Systems. IEEE Transactions on System, Man and Cybernetics, August.

1977 OSSIM

-

Credibility and Implementation. Simulation, August:

VII-IX

A. Bradshaw, D. Dairith

-

An analysis of "Industrial

Dynamics." International Journal on System Science, 8(10).

S.P. Chakrevarty

-

Econometric Models

-

Mathematical Ex- pression of Personal Prejudices? IEEE Transacations on Systems, Man and Cybernetics, June.

P.J. Vermenlen, D.C.J. De Jough

-

Growth in a Finite World

-

A Comprehensive Sensitivity Analysis. Automatica, 13.

1978 A.G. Barto

-

Discrete and Continuous Models. International Journal on General Systems, 4.

H. Nurmi

-

On Strategies of Cybernetics Models Building.

Kybernetes, 7.

W.S. Overton, C. White

-

Evolution of a Hydrology Model

-

an Exercise in Modeling Strategy. International Journal on General Systems, 4.

J. Pearl

-

On the Connection Between the Complexity and Credibility of Inferred Models. International Journal on General Systems, 4.

M.R. Rose, R. Harmsen

-

Using Sensitivity Analysis to Simplify Ecosystem Models

-

A Case Study. Simulation, July.

1979 Simulation Project Evaluation. Simulation, January:VII-X I.D. Greig

-

Validation, Statistical Testing and the De- cision to the Model. Simulation, August:55-60.

Terminology for Model Credibility. Report of the SCS Tech:

nical Committee. Simulation, March: 103-104.

J.P.C. Wejinen

-

Regression Metomodels for Generalizing Simulation Runs. IEEE Transactions on Systems, Man and Cybernetics, SMC-9 (2)

.

K.E. Sahin

-

System Dynamics Models: Some Obscurities.

IEEE Transactions on Systems, Man and Cyberntics, SMC-9(2).

A. Ford, O.C. Gardiner

-

A New Measure of Sensitivity for Social System Simulation Models. IEEE Transactions on Systems, Man and Cybernetics,SMC-9(3).

K.W. Thornton, A.S. Lessem, D.E. Ford, C.A. Stirgens

-

Improving Ecological Simulation Through Senstivity Analy- sis. Simulation, May.

M.G. Reggiani, E. Halfon

-

Comments to "Adequacy of Eco- sytem Models." ISEM Journal, 1:3-4.

V.A. Gelovani, S.V. Dubovsky, V.V. Yurchenko

-

~ethodological Problems in Global Models. Simulation, July.

(22)

1980 L.W. Schruben

-

Establishing the Credibility of Simula- tions. Simulation, March:101-105.

D.C. Montgomery, R.G. Conrad

-

Comparison of Simulation and Flight-test Data foryissile Systems. Simulation, February:63-72.

Verification, Validation and Transfer of Societal Models.

Simulation, February:VII-IX.

Conference Proceedings

1970 U. Larsson, R. Lundin

-

Some Methodological Views on Digital Simulation, Models and Simulation. Proceedings of a Sym- posium, Gothenburg School of Economics.

1972 T.H. Naylor

-

Simulation and Validation. Proceedings of the Sixth IFORS International Conference on Operational Research, Dublin.

1974 F.J. Charlwood, M. Norton

-

Uncertainty in World Modeling, New Trends in Mathematical Modelling. Proceedings of the Second International Seminar on Trends in Mathematical Modelling, Warsaw.

1975 D.K. Miller

-

Validation of Computer Simulations in the

Social Sciences. Proceedings of the Sixth Annual Pittsburgh Conference, 6(2):743-747.

J.C. Free

-

A Model of the Yodelling Process. Proceedings of the Sixth Annual Pittsburgh Conference, 6(2):747-753.

G.A. Mirham

-

Are Systems Scientists not Scientists? Pro- ceedings of the Sixth. Annual Pittsburgh Conference, 6(2):

795-799.

1976 Why Can't System Scientists be Scientists? Proceedings of the Seventh Annual Pittsburgh Conference, 7(2).

1978 N.I.J. Bailey

-

The Role of Validation in Health Care

Systems Modeling. Systems Modeling in Health Care, E. Shigan Editor, IIASA, CP-78-012.

N. Saavedre

-

^Rivano

-

A Critical Analysis of the Mesanovic- Pestel World Model. Proceedings of the Sixth IIASA Global Modeling Conference, Laxenburg, Austria.

H.D. Scolnik

-

A Critical Review of Some Global Models.

Proceedings of the Sixth IIASA Global Modeling Conference, Laxenburg, Austria.

E. Halfon

-

The Effects of Data Variability in the Develop- ment and Validation of Ecosystems Model. Proceedings of

the Symposium on Modleing and Simulation Methodology, Israel.

R.E. Kalman

-

A System Theoretic Critique of Dynamic Econ- omic Models

-

Global and Large Scale Models. Proceedings of the Center for Advanced Studies, Dubrovnik, Yugoslavia.

(23)

S.V. Dubovsky, G.G. Pirogov

-

Critical Survey of Some Global Modeling Approaches

-

Global and ~arge'scale Models. Pro- ceedings of the Center for Advanced Studies, Dubrovnik, Yugloslavia.

1979 Validation and Assessment Issues of Energy Models. Pro- ceedings of a Workshop held at the National Bureau of Standards, Gaithersburg, Maryland, January.

IIASA Reports

1974 A. Suzuki, L. Schrattenholzer

-

Sensitivity Analysis on Hydrogen Utilization Factor of the Haefele-Manne Model, RM-74-30.

H. Konno, T.N. Srinivesan

-

The Haefele-Manne Model of Reactor Strategies: Some Sensitivity Analysis, RM-74-19.

1975 H. Stehfest

-

Decision Theoretical Remarks on Sensitivity Analysis, RR-75-03.

1976 F. Willekens

-

Sensitivity Analysis, RM-76-49.

1977 R. Rinaldi

-

Sensitivity Analysis of Strecker-Pelps Model, RR-77-01.

1979 P. Markovich

-

Sensitivity Analysis of TECH 1

-

^{A Systems}

Dynamic Model for Technological Shift,. WP-79-78.

1980 W. Findeisen, E.S. Quade

-

The Method of Applied Systems Ans.lysis: Finding A Solution, WP-80-62.

Books

E.S. Quade, W.I. Boucher

-

Systems Analysis and Policy Planning.

New York, Elsevier, 1968.

R.C. Meir, W.T. Nevell, H.L. Parer

-

Simulation in Business and Economics. Prentice Hall, 1969.

J.R. Ernhoff, R.L. Sisson

-

Design and the Use of Computer Simu- lation Models. London: The MacMillan Company, 1970.

G.A. Mirham

-

Simulation-Statistical Foundations and Methodology.

New York and London: Academic Press, 1972.

M. Inbar, C.S. Stoll

-

Simulation and Gaming in Social Science.

New York: The Free Press, 1972.

J. Clark, S. Cole

-

Global Simulation Models, A Comparative Study.

Wiley, 1975.

J. Aitchison, I.R. Dunsmore

-

Statistical Prediction Analysis.

Cambridge: Cambridge University Press, 1975.

(24)

W.W. Schroeder 111, R.E. Sweeney, L.E. Alfeld

-

Readings in Urban Dynamics. Cambridge: Wright-Allen Press, 1 9 7 5 .

L. Waszkiewicz

-

Verification of Forecasting Procedures. PWN, War saw.

M.Greenberger, M.A. Grenson, B.L. Crissey

-

Models in the Policy Process. New York, Russel Sage Foundation, 1 9 7 6 .

B.P. Zeigler

-

Theory of Modeling and Simulation. New York: John Wiley, 1 9 7 6 .

C.W. Churchman, R.O. Mason

-

World Modeling

-

A Dialogue. Ameri- can Elsevier, North-Holland, 1 9 7 6 .

P.U. House, H. McLeod

-

Large-scale Models for Policy Evaluation.

New York: John Wiley, 1 9 7 7 .

A. Wierzbicki

-

Models and Sensitivity of Control Systems. Warsaw:

WNT, 1 9 7 7 (in Polish).

M.R. Osborne, R.O. Watts

-

Simulation and Modeling. Queensland:

University of Queensland Press, 1 9 7 7 .

R.S. Lehman

-

Computer Simulation and Modeling. New York: John Wiley, 1 9 7 7 .

N.R. Adam, A. Dogramaci

-

Current Issues in Compouter Ssmulation.

Academic Press, 1 9 7 9 .

S.L.J. Jacoby, I.S. Kovalik

-

Mathematical Models with Computers.

Prentice Hall, 1 9 8 0 .

J. Randers

-

Elements of the System Dynamics Method. MIT Press, 1 9 8 0 .

(25)

APPENDIX B: TERMINOLOGY FOR MODEL CREDIBILITY

The SCS (Simulation Council) Technical Committee on Model Credibility recently published the standard set of terminology dealing with modeling problems. For reader convenience we insert here the complete text of the report published in Simulation

March, 1979.

I ?STRODUCT I OH ~ D E L QUALIFICAION Determination of adequacy

Since the cornerstone for establishing the credibility of the CO!lCE?TJAL t:@D2:. to of a computer simulation is effcstive conmunicarion provide an acceptab:~ LEVEL bctvccn thc builder o f a simulation model and its . ^OFACfiED72:IT f o r t h e WYAit:

potential user, the SCS Tcchnlcal Comittce on bbdcl OF INTENDED AP?LICATICtl Credibility has dcvelopcd a standard set of terminol-

ogy to facil~tatc such comunicat~on.

To provide a proper framework to review the crcdibil- ity of a simulation, it is convenient to dividc the simulation cnvironncnt into thrce basic elcmcnts as dcpictcd in the following Figt~rc. The inncr arrows dcscribc the processes which relate thc clenents to cach othcr. and thc outcr arrows refer to the proccdures which evaluate the credibility of these processes.

qualification

Analysis I

I I I

Proqraminq Mode 1

va 1 ida t ion _I

I

M o d e l verificatlcn

COMPUTERIZED MOCEL An operational cmputer pro- graa which implcmcnts a CCNCEPTUAL XODEL

MODEL VERIFIG\T ION Substantiation that a CON- PUTERIZE2 E:ODEL represcr.:s a CCIICEPTUAL PiODEi within specified limits of accuracy 0 0 : U I N OF APPLICAaILITY Prescribed conditisns for

(OF COWUTERIZED HODEL) wllich the CCX4PUTERIZED MODZL has been tested. compared aqainst REALITY to the extent ,possible, and judged suitable for use (b-j MODEL VALJDATION, as described below1

RAYCE O F ACCURACY Demonstrated aqrcement be- (OF COMPUTERIZED MODEL) tween the COWUTEAIZED W S E L

and REALITY within a stipu- lated DOMAIN OF APPLICABILITY MODEL VALIDATION Substantiation that a con-

PUTERIZED MODEL within its COMAIN OF A P P L I C A B I L I n pssesses a satisfactory RAt:GE OF ACNR4CY consiscent with the intended application of c h e m o d e l

(26)

Each of t h e b a s i c clernents and t h e i r i n t e r r e l a t i o n s h i p s C"mlFICATCON a r e d e a l t w i t h i n t h e f o l l o v i n g s e t o f C c f i n i t i c n s . DOCU.\'EKPATION DESCRIPTION CI: TER~II:;OLOCY

REALITY An e n t i t y , s i t u a t i o n , o r s y s t e m whlch h a s been s e l e c t e d f o r a n a l y s i s CONCEPTUAL E:OCEL V e r b a i d e s c r i ? t i o n , equa-

t i o n s . g o v e r a i n q r e l a t i c n - s h i p s , o r " n a t u r a l laws"

t h a t . p u r p o r t t o d e s c r i b e REALITY

C@>tAIN OF INTEhDED P r e s c r i b e d conditions f o r

APPLiCXTION which t h e CONCETTUAL HOCZL

(OF CONCEPTUAL HODEL) is i n t e n d e d t o match REALITY LEVEL OF ACREE>IEl.;T Expected a g r e e m e n t between

(OF COXCEPTUAL HCDEL) t h e CONCEFTUAL K.:OCEL and

= I n , c o n s i s t e n t w i t h t h e DOEAIN OF IhTESCED APPLICATION and t h e p u r p o s e f o r v h i c h t h e a o d e l was b u i l t

?:23Ei CERTIFICATION Acc~.';"ance b y t h e m d e l s s c r oc cne CEhTIFICXTION DOCL'YZN- TATION a s a d e q u a t e ovider.ze t h ~ t t h e CCZIF'JTERIZED FICDLL c a n be e f f e c t i v e l y u t l l i z e d f o r a s p e c i f i c a p p l i c a t a o n CGYPUTER S IMULATICN E x e r c i s e of a t e s t e d a n d

c e r i i f i c d CCMPUTERIZED llCDEL t o g a i n l n s i g h t a b u t X A L I T i '

T h i s t e r m i n o l o g y u a s devclopcd by t h c committee, v h i c h i s composed o f members from d i v c r s c d i s c i p l i n e s and b a c k g r o u n d s , w ~ t h r h c i n t c n r t h a t ~t c o u l d bc cnploycd i n a l l t y p c s o f s i m u l a t i o n a p p l i c a t i o n s . C r c a t c n r c was taLcn t o d e v e l o p d e f i n i t i o n s which v o u l d bc e q u a l l y applicable t o s i m u l a t i o n s o f p h y s i c a l

s!.stcns ( c n b o 2 y 1 n ~ r c n d i l y m c a s u r a b l c phc~romc~ia) and s o c i a l and b i o l o g i c a l s y s t c n s ( f o r wl~iclr d a t a may be i l l - J c f i n d ) . AJlrcrcncc t o t h i s t c r l ; l i n o l o g y , and t h e d i s c i p l i n c i n p l i c d t l r c r c i n , w i l l g r e a t l y f u c i l i t n t e communication brtwecn v a r i o u s s i n ~ u l a t i o ~ r d c v c l o p c r s a s tic11 I S bc:ucccr d c v c l o p c r s and u s e r s . ' f l l e r c f o r c . t h c committec rccomncnds t h a t C J C ~ mcmber u s c t h i s t c r m i n o l o s y i n t l l d o c u n c n t a t i o n and p u b l i c s t i o ~ l s u h i s h pertain t o t h c c r e d i b i l i t y o f s i m u l a t i o n s .

Docunentat i o n t o Communicate l n f o r r r u t i o n :oncerniag a n o d e l ' s c r e d i b l l i t v aad a g p l i c a b i l l t y , c o n t a i n i n g , a s a mininum. t h e f o l l o w i n g b a s i c e l e m e n t s :

(1) S t a t e n e n t of p u r p o s e f o r which t h e m d e l h a s been b u i l t

( 2 ) V c r b a l a n d / o r a n a l y t i c a l d e s c r i p t i o n o f :he CC!I- CEPTUXL MODEL and COn- PUTERIZED MODEL ( 3 ) S p e c i f i c a t i o n o f t h e

DOMAIN OF APPLICABILITY

and W C E OF ACCURAC'f r e l a t e d t o t h e p u r p o s e f o r which t h e model is i n t e n d e d

( 4 ) D e s c r i p t i o n of t e s t s u s e d f o r NODEL VERIFICA- TION and NODEL VAI.IDATIOti and a d i s c u s s i o n of t h c i : adequacy