Comparing the Performance of Two Competing Models

Working Paper

C O I P A I U N G T E I E P ~ O I

Two COMPEIlWG Y m m s

K

C! F ~ d o r o u

November 1986 WP-86-73

International Institute for Applied Systems Analysis

A-2361 Laxenburg, Austria

NOT FOR QUOTATION WITHOUT THE PERMISSION OF

THE

_AUTHOR

COIPARMG

THE - P

OI

TIlO

_COYPEXING

HODEM

November 1986 UP-86-73

Worktng -pars are interim reports on work of t h e International Institute f o r Applied Systems Analysis and h a v e received only limited review. Views o r opinions e x p r e s s e d h e r e i n d o not necessarily r e p r e s e n t t h o s e of t h e Institute or of tLs Nattonal Member Organtzations.

INTERNATIONAL INSTITUTE FOR APPLIED SYSTEMS ANALYSIS 2361 Laxenburg, Austria

Computerization of t h e environmental sciences is one of t h e most typical trends of the last t w o decades. A number of different models describing t h e snme objects or phenomena are circulating In scientific media and interest in publications in which these models are compared h a s increased impressively.

Usually the comparison is based on intuitive Ideas, and t h e object of this p a p e r is to give some recommendations on how to use standard shtistical techniques f o r model comparison.The key theme consists of a n introduction to s e v e r a l statistical- ly reasonable discrepancy measures f o r competing models and their subsequent maximization by varying t h e location of a n experimental network.

In general. this p a p e r covers t h e author's results in this sector of mathematical statistics.

I am m o d grateful to Professor R.E. Mum f o r his constructive criticism and recommendations, whlch were extremely helpful in the preparation of this paper.

COMPARING THE PERFORMANCE OF TWO COWIRING YODELS

V. K Fedorov

1.

INTRODUCTION

Beginning with numerical weather prediction experiments in t h e 1920s, models of environmental p r o c e s s e s have become more and more complex, keeping p a c e with advances in computer technology. Some of t h e c u r r e n t models c a n b e run only on v e r y l a r g e computers such as t h e CRAY on which, f o r example, t h e Navier- Stokes equations are solved using v e r y s h o r t forward time s t e p s and many points in s p a c e .

Investigators have long been i n t e r e s t e d in testing t h e s e big models with field d a t a . In p a r t i c u l a r , when a new m o d e l i s proposed (due to a n advance in o u r physi- cal understanding of t h e p r o c e s s e s involved. or t o advances in computer capabili- t i e s ) i t is important t o determine whether t h e model is "better" b e f o r e adopting i t operationally for national weather f o r e c a s t s , acid r a i n predictions, etc.

One problem is t h e definition of t h e word "better", which involves value judge- ments. For example, in a n u r b a n a i r pollution m o d e l t h e predictions could b e wide- ly different from observed values simply because t h e forecast wind direction w a s 30 d e g r e e s in e r r o r . By rotating t h e axis, a much improved match of observation and prediction could b e obtained. Many similar examples could b e given in which objective c r i t e r i a must b e established and promoters of competing models may sometimes d i s a g r e e with one a n o t h e r because of t h e objective c r i t e r i a t h e y use.

Recent review a r t i c l e s on model performance in t h e a i r pollution field have been written, for instance, by Hayes and Moore (1986), Willmott (1982), Zwerver and Van Ham (1985), and a v e r y interested p a p e r by Fox (1981). H e r e w e shall t r y t o connect t h e ideas given in those p a p e r s with more formal r e s u l t s f r o m mathemat- i c a l statistics.

Probably a scientist who h a s worked with complex numerical models of physi- c a l p r o c e s s e s would b e v e r y s c e p t i c a l about t h e simplicity of t h e models con- sidered in t h i s p a p e r . Nevertheless, simple diagnostic cases illuminate t h e main ideas and final r e s u l t s , and give some orientation which usually cannot b e achieved in more complicated situations. Most certainly, t h e p r o c e s s of model comparison cannot b e imbedded in a routine scheme (even o n e t h a t i s quite p e r f e c t ) . Usually t h e r e is need for some integration of s t a n d a r d mathematical techniques with t h e in- tuition of a r e s e a r c h e r ( f o r details, see Munn, 1981).

2.

MAIN ASSUMPTIONS

Let a system "object under investigation

-

p r o c e s s of observation" b e described by t h e following m o d e l

- ^-

Yir

=

7t ( x i ) + cir (i = l , n , r = l , r i ) . (1)

A function q t ( x ) is a response function and xi is a v e c t o r of conditions under which t h e i-th set of observations are made. S u b s c r i p t "t " stands f o r "true" values which w e t r y to o b s e r v e o r measure in d i r e c t experiments and t o estimate in in- d i r e c t experiments. E r r o r s ci,, f o r instance, c a n r e f l e c t t h e imperfection of a n observation process; stochastically of a n o b j e c t under observation; approximate c h a r a c t e r of used representation f o r q t ( 2 ) ; and s o on.

One of t h e most crucial assumptions in t h e following is t h a t ci, a r e r a n d o m (stochastic) v a l u e s . This is a significant component of model (1) and one c a n s a y t h a t a stochastic model is wed f o r t h e description of ci,. I t is n e c e s s a r y t o e m - phasize t h a t t h i s assumption is essential to t h e whole concept of t h e p a p e r . Details ( t o b e supplied l a t e r ) can technically c h a n g e t h e final r e s u l t s , but they are adju- s t a b l e t o those details in t h e f r a m e of t h e main idea of t h e p a p e r .

Another significant assumption consists of t h e f a c t t h a t components of xi (or a t least p a r t of them) can be chosen (or controlled) by a n experimenter. I t will b e assumed t h a t xi E X ^C R k , where X i s compact.

The set of values

specifies a design. The f r a c t i o n s pi c a n b e considered as measures p r e s c r i b e d t o points xi and variations of t h e s e measures must b e proportional t o N-I in p r a c - tice.

The major e f f o r t s of this p a p e r will b e d i r e c t e d t o t h e case when i t is a p r i o r i known t h a t t h e function q t ( x ) ( o r t r u e r e s p o n s e ) h a s t o coincide with o n e of t h e two given functions, e i t h e r q1(2,191 o r q2(z,Q2), where 19, are p a r a m e t e r s t o b e estimated. 19, E

n, ^c

^Rmj. In general. t h e r e are no v e r y special demands t o t h e s e functions. For instance, they c a n b e numerical solutions of some system of dif- f e r e n t i a l equation. However, f o r a number of p r e s e n t e d results, t h e i r linearity ( q ( x , 9 )

=

g T f ( 2 ) ) will b e important.

What is really essential in t h e l a s t assumption i s t h a t one of the c o m p a r i n g fLcnctions coincides w i t h the t r u e r e s p o n s e . In p r a c t i c e , t h i s means t h a t a n ex-

perimenter believes in t h e closeness of ( a t l e a s t ) one model t o reality.

Cases where one needs t o compare more than two models lead t o c e r t a i n mathematical difficulties but t h e corresponding techniques are more o r less a straightforward generalization of t h e r e s u l t s p r e s e n t e d h e r e (compare with At- wood and Fedorov, 1975).

3. O ~ C R l T E R l A

The objective of a n experiment is t o choose t h e t r u e model. To start t h e dis- cussion of t h e experimental design problem one must apply t o some c r i t e r i a of op- timality (Atkinson, Fedorov, 1975; Fedorov, 1980). The main idea behind t h e s e c r i - t e r i a is based on introduction of some measure of discrepancy between r i v a l models depending upon t h e difference: q l ( z

-

q 2 ( z ,Q2)

.

To b e specific, suppose t h a t t h e f i r s t model is t r u e , i.e., t h e r e e x i s t s such t h a t q t ( x )

=

q1(~,191t)

.

If random errors cir are independent and normally dis- t r i b u t e d with variances

=

1 then i t is r e a s o n a b l e t o apply t o t h e following meas- u r e of discrepancy

The value

NT;

( t N . S l t ) coincides with t h e noncentrality p a r a m e t e r of X2-

distribution ( o r F-distribution, when t h e v a r i a n c e of cir i s unknown) if q 2 ( ~ 1 6 2 )

=

* f f Z ( z )

.

where /,(I) is a v e c t o r of t h e given basic functions. and

n2

coincides with R~'

.

In o t h e r cases ( ~ ~ ( 2 ~ 7 9 ~ ) is nonlinear o r f o r a r b i t r a r y

n)

t h i s f a c t h a s asymptotical (N-0) c h a r a c t e r . More d e t a i l s see in Atkinson, Fedorov, 1975; Fedorov, 1981.

I t will b e useful to consider also a generalized version of (2):

N

For instance, r o b u s t M-estimators can lead t o t h a t kind of discrepancy measures (see, Huber, 1981).

The design

i s called Tl-optimal design. To emphasize t h a t t h e c r i t e r i o n of optimality i s con- s t r u c t e d under assumption t h a t ql(z1791t)

=

q t ( z ) t h e design

#;

will some times b e called 'locally T-optimal design".

Together with locally optimal designs w e will consider maximin and Bayesian designs.

The design

#;

i s maximum if

#b =

A r g s u p inf inf n pi q l ( z i ,

-

q z ( z i , 79,)

=

(N *lEnl *aEn' i = l

I

⁽⁵⁾

= A r g sup inf T j ( t N , S j ) , j = 1 , 2 . (N 41 €01

where n j i s a p r i o r probability of j - t h model and k ( d 1 9 ~ ) is a corresponding p r i o r distribution of Jj ( j 4 . 2 )

.

^{t h e n} is a Bayesian optimal design.

4.

CONTINUOUS OPTMAL DESIGNS

In what follows only t h e continuous versions of optimization problems (4)-(6) will b e considered. In o t h e r words, t h e d i s c r e t e n e s s of pi i s neglected and

where t ( & ) c a n b e any probabilistic measure with a supporting set belonging t o X

.

I t is clear t h a t f o r t h e continuous c a s e t h e s u b s c r i p t "N" does not b e a r any addi- tional information and c a n b e omitted.

Formally optimization problems (4) and (5) are similar. Both of them c a n b e transformed t o t h e following optimization problem:

t' =

A r g s u p T ( t )

=

A r g s u p inf / F l q ( z , b ) j # ( d z ) .

4 4 *€a

k

T T

For instance. in (5) o n e h a s t o put d T

=

( 1 9 ~ . 79,)

.

q ( z , 79)

=

q l ( z

-

q 2 ( z , 1 9 ~ )

and

Q = Q,

X%

.

In t h e case of (4) when

Q =

^Q2 and q ( z , d )

=

q i ( z , d l t )

-

q 2 ( z , d 2 ) it is crucial t h a t t h e solutions of (8) will depend upon : t a

= t*

( d l t )

.

Let u s assume t h a t

(a) t h e sets and

Q

are compact and function q ( z , d ) b e continuous on X ^x

Q .

(b) t h e function F ( z ) i s monotonously increasing when 2 2 0 and monotonously de- creasing when z <O and continuous on

Z

= I z : z = q ( z , 9 ) , z f X , d € Q j

.

Theorem I.

(i)

There e x i s t s a t Least one s o l u t i o n of

(8).

The set of optimal designs i s convex.

(ii)

A necessary a n d s a c i e n t condition for a design t* to be optimal i s the existence of a measure

pa

( d

9 )

s u c h t h a t

where

and t h e measure pa h a s t h e supporting s e t

o* ⁼

I d ' : d 8

=Are

inf ~ F f q ( z 1 9 ) ] ~ * ( d z ) ~ , p 8 ( d 9 )

=

1 .

d e n

Q '

(iii)

The f u n c t i o n

r ( z ,

t

* )

achieves i t s u p p e r bound o n the s u p p o r t i n g set

vt*

If in addition t o (a) and (b):

(c) t h e function Ft q ( z , d )

1

i s a convex function of 9 f o r all z EX and Q i s a convex compact, then

Theorem

2.

There alwacys e z i s t s a n optimal design containing n o more t h a n m

+ 1

supporting points, where m i s the d i m e n s i o n of

9.

If, in addition t o previous assumptions: (e) t h e function F ( z ) is symmetrical, then:

Theorem

3.

The s u p p o r t i n g set of a n optimal d e s i g n for

(8)

belongs to Tche- b y s h e n eztremal basis:

( 9 '

,x*

)

= Arg

inf s u p

1

q ( z , 9 )

i .

U E Q Z E X

Theorems 1-3 a r e helpful in t h e understanding of general s t r u c t u r e of optimal designs and in some c a s e s in t h i s analytical construction (see, Fedorov, 1981; Den- isov, Fedorov, Khabarov, 1981).

In c a s e s when t h e definition of Q includes at l e a s t k linearly independent con- s t r a i n t s which are active f o r 9' , i.e., $($*)

=

⁰ , then t h e number of supporting points in Theorem 2 can b e reduced until

m

+1-k

.

Moreover, if t h e location of t h e s e points is known then t h e i r measures can easily b e calculated:

where t h e e x i s t e n c e of t h e corresponding derivatives and t h e r e g u l a r i t y of t h e ma- t r i c e s are assumed.

m m +1

Example 1. Let rll(z 1 9 ~ ) _-

=

Qla za and q2(z ,d2)

=

Q 2 a ~ a - 1

.

Assume

a = l ^- a =I

t h a t ~ ( z ) = z ~

,X =

[-1.11 and t h e r e a r e no o t h e r c o n s t r a i n t s e x c e p t t h a t m + I

g2(,

=

6

>

0

.

In t e r m s of (8) i t means t h a t ~ ( z , d )

= x

^{9, za-l and}

The supporting set of t h e Tchebysheff problem m +1

inf sup

1 x

^9,za-l

¹

d l z l s l a = 1

i s known (see, for instance, Karlin and Studden, 1966):

X * =

[z;

=

COS m + l - i r , = = 1 , r n + 1 j . m

The corresponding measures can b e calculated with t h e help of (9):

. . .

p ;

=

1/2m , p i

=

= 1 / m

.

5.

DUALITY

OF SOME MODEL TESTING

AND

PABAII[ETER ESTIMATION PROBLEMS

In t h i s section only t h e l i n e a r case when ~ ( z . 9 )

=

g T f ( z ) and F ( z )

=

z 2 will b e considered and all r e s u l t s will formulated in terms of (8).

Let us start with t h e most evident and simple case when one i s i n t e r e s t e d in some l i n e a r combination c T 9 of unknown parameters. F o r interpolation or extra- polation, c = f ( z o ) , where z o i s t h e point of i n t e r e s t . Then if o n e wants to esti- mate c T 9 , t h e following c r i t e r i o n (see, for instance, Fedorov, 1972; Silvey, 1980) c a n b e used:

where t h e s u p e r s c r i p t means pseudo-inversion, and M([)

= If

^{( z )/} '(2 )[(dz )

.

If X

t h e model q ( z ,19) i s tested u n d e r t h e constraint (c T9)2 2 1 then:

T([)=

inf q 2 ( z ) ( ) = inf s ~ M ( [ ) ~ .

(c Td) bl ⁽¹¹⁾

(c Td) bl

I t i s e a s y to check t h a t in (11). instead of 1, any positive constant c a n b e taken without influencing t h e optimal design if q ( z , 9 ) depends linearly on 9

.

A similar r e s u l t holds for t h e c r i t e r i a considered below and i t will b e used without comment.

I t i s n a t u r a l to suggest t h a t

[ ~ : c ~ M - ( ~ ) c

< -1 =

f o r any t y p e of pseudo-inverse matrix, or in o t h e r words, w e assume t h a t c 9 i s T estimable in t h e experiments defined by [

.

The n e c e s s a r y and sufficient condition f o r t h e estimability of c T9 i s

Designs satisfying (12) will b e called r e g u l a r .

I t is obvious t h a t a l l optimal d e s i g n s

# *

f o r (11) coincide with t h e optimal d e s i g n s for t h e s i m p l e r p r o b l e m

inf g T M ( # ) 9

.

c T t 9 = 1

Taking into a c c o u n t t h e condition ( 1 2 ) a n d using t h e s t a n d a r d L a g r a n g i a n t e c h - nique, w e g e t

with 9' =M-'(#)c

.

From t h e l a s t equation, i t immediately follows t h a t r e g u l a r op- timal designs, are t h e s a m e f o r both c r i t e r i a ( 1 0 ) a n d ( l l ) , more d e t a i l s see in F e d o r o v , K h a b a r o v , 1986. If t h e r e i s some p r i o r information o n t h e p a r a m e t e r s dl d e s c r i b e d by a p r i o r d i s t r i b u t i o n function, p,,(d9) , t h e n i t i s r e a s o n a b l e to u s e t h e mean of t h e n o n c e n t r a l i t y p a r a m e t e r as a c r i t e r i o n of optimality

If t h e d i s t r i b u t i o n & ( d 9 ) h a s a d i s p e r s i o n m a t r i x D o , t h e n

In p r a c t i c e , knowledge of Do is p r o b l e m a t i c a n d o n e c a n r e l a x t h i s demand a n d as- sume only t h a t t h e d e t e r m i n a n t of t h e d i s p e r s i o n m a t r i x h a s a v a l u e d g r e a t e r t h a n z e r o . In t h i s c a s e , t h e c r i t e r i o n

c a n b e t h e form of i n t e r e s t . If t h e m a t r i x M ( # ) i s nonsingular, t h e n

Evidently t h e maximization of ( 1 5 ) i s e q u i v a l e n t to t h e maximization of

I

M ( t ) / . T h i s c r i t e r i o n i s o n e of t h e most widely used c r i t e r i a i n t h e estimation problem.

Some p r o p e r t i e s of D-optimal d e s i g n s c o n n e c t e d with model t e s t i n g w e r e d i s c u s s e d by K i e f e r (1958) a n d S t o n e (1958). The a b o v e r e s u l t g i v e s a d d i t i o n a l e x p l a n a t i o n of t h e r e l a t i o n between t h e D - c r i t e r i o n a n d t h e model t e s t i n g problem.

L e t us now c o n s i d e r a v e r y n a t u r a l c r i t e r i o n f o r t h e model t e s t i n g p r o b l e m ,

@ ( t ) =

in f J y z . * ) t ( & )

=

inf s T ~ ( € ) 9 , ( 1 6 ) : 9 d T q ( ~ ) i a l a € . U S / ~ Tq ( X ) jbl

f o r a n y function q d e f i n e d o n U

.

I t i s n o t difficult to c h e c k t h e c h a i n of equalities:

inf 9 " ~ ( t ) 9

=

inf inf 9M

((119

suejdTq ( x ) j % l a C

x EU ( d T q ( + ) j % i

w h e r e , of c o u r s e , a d e s i g n

#

h a s to b e r e g u l a r f o r a n y c

=

q ( z ) , z EU .

In most c a s e s , t h e r e q u i r e m e n t of r e g u l a r i t y c a u s e s t h e nonsingularity of M (#). This h a p p e n s , f o r i n s t a n c e , when U

=

X

.

The c r i t e r i o n

belongs t o t h e family of g - c r i t e r i a (Ermakov, 1983). When U

=

X and q ( z )

=

f ( z ), one can b e t an even s t r o n g e r result because t h e c r i t e r i a

are equivalent in t h e c a s e of continuous designs, a r e s u l t which follows from Kiefer

& Wolfowitz's theorem. This leads immediately t o t h e equivalence of (16) and D-

c r i t e r i a .

The equivalence of some c r i t e r i a can b e proved with t h e help of t h e well- known result on eigenvalues of matrices (com are with Jones, Mitchell, 1978). Let M b e a symmetric matrix and l e t C

=

BB

F

b e a positive-definite matrix. If X l r .

. .

r A , are t h e r o o t s of

/

M -AC

I =

0 then

inf 1 9 ~ ~ 9

=

A, .

3 gTc19

From t h i s relation, t h e equivalence of t h e following two c r i t e r i a immediately oc- c u r s

* ( t ) =

xi1(()

.

^{T ( 0}

⁼

^inf

^j

r t 2 ( z , ~ ) ( ( d ~ )

.

dlcd*l

When C

=

I,, then *(() i s t h e popular E - c r i t e r i o n of design theory.

The r e s u l t s c a n b e summarized in t h e following theorem.

Theorem 4. The following c r i t e r i a a r e e q u i v a l e n t o n t h e s e t of r e g u l a r designs:

(i) cTM-(()c and inf y((,9) ; (c ld)%d

(ii)

i

M-'(()

i

and inf

j

y((.d)gO ( d 9 ) ; ID01

(iii) s u q

'

(z)M -(()q ( z ) and

, €5

^inf ~ ( 4 , d ) ;

::s/ ^s lc.

^)dl

(iv) X1~BTM-'(()B ( and inf A((,+) ,

d r ~ ~ r d a d

w h e r e 6>0 and y((, I?)

=

j ( d T f (z)j2((&) , w i t h t h e i n t e g r a l over t h e r a n g e X

.

The requirement of regularity i s essential f o r (i) and (iii) of t h e theorem. In o t h e r c a s e s f o r optimal designs t h e existence of t h e inverse matrix M-'(() is evi- dent.

The theorem is true both f o r d i s c r e t e and continuous designs. But t h e equivalence of (ii) and (iii) is based on Kiefer & Wolfowitz's theorem which is true only for continuous designs.

6. WUYEIUCAL PBOCEDUBES

Thwrem 2 gives possibility to w n s t r u c t T-optimal designs with t h e help of t h e algorithms developed f o r t h e parameter estimation problem. These algorithms were discussed repeatedly in t h e statistical l i t e r a t u r e (see. f o r instance, Fedorov, 1972; Ermakov, 1983). Therefore only Ute algorithms specially oriented to t h e model testing problem rill b e considered in this section. For t h e s a k e of simplicity they will be formulated f o r deslgn problem (8) and w e start with t h e algorithm, which is a generalized version of t h a t proposed by Atkinson, Fedorov (1975).

This algorithm i s based on t h e results of Thwrem 1 and belongs Lo t h e family of steepest descent algorithms.

To avoid difficulties related to singularities in optimization problems

w e assume t h a t for T-optimal design (17) has unique solution d(€')

.

In practice, t h i s assumption is not very r e s t r i c t i v e because instead of (8) one can apply to t h e regularized version of i t

where

4,

i s any design providing uniqueness of $(to)

.

^Due to concavity of T(t) (compare with Fedorov, Uspensky, 1975):

(i) Let t h e design

4,

was constructed at t h e previous iteration

where (p(z,t, )

= F f

q ( z ,d, ) j

-

^{T(€,) ,}

^X,

is t h e supporting set of

4,

and

(ii)

4,

^+I

=

( 1 7,

I t s

⁺ 7i€(zS ) , where 7,

=

as if sup ( ~ ( 2 ,

€,

2

-

^{inf O(Z,}

^4,

^{) I and 7,}

⁼

^-m"x

1

^{a, ,P, +I/ ( 1 q, +I) 0th-}

rcY

= a

envise. p, i s t h e me-e of point 2, prescribed by

4, .

The sequence

la, I

providing convergency of (i), (ii) can be chosen similarly to parameter estimation case (see, f o r instance, Ermakov, 1983; Denisov, Fedorov, Kha- barov, 1981):

a , - - + O e C a s = - , C a f < - .

If f o r given ~ ( z , d ) and p(d) formula (9) is admissible f o r computing then in- stead of (11) one can use this formula chosing f r o m t h e s t h 4 e s i g n (m -k) support- ing points with l a r g e s t values of q ( z ,

€,

)

.

Together with

z,

they will form a sup- porting s e t for

€, .

This modification of t h e iterative procedure converges to an optimal design containing non more than m +l-k supporting points and is very close to t h e Remez algorithm f o r t h e Tchebysheff best approximation problem (for details, see Demjanov, Malozemov, 1966; Denisov, Fedorov, Khabarov, 1981).

7. SEQUENTIAL DESIGN

Application to (4) and (5) makes i t c l e a r t h a t i t e r a t i v e p r o c e d u r e (i), (ii) c a n b e used in p r a c t i c e for t h e construction of maximin designs o r locally optimal designs f o r given

.

The l a t t e r design can b e useful f o r t h e clarifying of gen- e r a l s t r u c t u r e of T-optimal design. To b e more specific, o n e c a n u s e some sequen- tial design p r o c e d u r e s which were r e p e a t e d l y discussed by d i f f e r e n t a u t h o r s (see, f o r instance, Atkinson, Fedorov, 1975; Atkinson, 1978).

The simplest sequential p r o c e d u r e i s t h e following one:

(i) After N measurements o n e h a s to calculate

Jjn = A r p inf

2:

Flyi - q j ( z i , * j ) l .

d j E n j i =1

(ii) The (N +l)-th measurement h a s t o b e done a t t h e point:

This sequential p r o c e d u r e h a s i t s roots in i t e r a t i v e p r o c e d u r e (i), (ii) from t h e previous section. The similarity will b e more evident if o n e put q ( z , $ )

=

q l ( z

-

qz(z..rP2) , 7~

=

( N ~ + N ) - ~ , where N~ i s a number of measure- ments in a n initial experiment. Naturally t h e deletion of "bad" points permissible in t h e i t e r a t i v e p r o c e d u r e h a s no s e n s e f o r t h e sequential design.

Some numerical examples illuminating t h e efficiency of sequential p r o c e d u r e (i), (ii) were discussed by Box, Hill, 1967; Fedorov, 1972; Atkinson. 1978. The weak convergency:

where qj. ( z , d ) is a "wrong" model. follows from t h e convergency of i t e r a t i v e pro- c e d u r e (i), (ii) if o n e manages t o p r o v e t h a t f o r t h e t r u e model t h e p a r a m e t e r esti- mators are consistent f o r t h e sequence

ItN ^{1 .}

The consistency c a n b e a s s u r e d by application t o regularization (18).

8 . CONCLUSIONS

The r e s u l t s p r e s e n t e d in t h i s p a p e r (based on formal, mathematical tech- niques) confirm t h e validity of t h e following simple, intuitive idea:

"Observing stations should b e located a t s i t e s where t h e d i s c r e p a n c y between competing models i s g r e a t e s t

".

Indeed, in case of two competing models q l ( z ,'rP2) and q 2 ( z ,d 2 ) . Theorems 1 and 3 lead to t h e recommendation t h a t observing stations should b e located a t points w h e r e t h e function

a p p r o a c h e s i t s u p p e r bound f o r t h e (in t h e model testing s e n s e ) worst values of p a r a m e t e r s .rP1 and 'rP2.

The same idea c a n b e t r e a t e d in numerical p r o c e d u r e (i), (ii) of Section 6 and t h e sequential methodology of experimental design.

In t h e f i r s t c a s e , at e v e r y s-th s t e p o n e h a s t o r e l o c a t e a possible point of ob- s e r v a t i o n from an area where t h e discrepancy q l ( x ,9, )

-

^{q2(x ,9, )} i s small, to a n a n area where i t h a s i t s l a r g e s t value.

In t h e sequential design, e v e r y new observation h a s to b e located at a point w h e r e t h e c u r r e n t measure of discrepancy i s l a r g e s t ( s e e (i), (ii), Section 7). I t is evident t h a t , to some e x t e n t , similar sequential p r o c e d u r e s are used r e g u l a r l y in operational p r a c t i c e . H e r e , statistical t h e o r y provides a r e a s o n a b l e (from a sta- t i s t i c a l point of view) c r i t e r i a of optimality, n e c e s s a r y formulae f o r calculations, a n d (this seems a most useful r e s u l t ) global optimality of t h e p r o c e d u r e : sequential designs generated by (i), (ii) converge to a design which i s optimal in t h e s e n s e of (3). (4).

Section 5 confirms t h e common feeling amongst p r a c t i t i o n e r s t h a t t h e prob- lems of model testing and p a r a m e t e r s estimation a r e essentially overlapping. If o n e c a n efficiently estimate t h e most c h a r a c t e r i s t i c p a r a m e t e r s f o r competing models, then model discrimination can b e performed a p p r o p r i a t e l y .

Atkinson, A.C. (1978). P o s t e r i o r probabilities f o r choosing a r e g r e s s i o n model.

Biometrika 65, 39-48.

Atkinson, A.C. & Fedorov, V.V. (1975). The design of experiments f o r discriminat- ing between t w o r i v a l models. Biometrika 62, 57-70.

Box, G.E.P & Hill, W. J. (1967). Discrimination among mechanistic models. Tech- nometrics 8 , 57-71.

Denisov, V., Fedorov, V. & Khabarov, V. (1981). Tchebysheff's approximation in t h e problem of asymptotical locally-optimal designs construction f o r discrim- inating experiments. In Linear a n d Nonlinear P a r a m e t e r i z a t i o n i n E z p e r i - mental Design Problems, Problems i n Cybernetics, Ed. V . Fedorov and V.

Nalimov, pp. 3-10 (in Russian). Moscow.

Ermakov, S.M., Ed. (1983). Mathematical t h e o t y of experimental d e s i g n . Nauka, Moscow (in Russian).

Fedorov, V.V. (1972). Theory of m t i m a l Ezperiments. New York: Academic P r e s s .

Fedorov, V.V. & Uspensky, A.B. (1975). Numerical a s p e c t s of least s q u a r e s methods. Moscow S t a t e University, Moscow.

Fedorov, V.V. (1980). Convex design t h e o r y . Math. Oper. S t a t i s t . , Ser. S t a t i s t . fi, 403-13.

Fedorov, V.V. (1981a). Design of model testing experiments. In S y m p o s i a Mathematica. pp. 171-80. Bologna: Monograph.

Fox, D.G. (1981). Judging a i r quality model performance. B u l l e t i n American Meteorological Society 62, 599-608.

Hayes S.R. and Moore, G.E. (1986). Air quality model performance: a comparative analysis of 15 model evaluation studies. Atmospheric Environment, ^20, 1897-1911.

H u b e r , P. J . (1981). Robust s t a t i s t i c s . Wiley & Sons, N.Y.

Jones, E.R. & Mitchel, T.J. (1978). Design c r i t e r i a f o r detecting model inadequacy.

Biometrika 65, 541-51.

Karlin, S. & Studden, W.J. (1966). Tchebyshefl systems: with appLication i n a n d y s i s and statistics. Wiley & Sons, N.Y.

Kiefer, J. (1958). On t h e nonrandomized optimality and randomized nonoptimality of symmetrical designs. Ann. Math. Statist. 29, 675-99.

Munn, R.E. (1981). The design of a i r quality monitoring networks, MacMillan Pub- l i s h e r s Ltd., London, p.109.

Silvey, S.D. (1980). @timarl Design. London: Chapman and Hall.

Stone, M. (1958). Application of a measure of information t o t h e design and com- parison of r e g r e s s i o n experiments. Ann. Math. Statist. 29, 55-70.

Willmott, C. J . (1982). Some comments on t h e evaluation of t h e model performance.

BuLLetin American MeteoroLogicd Society, 83, 1309-1313.