Use of Kalman Filtering Techniques when the Parameters of a Regression Relationship are Changing over Time According to a Multivariate ARIMA Process

USE O F KALMAN F I L T E R I N G TECHNIQUES WHEN THE PARAMETERS O F A R E G R E S S I O N R E L A T I O N S H I P ARE CHANGING OVER T I M E ACCORDING

TO A MULTIVARIATE ARIMA P R O C E S S

J o h a n n e s L e d o l t e r

J u n e 1 9 7 6

Research Memoranda are interim reports o n research being con- ducted by the International I n s t i t ~ t e for Applied Systems Analysis, and as such receive only limited scientific review. Views or opin- ions contained herein do not necessarily rcpresent those o f the lnstitute or o f the National Member Organizations supporting the Institute.

P r e f a c e

I n many a r e a s o f a p p l i e d r e s e a r c h a t IIASA r e g r e s s i o n models a r e e n t e r t a i n e d t o e x p l a i n t h e r e l a t i o n s h i p among v a r i a b l e s . Assuming t h a t t h e p a r a m e t e r s a r e n o t c h a n g i n g o v e r t i m e , l e a s t s q u a r e s methods p r o v i d e minimum s q u a r e e r r o r (MMSE) e s t i m a t e s . I n some c a s e s , however, t h e

assumption o f c o n s t a n t p a r a m e t e r s i s r e s t r i c t i v e , and ways o f i n c o r p o r a t i n g p a r a m e t e r changes have t o be found.

T h i s p a p e r was o r i g i n a l l y p r e p a r e d u n d e r t h e t i t l e " M o d e l l i n g f o r Management" f o r p r e s e n t a t i o n a t a N a t e r R e s e a r c h C e n t r e

(U.K. ) Conference on " R i v e r P o l l u t i o n C o n t r o l " , Oxford, 9 - 1 1 A s r i l , 1979.

Use of Kalman filtering techniques when the parameters of a regression relationship are changing over time according to a multivariate ARIMA process.

Abstract

It is shown how Kalman filtering methodology can be applied to the estimation of the parameters in a regression model, when the parameters are subject to change over time.

A multivariate ARIMA model for the parameters of the re- gression relationship is entertained and it is shown how this model can be brought into the state variable form.

Furthermore it is shown how this procedure specializes to various cases already discussed in the literature.

1. State variable representation of dynamic and st0chast.i~

-

systems

-

Kalman filtering

A dynamic system with stochastic disturbances may be modelled in a state variable form

where

xt

is a vector state variable which should be considered as an abstract quantity and which does not necessarily have a physical interpretation such as the input vector

ut

^{and the}

output vector

yt. yt

^and

yt

are uncrosscorrelated Normal white

n o i s e s e q u e n c e s w i t h

The p a r a m e t e r m a t r i c e s A , G and H may b e e i t h e r c o n s t a n t o r t i m e v a r y i n g .

Given t h e dynamic s t o c h a s t i c model w i t h known dynamic and s t o c h a s t i c p a r a m e t e r s , Kalman [ 3 , 4 , 5 ] o b t a i n s a n e s t i m a t e f o r t h e s t a t e v e c t o r

xt

g i v e n t h e o b s e r v a t i o n s on t h e i n p u t and o u t p u t v a r i a b l e s u p t o t i m e t . H e shows t h a t t h e c o n d i t i o n a l d i s t r i b u t i o n o f x t g i v e n o b s e r v a t i o n s u p t o t i m e t i s a Normal

-.

w l t h mean 3

-tl t a n d v a r i a n c e Pt

t

^where

a n d

where t h e Kalman g a i n i s g i v e n by

T h i s s e t o f r e c u r s i v e e q u a t i o n s , t o g e t h e r w i t h s p e c i f i e d i n i t i a l c o n d i t i o n s , p r o v i d e t h e e s t i m a t e s and t h e i r u p d a t i n g e q u a t i o n s f o r t h e s t a t e v a r i a b l e s a n d t h e i r c o v a r i a n c e m a t r i x .

An e x c e l l e n t d e s c r i p t i o n of t h e s t a t e v a r i a b l e a p p r o a c h

to dynamic and stochastic systems is given by MacGregor [ 6 ] . 2. State variable representation of a regression model when

its parameters change according to a multivariate ARIMA

model.

Kalman's approach can be used to estimate the parameters in regression models where it is assumed that the parameters follow a general multivariate ARIMA process:

where

Yt is the dependent variable

r is a (k x 1 ) vector of predetermined variables

-

^t

B

is a ( k x l ) vector of parameters -t

where the Oi (I _- < i ( _- p+d) and the 0. (1

5

^j

5

q) are known (k k) I

matrices. B is the backshift operator

B _- ~

=

B et-m. ~

^{at is a}

2 2

white noise sequence with Eat = 0 and Eat = oa. c~~ is a multi- variate white noise sequence with Ea = 0 and Ea a' = La.

-t -t-t It is

assumed that the zeros of det{@(B)} lie on or outside the unit circle. The zeros of det{O(B)} are assumed to lie outside the unit circle and det{@(B)} and det{O(B)} do not have common roots.

Furthermore at and at are uncorrelated, i.e. Ea

-

^t-t a = 0 .

-

Multi- variate ARIMA processes are generalizations of the univariate

AXIMA p r o c e s s e s d i s c u s s e d i n g r e a t d e t a i l by Box and J e n k i n s [ I ] . E x t e n s i v e d i s c u s s i o n of t h e m u l t i v a r i a t e e x t e n s i o n , f o r e x a n p l e , i s g i v e n i n Hannan [21.

I n o r d e r t o a p p l y t h e Kalman f i l t e r i n g t e c h n i q u e (equa- t i o n s ( 1 . 2 )

-

( 1 . 6 ) ) w e have t o w r i t e e q u a t i o n s ( 2 . 1 ) i.n form of s t a t e v a r i a b l e s . The f o l l o w i n g theorem g i v e s an e q u i v a l e n t s t a t e v a r i a b l e form f o r s y s t e m ( 2 . 1 ) , t h u s i d e n t i f y i n g t h e n a t r i c e s H , A , G I R1 and R 2 .

Theorem: The model g i v e n i n ( 2 . 1 ) h a s t h e e q u i v a l e n t s t a t e v a r i a b l e r e p r e s e n t a t i o n g i v e n below.

For p + d > q :

where I* i s t h e [ k (p+d-1) x k ( p + d - 1 ) ] i d e n t i t y m a t r i x and 0*

i s a [ k x ( p + d - l ) ] m a t r i x of z e r o s . 0 i s a [ k x kl m a t r i x o f z e r o s . 0'

-

_{i s} a [ I x k ] v e c t o r o f z e r o s a n d I i s t h e [ k x kl

i d e n t i t y m a t r i x .

F o r p+d<q: -

where I** i s t h e [ k q x k q ] i d e n t i t y m a t r i x and

O*.

i s a [ k x k q ] m a t r i x o f z e r o s .

The p r o o f o f t h i s t h e o r e m f o l l o w s by s i m p l e s u b s t i t u t i o n showing t h a t

3. Several special cases discussed in the literature:

i.) Young [ 8 1 considers the case where the parameters follow a first order autoregressive process

In this case, the updating equations reduce to

ii.) For the special case O = I and Z a = 0 (i.e. B = 6 -t+l -t constant parameters) the updating reiations in (3.2) and (3.3)

-

simplify to

One recognizes the recursive updating formulae in (3.4) and (3.5) as the recursive updating algorithm for the least squares estimate

Et

and its covariance matrix Pt given by Plackett [ 7 ] .

i i i . ) For t h e c a s e k ⁼ 1 , 4 = 1, J3a2 = 0 2 and Ea 2 ⁼ o ²

~1 a

t h e r e c u r s i v e a l g o r i t h m i s g i v e n by

We n o t i c e t h a t i n a l l t h e s e u p d a t i n g f o r m u l a e t h e e s t i m a t e of t h e p a r a m e t e r a t t i m e t i s a l i n e a r c o m b i n a t i o n of t h e p a r a m e t e r e s t i m a t e a t t i m e t - 1 and t h e one s t e p ahead f o r e c a s t e r r o r a t t i m e t . I n ( 3 . 4 ) t h e p a r a m e t e r i s u p d a t e d by g i v i n g e q u a l w e i g h t s t o a l l t h e o b s e r v a t i o n s . I n ( 3 . 6 ) t h e i n t r o d u c t i o n of a 2 i s s i m i l a r i n e f f e c t t o an e x p o n e n t i a l d a t a w e i g h t i n g f u n c -

C1

t i o n (Young [ 8 ] 1 .

i v . ) Box and J e n k i n s [ I ] c o n s i d e r a random walk model w i t h added n o i s e ( k = 1 , 4 = 1 , rt = 1 f o r a l l t )

They show t h a t t h i s model i s e q u i v a l e n t t o t h e i n t e g r a t e d f i r s t o r d e r moving a v e r a g e model

with

Bt+l is the one step ahead forecast Qt(l) and is given by an exponential weighted sum of previous observations.

and the forecasts are updated by

R e f e r e n c e s

BOX, G.E.P. a n d J e n k i n s , G . N . , T i m e S e r i e s A n a l y s i s , F o r e c a s t i n g a n d C o n t r o l , Holden-Day; S a n F r a n c i s c o ,

1 9 7 0 .

Hannan, E . J . , M u l t i p l e Time S e r i e s , J o h n W i l e y , N e w Y o r k , 1 9 7 0 .

Kalman, R . E . , "A N e w A p p r o a c h t o L i n e a r F i l t e r i n g a n d

P r e d i c t i o n P r o b l e m s , " J o u r . B a s i c E n g . , - 8 2 , 3 5 , 1 9 6 0 . Kalman, R . E . , " C o n t r i b u t i o n s t o t h e T h e o r y o f O p t i m a l

C o n t r o l , " B o l . d e S o c . M a t . Mexicans, 1 0 2 , 1 9 6 0 . Kalman, R . E . , "The X a t h e m a t i c a l D e s c r i p t i o n o f L i n e a r

D y n a m i c a l S y s t e m s , " SIAM J o u r . o n C o n t r o l , - 1 , 1 5 2 , 1 9 6 3 . MacGregor, J . F . , " T o p i c s i n C o n t r o l : 1 . S t a t e V a r i a b l e

A p p r o a c h t o Time S e r i e s R e p r e s e n t a t i o n a n d F o r e c a s t i n g , "

T e c h n i c a l R e p o r t N o . 3 0 6 , U n i v e r s i t y o f W i s c o n s i n , M a d i s o n , 1 9 7 2 .

P l a c k e t t , R . L . , "Some Theorems i n L e a s t S q u a r e s , ' ' B i o m e t r i k a ,

-

3 7 , 1 4 9 , 1 9 5 0 .

Young, P . C . , " A p p l y i n g P a r a m e t e r E s t i m a t i o n t o Dynamic S y s t e m s - - P a r t I a n d 11," C o n t r o l ~ n ~ i n e e r i n g , - 1 1 9 - 1 2 5 , 118-124, 1 9 6 9 .