On a Definition of the Description Complexity of Finite Systems

NOT FOR QUOTATION WITHOUT PERMISSION

OF THE AUTHOR

ON A DEF'INITJON OF THE DESCRIPTION

C O W ~ O F ~ ~

Ashot Nersisian

January

1984 WP-84-4

Working

Papers

a r e interim reports on work of the International Institute for Applied Systems Analysis and have received only limited review. Views or opinions expressed herein do not necessarily represent those of the Institute or of its National Member Organizations.

INTERNATIONAL INSI'ITUTE FOR APPLIED SYSTEMS ANALYSIS 2361 Laxenburg, Austria

PREFACE

Most of the systems studied a t

IIASA

a r e characterized by large numbers of components and by complex interactions. To minimize the problems caused by these factors it is often useful to find t h e most economical (minimal) representation of the system (or of t h e model used t o describe it).

In this paper t h e author, a participant in the 1983

IlASA

Young Scientists' Summer Program, investigates the problem of finding minimal descriptions of finite deterministic systems with a given accu- racy. An algorithm for obtaining an asymptotically minimal description of discrete systems with constraints on t h e modeling accuracy is pro- pose d.

ANDRZEJ WIERZBlCKl Qurirmcm

System and Decision Sciences

ON A DEFINITION OF THE DESCRIPTION COYlpLMITY OF F'INlTE sYsrExs

Ashot Nersisian

1.

INTRODUCTION

Modern complex systems a r e characterized by large numbers of components and by complicated interactions. When studying such sys- tems, therefore, i t is useful t o consider ways of finding t h e most econom- ical representation of t h e system. The size of t h e minimal representa- tion of a system is usually called its description complexity.

In practice models of complex systems (approximations of a given accuracy) a r e constructed, and these a r e then analyzed r a t h e r than t h e systems themselves. In t h i s case t h e description complexity of a system should be i n t e r p r e t e d a s t h e complexity of its feasible approximation.

If methods for t h e construction of minimal descriptions a r e t o be used in practice, they m u s t be computationally efficient (i.e., they should have polynomial t i m e complexity ).

*

The time complexity of an algorithm i s the number of computational steps necessary on m y formal machine which carries out letter-by-letter operations.

For probabilistic systems t h e minimal description for a given fidel- ity criterion can be obtained by means of Shannon's rate-distortion theory (Shannon, 1959), which is described in more detail by Berger (1971). However, Shannon's results cannot be applied directly to practi- cal problems because they involve a time complexity of

z ~ ,

where n is the size of the system.

This paper investigates t h e problem of finding minimal descriptions of finite deterministic systems with a given accuracy. Deterministic methods of information compression, developed by Lupanov (1965), Nechiporuk (1985) and Sholomov (1967) for t h e synthesis of logical net- works, a r e used. A particular case of this problem, which deals with t h e description complexity of partially-specified systems, h a s already been analyzed by Nersisian (1981).

An algorithm for obtaining an asymptotically minimal description of discrete systems with constraints on t h e modeling accuracy is proposed.

The time complexity of this algorithm on any formal machine which car- ries out letter-by-letter operations (see Aho, Hopcroft and Ullman, 1976) does not exceed nl*, where n characterizes t h e size of the system a n d 7

>

0 is an arbitrary constant.

2.

STATEMENT OF THE PROBLEX AND FORMULATION OF THE

RESULTS

Let

a system

S

be given by

s =

< V . P > , (1) where V

=

¹¹

....,

^vj is a s e t of objects and P

=

) P I , . ..,P, j a collection of relations given on s e t V.

Let a model

M

of the system be given by

M =

< V , Q > ,

where Q

=

Q1.

...,Qp

j is a collection of relations "simpler" than P. It can easily be shown (see, for example, Nersisian, 1981), t h a t a system s u c h as (1) can be represented by a sequence of symbols 2"

=

( z l , ..., z,) in some finite alphabet

A =

l a l ,

...,

a,{. Similarly, model

M

of system S c a n be represented by a sequence y"

=

( Y ~ , ...,y,) in some finite alphabet

B =

ib l,...,bt j. We shall call sequence an approximation (model) of

2".

The accuracy of modeling is determined by several criteria f l....,f N.

Each criterion is characterized by a distortion measure p(d)(a,.bj)

=

a p ) for each l e t t e r , where 0

r

^{C I ~ ) 4 m} is t h e "penalty*' incurred by replacing symbol

q

by symbol b j in the model. The overall distortion on going from sequence

2"

to sequence y" is t h e s u m of t h e distortions for t h e indi- vidual letters:

where w,, is t h e number of positions u for which zu

= q ,

yu

=

b,. We shall consider the case in which the permitted level of relative distortion

dd)5

^{0, d}

=

1.

.... N .

for each criterion f d is given, and approximation y"

=

y"(z") satisfies t h e constraints:

p ( d ) ( 2 " , g ) r ~ ( d ) n . d = 1 .

...,

N .

This type of approximation will be called 2-accurate, where 3

=

(~('1, ...,

dN)).

I t can easily be seen t h a t to satisfy these conditions it is necessary t h a t for each i t h e r e exists a j

=

j ( i ) such t h a t

a$) =

⁰ for all d (otherwise for some d the distortion introduced by modeling will be

greater t h a n , d d ) n ) .

I t

should be noted that if

=

^a t h e n wij

=

0 (i.e..

no

q

is replaced by b , ) .

Let Mn(k l . . . . , k s ) denote the class of all sequences of length n in alphabet

A

such t h a t each sequence contains ki symbols

q ,

i

=

^1. ..., s .

k l + .

. .

^+k,

=

n . Let parameters k ,..., k, be functions of n , i.e.,

k

=

k l ( n ) . . . . , k s

=

k , ( n ) . We shall assume t h a t t h e r e exists a certain encoding technique K

= &

which associates each sequence

2"

E M n ( k l ,

....

k , ) with a binary sequence (code) K(z"). and t h a t t h e r e exists a decoding technique D

=

Dn such t h a t

D ( K ( ~ " ) )

is a word of length n in alphabet B. The word D ( K ( ~ " ) ) will be considered to be t h e approxi- mation of

2".

Let l(2") denote t h e length of codeword K(2"). Let us assume t h a t

l n ( k l , . . . . k s )

=

max l ( f ) .

Z€M,,(k ,,..., k,)

This quantity will be called the description c o m p l e x i t y of class Mn(k

....,

k s ) .

The accuracy of t h e model will be characterized by the following quantities:

p(d)(z")

=

p ( d ) ( ~ ,

D ( K ( z ) ) ) ,

d

=

1

...., N .

The above encoding--decoding method will be called Z-accurate, where

=

( & ( I )

,.... &(w),

^{&(I) 2 0}

^....,

^{F(W 2 0,} if for each

i

E 4 ( k l ...., k s )

We now define ' S e n t r o p y for the given P

=

_{and P}

= ( p

^{l ,} . , . , p s ) ,

All logarithms are assumed to be to base two, i.e., logz z.

HE(

P )

=

_IIpvII min

C ^pi,

^log

Pij

i , j

piCpuj

u '

where t h e minimum is taken over all (s x t ) matrices

I Ipij I I

^{for which}

Theorem

¹

( 1 ) Let the condition

log log n be satisfied.

Then for an arbitrary function

a ( n )

-, m there exist Z-accurate encoding and decoding techniques which ensure t h a t t h e descrip- tion complexity satisfies t h e following relationship:

-

n H i [ > , . . . , k ]

+

n a ( n ) log log n

l n ( k l,...,ks) - log n

(2) If function

a ( n )

is computable with time complexity n , then t h e encoding operation

II;,

and the decoding operation Dn have time complexities not g r e a t e r than nl*, where y

>

⁰ is an arbitrary con- stant.

( 3 ) For each :-accurate encoding and decoding technique

*

l,(k

,,...,

k , ) a n H E

1:'

- ,..,,

- :.I +

^{cllog n}

*

The letter c , possibly with a subscript, denotes a constant here and elsewhere.

3. PROOF OF THEOREM

1

1. Let

W = 1

(wij

I I

be an

( s x t )

matrix satisfying t h e condition

The function

I ( W )

= n l o g n

- C

^{kilog ki}

- C

m j l o g m j

- C

^{w . - =I} l o g q j

i i J

( 5 ) j

is then associated with matrix W, where

I t is easy t o show t h a t

where

kmma

1

Given both an

( s x t

) matrix W

= 1 1

^wij (

1

^{and an}

( s x t )

m a t r i x

W' =

(

I

wIij

1 1

such t h a t

t h e n

*

The notation pn

*

^{$, or p, = 0($,)} means that lim" < -, while the notation pn = ^o

(qn)

n +-$n

means t h a t lim

-

Pn = 0.

n+-+n

The inequality ( 0 ) Follows immediately from expression (5), using condition (7) a n d t h e f a c t t h a t s and t a r e limited.

2. ~ e t *

Consider all possible ( s x t ) m a t r i c e s W

=

(

1

wij

1 I

which satisfy t h e condi- tions

For e a c h j # j(i) l e t

w . .

=

l . . A ,

21 51

where t h e lij a r e integers; f o r each j

=

j ( i ) l e t

Now, from t h e s e , l e t us find t h e m a t r i x Wo with t h e smallest value of J ( W ) .

Lemma 2

(1) The following condition holds for matrix Wo

(2) The t i m e complexity of t h e algorithm for finding m a t r i x

W o

is lim- i t e d by t h e value n(1og n ) 2 s t + 4 .

[Z] uZ[) denotes the largest (smallest) integer which does not exceed Z.

Proof

Let the minimum in expression (2) be achieved on collection (p*j

.

Define

and introduce matrix

w = 1 1

w;

I 1.

From (5) it follows that

Let

Then if wij is found from (9)--(12), t h e inequalities

are satisfied, and from Lemma 1

II(W)

- I ( w * ) (

5

A

l o g n

.

Combining ( 7 ) , (9), ( 1 4 ) and ( 1 5 ) leads to t h e expression

lr(w) - r ( w v ) l

^{S A I O ~}

.

^~

Since t h e condition

wij

s

wij

=

pijn

holds for all j # j ( i ) a n d condition ( 1 0 ) is valid for matrix

w*,

we have

and therefore matrix W also satisfies condition ( 1 0 ) . Inequality ( 1 3 ) fol- lows from ( 1 6 ) and from t h e relation

Let u s now evaluate t h e complexity of finding matrix Wo. Instead of function I( W) we can consider t h e monotonically related function

This expression contains numbers of t h e form s S , s I n , whose dimen- sions ( n u m b e r s of binary digits) do n o t exceed n log n . To compute sS requires only log s I log n multiplications (see, for example, Valski, 1959). According to Schonhage a n d Strassen (1971). n o t m o r e than n l o g 3 n e l e m e n t a r y operations a r e needed to multiply t o g e t h e r two ( n log n)-digit numbers and t h e general n u m b e r of operations required to compute s S , s S n , is of t h e order of n l o g 4 n , or less. After finding all t h e numbers of t h e form sS involved in (17) a finite n u m b e r of multi- plication a n d division operations m u s t be performed. This requires of the order of n log3 n operations or less. To verify condition (10) requires no more t h a n log n operations. Since t h e r e a r e a t most ((log n ) 2

+

l)St versions of ( s x t ) m a t r i x W, t h e overall t i m e complexity of finding m a t r i x

Wo is of t h e o r d e r of n(1og n)2St+4 o r less.

3. Let u s introduce quantities

and c o n s t r u c t t h e ( s x t ) matrix QO

= I

I q O ( j / i ) (

1 .

Assume t h a t an integer v ^S n is given. We shall consider an arbitrary collection x I , , . . , ~ such t h a t

xXi ⁼

^{v .} Let M,(xl, ...,&) be t h e class of all sequences contain-

i

ing

xi

^symbols %. We shall use t h e following notation:

Next let us form a n ( s

xl

) m a t r i x

Cl = 1 I

wij ( (

.

Since

we d e d u c e from Lemma 1 t h a t

Xl

x,

I ( R ) - v I (

-,..

.

,

-

^;

^{QO) r}

^{log v ,}

V V

where

It is also evident t h a t

E ^{aif)ug r} C a $ l ) x i r l O ( j / i ) .

d = 1 .

.... N . ( 2 1 )

i ,j i ,I

We shall say t h a t a s e t N of sequences in alphabet B ?-accurately approximates a s e t

M

of s e q u e n c e s in alphabet A if for e a c h s e q u e n c e from

M

t h e r e is a corresponding ;-accurate approximation in t h e N. Let us denote by

T E ( x I , . , . , & )

t h e minimum cardinality of a s e t which

?-

a c c u r a t e l y approximates

M , ( x l , . . . , x , ) .

lemma

3

( 1 )

The following relation holds:

Xl x,

log

T Z ( x l . . . . , & )

⁵

v I ( - ,

...,

-

;

QO) +

^O(1og

V )

,

V V

where

8 =

(6(1)....,6(w) and

(2) A set

N

which ?-accurately approximates

M,(xI,.. .,xs)

and satisfies e s t i m a t e (22) can be found with t i m e complexity not g r e a t e r t h a n c z

+

^{c g} l o g 2 n .

Proof

Estimate (22) can be obtained using t h e gradient procedure sug- gested by Sholomov (1967); this is a modification of t h e procedure pro- posed by Nechiporuk (1965). A table i s formed with 5 s v columns

!xi !

a

corresponding t o t h e sequences

7

^E

M , & ~ ,

...,&) and with -<

npj!

Y' ^- tV rows

j

correspondmg t o the sequences

5

containing pj symbols b j . At t h e intersection of row

5

and column we p u t a ''1'' if we have

H

oij

(i

=

¹ ...., s , j

=

¹ ... t) positions with symbol a, in sequence

<

^and

symbol b j i n sequence

5 ;

otherwise we i n s e r t a "0". It is evident t h a t if a

"1" is found a t t h e intersection of row

5

and column

t

^then

^jj

^{is a}

^8-

accurate approximation of

7.

The gradient procedure is t h e n used. A t each s t e p of this procedure it is necessary t o find the row with t h e max- i m u m n u m b e r of ones i n t h e c u r r e n t table, a n d delete this row a n d ' t h e columns containing t h e ones. The procedure t e r m i n a t e s when all t h e columns have been deleted, and t h e desired s e t N is formed by t h e sequences corresponding to t h e deleted rows. Calculations similar t o those described by Sholomov (1967) give the e s t i m a t e

which, taken together with (20), yields ( 2 2 ) .

The upper bound of t h e size of t h e table is

( ~ , t ) ~ .

The t i m e complex- ity of constructing t h e table and performing t h e gradient procedure can- not exceed t h e polynomial of t h e table size, a n d is limited by t h e value of

rxiwij

1

c;. The complexity of finding p a r a m e t e r s

wij = I , ]

^{and gj}

^{= C}

^0ij

j does not exceed t h e value of c 4 log2 n . The resulting total complexity is

4. Let t h e r e be a sequence z" E

M,

( k I , . . . , k , ) i n alphabet

A .

Assume t h a t a n a t u r a l p a r a m e t e r v i s given and t h a t t h e sequence z" i s broken i n t o pieces of length v:

N H

Z = t . - . M = ] q .

V

Let

Tu

belong t o class

M , ( ~ ? )

....,$)). We shall denote by

qU

a n y

8-

a c c u r a t e approximation of

tu,

where

8

is d e t e r m i n e d using (19) and (23).

Lemma

4

The s e q u e n c e

?

^{=;j ,,...,,,} N

is a n ?-accurate approximation of sequence 2 .

Proof

The overall distortion between t h e sequences z" and y" is equal t o t h e s u m of t h e distortions between t h e pieces and their &accurate

approximations

f ,

:

Substituting in ( 2 5 ) t h e value of GI$') from ( 1 9 ) we obtain

M

From t h e obvious equality

C xiu) ⁼

^ki and relations ( 1 0 ) and ( l B ) , we u =1

find t h a t

which proves the Lemma.

5 . We shall now describe the e n c o d n g procedure.

The codeword K(z") for a sequence z" E M,(k l . . . . , k , ) consists of t h r e e parts:

K(2) = A

C 2

.

These a r e known as t h e reference, main and auxiliary parts, respectively.

We shall consider first the main part C of t h e codeword

K(2).

Sequence

i

E & ( k l , . . . , k , ) is assumed to have been broken down into pieces of length n . We group the pieces with t h e same parameters

x l , ...,&,

x l + ^,

.

+&

= ^v

into separate classes labelled

M I , . . . , M R .

For each class Mi the 8-accurate approximations (where

8

is determined from ( 2 3 ) ) a r e found with, th e help of the gradient procedure described earlier (see t h e proof of Lemma 3). We shall denote t h e number of approximations by ( x l , . . . , ) . Now arrange all these 8-approximations

f

for sequences

from

MV(x

in a certain (e.g., lexicographical) order and number t h e m using binary sequences %(.fj) of length

T,

= ]

log

Tv [ .

⁽²⁶⁾

Let

P = ( i l . ...,4,)

denote t h e binary representation of number i a n d

r*

^be

t h e corresponding binary sequence, where

2" =

(ili

, . . . ~ $

⁰¹⁾

.

I t is obvious f h a t n u m b e r s i and j can be found uniquely from sequence

5.7.

The code z(&) for t h e piece

$

^E

^Mu = M ( ' ) ( ~ ~ , . . . . X ; )

will t h e n be of the form

I * * - N

a(?,) =

v

r ( q j ) .

^{(2 7 )}

where

Cj

^{denotes a} %accurate approximation of

&. ^A

sequence of codes of adjacent pieces

ti

constitutes t h e main part of t h e codeword

C

= %(TI) . . . ^.

The auxiliary p a r t of t h e codeword is a list of 8-approximations. Let the 8-approximations for t h e class

Mu = MV(xl,

...,&) be

N

i ) V 1 8 . . . . ~ ~ ~ *

where

Assume t h a t

N

((4) = 4, ..

. . i v p ,

.

where i j is obtained from

5

by replacing each symbol b j E B by a

~5

^v pv

binary representation of length ]logt [.

The auxiliary p a r t

Z

will be of t h e form Z

=

N [(MI)

.

. . ,

where

R

is t h e n u m b e r of classes M , ( X ~ , . . . , ~ ) . We shall denote t h e length of T(Afi) by

Ji

and t h e length of t h e auxiliary part

E

by 1;;.

The reference p a r t

A

contains t h e numerical parameters required to decode t h e main a n d auxiliary parts. I t is of t h e form

N. l * * N C W C N C l

~ = k 1 n. . . k , L~ L: r l S

- i i l ; . . . J i .

Let t h e length of t h e reference p a r t be

Lk

It is evident t h a t t h e code- word K ( 3 ) can be decoded into x-accurate approximations

5,

of pieces

tu,

which since t h e y a r e adjacent (according t o Lemma 4) give an

F

a c c u r a t e approximation y" of t h e sequence z".

6. The length L(K(9)) of codeword

K(2)

is given by:

L(K(E))

=

L*

+ L~ +

L:

.

Let u s e s t i m a t e t h e length Lz of t h e main p a r t C of t h e codeword. This is

where 1 (%&)) is t h e length of the piece of code

iiRi).

From (26) and (27) i t follows t h a t

(a(?,)) 5 2

]

log

R [ + ]

log Q ( X ~ , . . . . ~ )

[ +

2

.

(29) Since t h e n u m b e r R of classes cannot exceed (v+l)', we may write (tak- ing into account Lemma 3)

One m o r e Lemma m u s t be proved before we can estimate t h e value of Lz.

For

?

^E

M,kI

^,...,

xs)

we introduce t h e notation

Lemma 5

Proof

I t is sufficient t,o consider t h e case g = 2 ( a s will be seen later). Let

ti

^E M , , ( ~ ~ ( ~ ) ,,.., &)), i=1. ...,

M .

Using t h e upward convexity property of t h e average mutual information function, which holds when t h e collec- tion ^{Q O} of transitional probabilities is fixed (see Gallager, 1968), we can write

Multiplying both parts of t h e inequality by vl+v2 and making use of (31) and

1 = TIT2

^-E Evl+vz(X1(1)+X~2) ,...,X(l)+d2)), we obtain inequality (32).

which proves t h e Lemma. Substituting (29) into (28), using Lemmas 2 and 5 and t h e relation

I(Yd =

ⁿ , . . . , ; QO], we obtain

, . . . , ;

QO] +

O ( n log v )

s

n log v n H [ , . . . . ~ ] + O [ Z ] + log n 01

1.

Let us now estimate t h e length of t h e auxiliary part of t h e codeword.

The binary length of one approximation

6,

is n o t more than v ] log t [ . From t h e fact t h a t t h e number of 8-approximations for one class does not exceed s v and t h e number of classes is not more than ( v + l ) ' , we arrive a t t h e e s t i m a t e

Lz

r O ( ( V + I ) ~ v s V ) r C;

.

The value of

LA

is t h e sum of t h e lengths of t h e

5'

form representa- tions. Since t h e length of each parameter does n o t exceed n , it is not difficult t o s e e t h a t

LA

S R log n s ( v + l ) ' log n I v C 6 log n

.

Finally we have

9. Let a

=

a(n)-.m be an arbitrary function which satisfies t h e con- dition

an log n log log n

-.-.

and assume t h a t

I t can be shown directly t h a t substitution of t h i s value i n t o (33) yields the e s t i m a t e (3).

10. Computing experience shows t h a t t h e time complexity of encod-

ing and decoding techniques satisfies t h e e s t i m a t e given in point (2) of Theorem 1.

11. Using t h e technique described in Sholomov (1967), t h e following inequality c a n be obtained:

log TE(k ,..., k,)

I

2

,'

n H E

["

- ,..., -

+

c log n ,

from which, by m e a n s of "power" considerations, we arrive a t e s t i m a t e (4).

4. CONCLUDING

REMARKS

1. The encoding technique proposed in t h e present paper m a y be used t o obtain a proof of Shannon's Theorem of encoding of discrete s o u r c e s with a fidelity criterion without resorting t o t h e r a n d o m encoding technique.

2. The proposed encoding technique and codeword construction c a n be used t o compress large a r r a y s of information if a certain distortion of t h e initial a r r a y is allowed in decoding. A data-compression algo- r i t h m for the c a s e A

=

10,1,+{. 8

=

10,1,j, aij E iO,mj was con- s t r u c t e d a t

llASA

by t h e a u t h o r . Here

*

represents an unspecified symbol which can be replaced arbitrarily by 0 or 1. This algorithm h a s been implemented on t h e

VAX

c o m p u t e r a t

IIASA

by Z. Fortuna.

A C K N O W L E D G ~

The author would like to express profound gratitude to Zenon For- t u n a and Alexander Sarkisov for t h e i r useful suggestions, and t o thank Helen Gasking for editorial support, a n d Nora Avedisian for preparing this paper on IIASA's computerized text-processing system.

REFERENCES

Aho, k , Y. Hopcroft, and U. Ullman (1976). % Design a n d A n a l y s i s of C o m p u t e r Algorithms. Addison Wesley, Massachusetts.

Berger,

T.

(1971). Rate D i s t o r t i o n l h e o r y : A Mathematical Basis f o r Data C o m p r e s s i o n . Frentice-Hall, Englewood Cliffs, New Jersey.

Gallager, R.G. (1968). h f o m a t i o n l h e o r y and Reliable C o m m u n i c a t i o n . John Wiley and Sons, New York a n d London.

Lupanov, O.B. (1965). On one approach t o t h e synthesis of control sys- tems: t h e principle of local e n c o d n g . R o b l e m i K i b e r n e t i k i , Vol. 14, Moscow, Nauka. (In Russian).

Nechiporuk, E.I. (1965). On t h e complexity of valve schemes t h a t realize Boolean m a t r i c e s with undefined elements. b k l a d i A k a d e m i i Nauk

USSR Vol. 163. (In Russian).

Nersisian, k L . (1981). Asymptotically optimal represen tation of partially

specified systems. In Metodi I s s l e d o v a n i a S o j n i k h %tern, All-Union Institute of Systems Studies (VNIISI), Moscow. (In Russian).

Schenhage, A. and V. Strassen (1971). Schnelle Multiplikation grosser Zahlen. C o m p u t i n g , Vol. 7, No. 3-4.

Shannon, C.E. (1959). Coding theorems for a discrete source with fidelity criterion. IRE Nat. Con. Record.

Sholomov, L.A. (1967). On functionals t h a t characterize the complexity of systems of incompletely specified Boolean functions. P r o b l e a i K i b e m e t i k i Vol. 19, Moscow, Nauka. (In Russian).

Valski, P.E. (1959). On t h e minimum number of multiplications neces- sary to raise to a given power. ProblePni K i b e m e t i k i , Vol. 2 , Moscow, Nauka. (In Russian).