A Differential Model for a 2x2-Evolutionary Game Dynamics

(1)

A Diﬀerential Model for a

2 × 2-Evolutionary Game Dynamics

A. M. Tarasyev

WP-94-63 August 1994

IIASA

International Institute for Applied Systems Analysis A-2361 Laxenburg Austria Telephone: 43 2236 807 Fax: 43 2236 71313 E-Mail: info@iiasa.ac.at

(2)

A Diﬀerential Model for a

2 × 2-Evolutionary Game Dynamics

A. M. Tarasyev

WP-94-63 August 1994

Working Papers are interim reports on work of the International Institute for Applied Systems Analysis and have received only limited review. Views or opinions expressed herein do not necessarily represent those of the Institute, its National Member Organizations, or other organizations supporting the work.

IIASA

International Institute for Applied Systems Analysis A-2361 Laxenburg Austria Telephone: 43 2236 807 Fax: 43 2236 71313 E-Mail: info@iiasa.ac.at

(3)

Foreword

A dynamical model for an evolutionary nonantagonistic (nonzero sum) game between two populations is considered. A scheme of a dynamical Nash equilibrium in the class of feedback (discontinuous) controls is proposed. The construction is based on solutions of auxiliary antagonistic (zero-sum) differential games. A method for approximating the correspondingvalue functions is developed. The method uses approximation schemes for constructinggeneralized (minimax, viscosity) solutions of first order partial differential equations of Hamilton-Jacobi type. A numerical realization of a grid procedure is described. Questions of convergence of approximate solutions to the generalized one (the value function) are discussed, and estimates of convergence are pointed out. The method provides equilibrium feedbacks in parallel with the value functions. Implementation of grid approximations for feedback control is justified. Coordination of long- and short-term interests of populations and individuals is indicated. A possible relation of the proposed game model to the classical replicator dynamics is outlined.

(4)

(5)

A Diﬀerential Model for a

2 × 2-Evolutionary Game Dynamics

A. M. Tarasyev

^∗

Introduction

We consider a nonantagonistic dynamical game of two large groups (populations) of individuals. The dynamical system describingpopulation evolution is motivated by differential (see [Isaacs, 1965]) and evolutionary game-theoretical models (see [Friedman, 1991], [Young, 1993]) relevant to problems of economic change (see [Nelson and Winter, 1982]) and population dynamics (see [Hofbauer, Sigmund, 1988]). For a particular class of 2×2 deterministic evolutionary game dynamics, an approach to analyse populations’ behaviors via methods of the theory of differential games was proposed in [Kryazhimskii, 1994]. In the present paper we develop some aspects of this approach for a stochastic 2×2 evolutionary game dynamics. Namely, we focus on finding equilibrium populations’ behaviors within the totally centralized regulation pattern. The model is reduced to a closed-loop differential game ([Krasovskii, Subbotin, 1988], [Krasovskii, 1985], [Kleimenov, 1993]) and analysed via methods of the theory of generalized (minimax, viscosity) solutions of Hamilton-Jacobi equations ([Crandall, Lions, 1983, 1984], [Subbotin, 1980, 1991]).

It is supposed that at each time instant, individuals of each population are divided into two parts playingdifferent strategies. Individuals from different populations meet pairwise randomly and get their current payoffs determined by a combination of their strategies. Populations’ goals are to maximize “long-term” payoffs represented as intergals of mathematical expectations for current payoffs, with an appropriate discount.

The right-hand side of the considered dynamical system depends on control parameters making individuals change their current strategies in accordance with a chosen feedback.

The nonantagonistic game in question consists in constructing Nash equilibrium feedbacks with respect to the “long-term” dynamical payoﬀ functionals.

We consider the problem within the framework of the theory of positional differential games ([Krasovskii, Subbotin, 1988]). Following to [Kleimenov, 1993] we compose a Nash equilibrium with the help of solutions of auxiliary antagonistic (zero-sum) differential games. Solutions of these antagonistic games are based on algorithms of building the value functions. It is known ([Crandall, Lions, 1983, 1984], [Subbotin, 1980, 1991]) that the value function is the generalized solution of the Bellman-Isaacs equation being a first-order partial differential equation of Hamilton-Jacobi type. To construct value functions we use appropriate approximation schemes ([Dolcetta, 1983], [Souganidis, 1985], [Tarasyev, 1993], [Adiatulina, Tarasyev, 1987], [Bardi, Osher], [Subbotin, Tarasyev, Ushakov]). The correspondingnumerical procedure is reduced to the method of contraction operators.

Alongwith information of the value functions, the method provides equilibrium feedbacks.

Stress once again that a solution is obtained within the centralized scheme implying that long-term-equilibrium behaviors can, generally, contradict to short-term interests

∗Instinute of Mathematics and Mechanics, Russian Acad. Sci., Ekaterinburg, Russia

(8)

of individuals. We conclude the paper with a discussion of possible problem settings combininglong- and short-term principles for constructingdynamical Nash equilibria. In particular, we consider the possibility of linkingthe proposed dynamical Nash equilibrium approach with the classical replicator dynamics (see [Hofbauer, Sigmund, 1988]).

1 The Model of Game Dynamics

1.1 Dynamics, Payoﬀs, Player’s Preferences

We consider the followingdynamical system which describes game interaction of two populations of individuals. We can assume, for example, that one of this population is an aggregate of sellers and another is an aggregate of buyers. For clearness of arguments suppose that individuals of populations can choose at each moment of time one of two simple actions (strategies): buyers can ”buy” or ”not buy”, sellers can sell at ”high price”

or ”low price”. Actions of individuals of the ﬁrst population are denoted by index i:

index i = 1 corresponds to action ”buy”, index i = 2 corresponds to action ”not buy”.

Analogously, actions of individuals of the second population are denoted by indexj: index j = 1 corresponds to ”high price”, indexj = 2 corresponds to ”low price”.

Let us consider an arbitrary pair composed by individuals of diﬀerent populations.

This pair is interpreted as a situation (i, j) in the current game generated by strategyi of a player from the ﬁrst population and by strategyj of a player from the second population.

Assume that payoff of players of the first population is determined by coefficients aij of the payoff matrix A = {a_ij}. Analogously, payoff of players of the second population in situation (i, j) is determined by coefficientsbij of the payoff matrix B ={bij}.

Let us assume that the first population consists of N individuals and at the instant of time t one part of them N₁(t) plays the first strategy and another part N₂(t) plays the second strategy. Of course, N = N₁(t) +N₂(t). Similarly, assume that the second population consists of M individuals and at the momentt one part M1(t) plays the first strategy and another part M₂(t) plays the second strategy, M =M₁(t) +M₂(t).

Let us suppose that dynamics of the process in which individuals change their strategies from one to another is described by the multistep system of equations

N₁(t+δ) = N₁(t)−n₁₂(t)δ+n₂₁(t)δ N₂(t+δ) = N₂(t) +n₁₂(t)δ−n₂₁(t)δ M1(t+δ) = M1(t)−m12(t)δ+m21(t)δ

M₂(t+δ) = M₂(t) +m₁₂(t)δ−m₂₁(t)δ (1.1) The peculiarity of such dynamics consists in the fact that the number of individuals in populations which can change their strategies at the momenttis proportional to the time step δ. More precisely,

(9)

n₁₂(t)δ is the number of individuals of the ﬁrst population which change their strategies from the ﬁrst to the second, 0≤n₁₂(t)≤N₁(t);

n21(t)δ is the number of individuals of the ﬁrst population which change their strategies from the second to the ﬁrst, 0≤n₂₁(t)≤N₂(t);

m₁₂(t)δ is the number of individuals of the second population which change their strategies from the ﬁrst to the second, 0≤m₁₂(t)≤M₁(t);

m21(t)δ is the number of individuals of the second population which change their strategies from the second to the ﬁrst, 0≤m₂₁(t)≤M₂(t).

The fact that at the moment t only a part of individuals in population proportional to the time step δ can change their strategies has the following interpretations. For example, such inertia of behaviour of population can be explained if we assume that only

”small” part of individuals is active in changing of their behaviour. We can give another explanation if assume that there are some restrictions (”queues”) in case when ”large”

group of individuals change actions.

On the other hand we make rather natural assumption when suppose that the number n_ik(t) orm_jl(t) of individuals which potentially may wish to change their actions (but not obligatory change, because the number of those who change is equal ton_ik(t)δorm_jl(t)δ) satisﬁes the restrictions

0≤n_ik ≤N_i(t), i, k= 1,2, i =k 0≤m_jl ≤M_j(t), j, l = 1,2, j =l

Let us suppose that at the moment t players of diﬀerent populations compose pairs randomly with equal probabilities. The probability of the fact that the randomly chosen pair plays the situation (i, j) is determined by the formula

pij(t) = N_i(t)M_j(t)

N M (1.2)

It is easy to verify standard relations for probabilities p_ij(t) pij(t)≥0,

i,j

pij(t) = 1, i, j = 1,2 (1.3) Let us pass from the multistep dynamical system (1.1) which connects quantities Ni(t+δ) and Mj(t +δ) with quantities Ni(t) and Mj(t) to the system which connects probabilities pij(t+δ) and pij(t).

Let us compose, for example, the correspondingdynamical equation for the probability p11(t+δ). We have

p₁₁(t+δ) = N₁(t+δ)M₁(t+δ) N M

= (N₁(t)−n₁₂(t)δ+n₂₁(t)δ)(M₁(t)−m₁₂(t)δ+m₂₁(t)δ) N M

= N₁(t)M₁(t)

N M −

(10)

N₁(t)M₁(t) N M

m₁₂(t)

M1(t)δ+N₁(t)M₂(t) N M

m₂₁(t) M2(t)δ− N₁(t)M₁(t)

N M

n₁₂(t)

N₁(t)δ+ N₂(t)M₁(t) N M

n₂₁(t) N₂(t)δ+ (−n12(t) +n21(t))(−m12(t) +m21(t))

N M δ²

Takinginto account notations for probabilities p_ij(t) we obtain the equation

p₁₁(t+δ)−p₁₁(t) =−p₁₁(t)v₁δ+p₁₂(t)v₂δ−p₁₁(t)u₁δ+p₂₁(t)u₂δ+φ(t)δ² (1.4) Here

u₁ =u₁(t) = n₁₂(t) N1(t) u2 =u2(t) = n₂₁(t) N₂(t) v₁ =v₁(t) = m12(t)

M₁(t) v₂ =v₂(t) = m₂₁(t)

M2(t) (1.5)

0≤ui≤ 1, i= 1,2

0≤v_j ≤ 1, j = 1,2 (1.6)

|φ(t)| ≤1

Dividingequation (1.4) into δ > 0 and passingto limit when δ ↓ 0 we come to the diﬀerential equation

˙

p₁₁(t) = −p₁₁(t)u₁(t) +p₂₁(t)u₂(t)−p₁₁(t)v₁(t) +p₁₂(t)v₂(t) Analogously one can deduce diﬀerential equations for ˙p12(t), ˙p21(t), ˙p22(t).

Let us write diﬀerential equations which describe the motion of the considered dynamical system usingstandard notations

x₁ =p₁₁, x₂ =p₁₂, x₃ =p₂₁, x₄ =p₂₂ (1.7) We obtain the followingbilinear system of diﬀerential equations with respect to probabilities x₁, x₂, x₃, x₄

˙

x₁ = −x₁u₁+x₃u₂−x₁v₁+x₂v₂ =f₁(x, u, v)

˙

x₂ = −x₂u₁+x₄u₂+x₁v₁−x₂v₂ =f₂(x, u, v)

˙

x₃ = x₁u₁−x₃u₂−x₃v₁+x₄v₂ =f₃(x, u, v)

˙

x₄ = x₂u₁−x₄u₂+x₃v₁−x₄v₂ =f₄(x, u, v)

(1.8)

Here

x= (x₁, x₂, x₃, x₄), u= (u₁, u₂), v = (v₁, v₂)

(11)

1.2 Properties of Dynamical System

Let us turn our attention to some properties of control dynamical system (1.8). This dynamics conserves the followingproperties of probabilities.

Lemma 1.1 If

x1(0) +x2(0) +x3(0) +x4(0) = 1 (1.9) then

x₁(t) +x₂(t) +x₃(t) +x₄(t) = 1 ∀t (1.10)

Proof. Actually

f₁(x, u, v) +f₂(x, u, v) +f₃(x, u, v) +f₄(x, u, v) = 0 and, hence,

˙

x1(t) + ˙x2(t) + ˙x3(t) + ˙x4(t) = 0 ∀t We obtain

x₁(t) +x₂(t) +x₃(t) +x₄(t) =c ∀t From (1.9) we have c= 1.

Lemma 1.2 If

x₁(0)x₄(0)−x₂(0)x₃(0) = 0 (1.11) then

x1(t)x4(t)−x2(t)x3(t) = 0 ∀t (1.12)

Proof. Let us note that (1.11) takes place for our model because x₁(0)x₄(0)−x₂(0)x₃(0) = N1M1N2M2

N M − N1M2N2M1

N M = 0

Let

z(t) =x₁(t)x₄(t)−x₂(t)x₃(t), z(0) = 0 We have

˙

z = ˙x₁x₄ +x₁x˙₄−x˙₂x₃−x₂x˙₃= (−x1x4+x2x3)(u1 +u2+v1+v2) =

−z(u₁+u₂+v₁+v₂) Hence

z(t) =z(0) exp(− ^t

0

(u₁(s) +u₂(s) +v₁(s) +v₂(s))ds) Since z(0) = 0 then z(t)≡0.

Thus the relations

x₁+x₂+x₃+x₄ = 1 (1.13)

x₁x₄−x₂x₃ = 0 (1.14)

are ﬁrst integrals for dynamical system (1.8).

One can prove also that system (1.8) conserves the followingproperties of probabilities.

(12)

Lemma 1.3 If

0≤x_i(0)≤1 (1.15)

then

0≤x_i(t)≤1 ∀t i = 1,2,3,4 (1.16)

Proof. We prove this fact below for the reduced system.

1.3 Reduced Dynamical Systems

Let us note that since there are exist two ﬁrst integrals (1.13),(1.14) for dynamical system (1.8) then its order can be reduced from the fourth to the second. We shall make this reduction in convenient way for us. To this end we introduce the followingvariables

y1 =x1+x2 is the probability of the fact that a player from the ﬁrst population holds the ﬁrst strategy y₂ =x₃+x₄ is the probability of the fact that a player from

the ﬁrst population holds the second strategy y₃ =x₁+x₃ is the probability of the fact that a player from

the second population holds the ﬁrst strategy y4 =x2+x4 is the probability of the fact that a player from

the second population holds the second strategy It is obvious that

y₁+y₂ = 1, y₃+y₄ = 1 (1.17)

Using(1.13),(1.14) one can prove also that

y₁y₃ =x₁, y₁y₄ =x₂, y₂y₃ =x₃, y₂y₄ =x₄ (1.18) Actually, we have, for example,

y₁y₃ = (x₁+x₂)(x₁+x₃) =x₁(x₁ +x₂+x₃) +x₂x₃=x₁(x₁+x₂+x₃+x₄) = x₁ From dynamical system (1.8) we obtain the followingcontrol system with respect to probabilities y₁, y₂, y₃, y₄

˙

y₁ = −y₁u₁+y₂u₂

˙

y₂ = y₁u₁−y₂u₂

˙

y3 = −y3v1+y4v2

˙

y₄ = y₃v₁−y₄v₂

(1.19) Introducingnotations y₁ = x, y₃ = y we obtain from (1.17) that y₂ = 1−x, y₄ = 1−y. Substitutingthese relations to (1.19) we come to the followingbilinear system of diﬀerential equations with respect to probabilities x and y

˙

x = −xu1+ (1−x)u2

˙

y = −yv₁+ (1−y)v₂ (1.20)

Let us remind that controlsu1, u2, v1, v2 are pure numbers here. They are determined by relations (1.5) and satisﬁes restrictions (1.6). The extreme values of controlsu_i = 0 or u_i = 1, i = 1,2 and v_j = 0 or v_j = 1, j = 1,2 can be enterpreted as signals to individuals of corresponding populations to change or not to change their actions. For example, the

(13)

value u1 = 0 signals to individuals of the first population to hold the first strategy, not to change actions, and the value u₁ = 1 signals about necessity to alter the first strategy for the second.

In reality the system (1.20) can be replaced by the equivalent system of more simple type. Let us consider, for example, the ﬁrst equation of the system (1.20) and transform it in the followingway

˙

x=−xu₁+ (1−x)u₂ =−x+x(1−u₁) + (1−x)u₂ Let us introduce the new control parameter

u =x(1−u₁) + (1−x)u₂ (1.21)

We determine now the restrictions for the control parameter u. We have u∈P(x), P(x) =P₁(x) +P₂(x)

P₁(x) ={x(1−u₁) : 0≤u₁≤1, 0≤x≤1}= [0, x]

P₂(x) = {(1−x)u₂ : 0≤u₂ ≤1, 0≤x≤1}= [0,1−x]

Hence, the set P(x) = [0, x] + [0,1−x] is the seg ment [0,1] and it does not depend on x. Thus, the ﬁrst equation of the system (1.20) can be replaced by the equation

˙

x=−x+u, 0≤u≤1 Analogously, if we introduce the new control parameter

v=y(1−v1) + (1−y)v2 (1.22)

we obtain the diﬀerential equation with respect to the probability y

˙

y=−y+v, 0≤v≤1

Thus, we have the followingsystem of diﬀerential equations with respect to probabilities xand y

˙

x = −x+u, 0≤u≤1

˙

y = −y+v, 0≤v ≤1 (1.23)

Finally, let us verify that system (1.23) conserves properties of probabilities x and y.

Proof of Lemma 1.3.

Let u(s) : [0,+∞)→[0,1] be a control function measurable in the sense of Lebesgue.

Then accordingto Cauchy formula we have x(t) = (x0+

t 0

exp(s)u(s)ds) exp(−t)

If x(0) =x₀ ≥0 then it is obvious that x(t)≥0. Let x(0) = x₀ ≤1. Then x(t)≤(1 +

_t

0

exp(s)u(s)ds) exp(−t)≤ exp(−t) +

₀

−t

exp(s)u(s+t)ds≤ exp(−t) +

0

−texp(s)ds≤1 (1.24)

Since system (1.23) conserves properties of probabilities then equivalent systems (1.8) and (1.20) also conserve these properties.

(14)

2 The Model of ”Short-Term” and ”Long-Term”

Payoﬀs

2.1 ”Short-Term” Payoﬀs

Let us pass now to the question about evaluation of payoffs of populations. It is naturally to assume that the mathematical expectation connected with the correspondingpayoff matrix is the payoff of population at the moment of time t. Namely, quality of the state x(t) = (x1(t), x2(t), x3(t), x4(t)) of dynamical process (1.8) is evaluated for the first population by the mathematical expectation

EA(t) =a11x1(t) +a12x2(t) +a21x3(t) +a22x4(t) (2.1) and for the second population - by the mathematical expectation

E_B(t) =b₁₁x₁(t) +b₁₂x₂(t) +b₂₁x₃(t) +b₂₂x₄(t) (2.2) Takinginto account the notations of the equivalent system (1.23) we can rewrite formulas (2.1),(2.2) by means of probabilities x(t), y(t) in the followingway

EA(t) =a11x(t)y(t) +a12x(t)(1−y(t)) + a₂₁(1−x(t))y(t) +a₂₂(1−x(t))(1−y(t)) = (a₁₁−a₁₂−a₂₁+a₂₂)x(t)y(t)−(a₂₂−a₁₂)x(t)−

(a₂₂−a₂₁)y(t) +a₂₂ (2.3) E_B(t) =b₁₁x(t)y(t) +b₁₂x(t)(1−y(t)) +

b21(1−x(t))y(t) +b22(1−x(t))(1−y(t)) = (b₁₁−b₁₂−b₂₁+b₂₂)x(t)y(t)−(b₂₂−b₁₂)x(t)−

(b₂₂−b₂₁)y(t) +b₂₂ (2.4) Let us note that in the theory of static bimatrix games (see, for example [Vorobjev, 1984]) there are special notations for the coeﬃcients of formulas (2.3),(2.4)

C_A=a₁₁−a₁₂−a₂₁+a₂₂ α₁ =a₂₂−a₁₂

α₂ =a₂₂−a₂₁

(2.5) C_B =b₁₁−b₁₂−b₂₁+b₂₂

β1=b22−b12

β₂=b₂₂−b₂₁

(2.6) It is naturally to assume that formulas (2.3)-(2.4) describe the ”short-term” (calculated at the ﬁxed moment of timet) payoﬀs of populations.

2.2 ”Long-Term” Payoﬀs

Let us consider now the dynamical system (1.23) on the inﬁnite interval of time [0,+∞).

Inﬁnity of the interval of time means that we are interested namely in the evolutionary character of behaviour of trajectories generated by the dynamical system.

(15)

Let (x(·), y(·)) = {(x(t), y(t)) : t∈ [0,+∞)} be an arbitrary trajectory of the system (1.23). We shall estimate the quality of this trajectory by the integral functionals with discount coeﬃcients. For the ﬁrst population we determine the quality of a trajectory by the functional

J₁ =J₁(x(·), y(·)) =

+∞ 0

exp(−λt)E_A(t)dt=

+∞ 0

exp(−λt)g₁(x(t), y(t))dt=

₊_∞

0

exp(−λt)(CAx(t)y(t)−α1x(t)−α2y(t) +a22)dt (2.7) and for the second population - by the functional

J₂ =J₂(x(·), y(·)) =

₊_∞

0

exp(−λt)E_B(t)dt =

₊_∞

0

exp(−λt)g₂(x(t), y(t))dt=

+∞

0 exp(−λt)(CBx(t)y(t)−β1x(t)−β2y(t) +b22)dt (2.8) Hereλ >0 is the so-called coeﬃcient of discount (which means discountingof the process with growth of time). Functionals of such type are rather traditional for mathematical models in economics (see, for example, references in [Dolcetta, 1983]). Let us note that integrals (2.7), (2.8) always exist since 0≤x(t)≤1, 0≤y(t)≤1.

Functionals (2.7),(2.8) determine ”long-term” (on the infinite interval of time) payoffs of populations in contrast to ”short-term” payoffs (2.1)-(2.4).

Integrals (2.7),(2.8) can be interpreted also in terms of ”average” mathematical expectations. Indeed, let us, for example, consider integral (2.7). We normalize it by multiplyingon the discount coeﬃcientλ. We have

λ

₊_∞

0

exp(−λt)(C_Ax(t)y(t)−α₁x(t)−α₂y(t) +a₂₂)dt= a₁₁

+∞ 0

λexp(−λt)x₁(t)dt+a₁₂

+∞ 0

λexp(−λt)x₂(t)dt+ a₂₁

₊_∞

0

λexp(−λt)x₃(t)dt+a₂₂

₊_∞

0

λexp(−λt)x₄(t)dt=

i,j

a_ij

₊_∞

0

λexp(−λt)p_ij(t)dt= a₁₁p₁₁+a₁₂p₁₂+a₂₁p₂₁+a₂₂p₂₂=

E_A(x(·), y(·)) (2.9) Here

p_ij =

₊_∞

0

λexp(−λt)p_ij(t)dt, i, j = 1,2 (2.10) It is easy to verify that 0≤pij ≤1 and_i,jpij = 1. Hence, one can interpret numbers p_ij as some special averaging (2.10) of probabilities p_ij(t), t ∈ [0,+∞) on the infinite interval of time. Therefore, it is naturally to regard number p_ij as ”average” probability of the fact that random pairs of individuals play the situation (i, j) on the infinite interval of time. The functional (2.9) is interpreted then as ”average” mathematical expectation E_A(x(·), y(·)) of payoff for the first population on the infinite interval of time.

(16)

Analogously, the normalized functional (2.8) can be considered as ”average” mathematical expectation E_B(x(·), y(·)) of payoﬀ for the second population in inﬁnite horizon

λ

₊_∞

0

exp(−λt)(CBx(t)y(t)−β1x(t)−β2y(t) +b22)dt = b₁₁

₊_∞

0

λexp(−λt)x₁(t)dt+b₁₂

₊_∞

0

λexp(−λt)x₂(t)dt+ b21

+∞ 0

λexp(−λt)x3(t)dt+b22

+∞ 0

λexp(−λt)x4(t)dt =

i,j

b_ij

₊_∞

0

λexp(−λt)p_ij(t)dt = b₁₁p₁₁+b₁₂p₁₂+b₂₁p₂₁+b₂₂p₂₂ =

E_B(x(·), y(·)) (2.11)

3 Nash Equilibria in Dynamical System

3.1 Feedback Controls, Trajectories of Dynamical System

Let us assume that controls u and v for the ﬁrst and the second populations in system (1.23) can be formed on the feedback principle. We suppose also that feedback controls (strategies) U = u(t, x, y, ε) and V = v(t, x, y, ε) can be discontinuous functions of phase variables (x, y). For deﬁnition of motions of the system generated by discontinuous controls U = u(t, x, y, ε), V = v(t, x, y, ε) we shall use the approach proposed in [Krasovskii, Subbotin, 1988]. Namely, let [0, T] be an interval of time,

∆ = {t₀ = 0 < t₁ < t₂ < ... < t_N = T} be its partition with an instant time-step δ = t_k+1−t_k and ε > 0 be a parameter of accuracy 0 < δ < β(ε) where β(ε) ↓ 0 when ε ↓ 0. Consider piecewise diﬀerentiable trajectory (x_∆(·), y_∆(·)) which is called ”Euler spline” and is actually the step-by-step solution of the followingdiﬀerential equation

˙

x∆(t) =−x∆(t) +u(tk, x∆(tk), y∆(tk), ε) (3.1)

˙

y_∆(t) =−y_∆(t) +v(t_k, x_∆(t_k), y_∆(t_k), ε) t∈[tk, tk+1), k = 0,1, ..., N−1

x_∆(0) =x₀, y_∆(0) =y₀

The uniformly continuous limits of ”Euler splines” when N → ∞, ε ↓ 0, δ ↓ 0 are called limit motions or simply motions of the system. The set of all these motions will be denoted by the symbol XT(x0, y0, U, V). This set is a compactum in the space of continuous functions determined on [0, T].

Definition 3.1 A continuous function (x(t), y(t)) : [0,+∞) →R² is called trajectory of the system (1.23) on the infinite interval of time generated by strategies U and V from the initial position (x₀, y₀) if for any moment T,0 < T < ∞ there exists a trajectory (x_T(t), y_T(t))∈X_T(x₀, y₀, U, V) which satisfies the condition (x(t), y(t)) = (x_T(t), y_T(t)), t ∈ [0, T]. The set of all these trajectories(x(t), y(t)) : [0,+∞)→ R² will be denoted by the symbol X(x₀, y₀, U, V).

(17)

3.2 Nash Equilibria

Let us introduce deﬁniton of Nash equilibria for pairs of feedback controls (U =u(t, x, y, ε), V =v(t, x, y, ε)).

Deﬁnition 3.2 A pair of feedback controls(U⁰, V⁰) is called optimal (equilibrium) in the sense of Nash for the ﬁxed initial position (x₀, y₀)∈[0,1]×[0,1]if for any other feedback controls U and V the following condition holds. For all trajectories

(x⁰(·), y⁰(·))∈X(x0, y0, U⁰, V⁰), (x1(·), y1(·))∈X(x0, y0, U, V⁰) (x₂(·), y₂(·))∈X(x₀, y₀, U⁰, V) inequalities

J1(x⁰(·), y⁰(·))≥J1(x1(·), y1(·))

J₂(x⁰(·), y⁰(·))≥J₂(x₂(·), y₂(·)) (3.2) take place.

Remark 3.1 For construction of Nash equilibria we shall use the scheme proposed in [Kleimenov, 1993]. We shall give the short description of this scheme below in Section 4.

We need to modify slightly Deﬁnition 3.2 because in reality we construct some ε- approximations of optimal feedback controls. Therefore, let us introduce the notion of ε-equilibria.

Deﬁnition 3.3 Let ε >0and (x₀, y₀)∈[0,1]×[0,1]. A pair of feedback controls (U_ε, V_ε) is called ε-optimal (ε-equilibrium) in the sense of Nash for the ﬁxed initial position(x0, y0) if for any other feedback controls U and V the following condition holds. For all trajectories

(x⁰(·), y⁰(·))∈X(x₀, y₀, U_ε, V_ε), (x₁(·), y₁(·))∈X(x₀, y₀, U, V_ε) (x₂(·), y₂(·))∈X(x₀, y₀, U_ε, V) inequalities

J₁(x⁰(·), y⁰(·))≥J₁(x₁(·), y₁(·))−ε

J2(x⁰(·), y⁰(·))≥J2(x2(·), y2(·))−ε (3.3) take place.

Remark 3.2 Nash equilibria which will be constructed below possess indeed the more strong properties than the properties indicated in Deﬁnitions 3.2, 3.3. These strong properties can be interpreted as dynamical stability of equilibria. Namely, we say that the Nash equilibrium has the property of dynamical stability if it is not proﬁtable for populations to deviate from equilibrium feedback controls even along the whole equilibrium trajectory (x⁰(·), y⁰(·)), i.e. the following enequalities hold for all t_∗ >0

+∞ t_∗

exp(−λt)g1(x⁰(t), y⁰(t))dt≥ ⁺^∞

t_∗

exp(−λt)g1(x1(t), y1(t))dt−ε

+∞ t_∗

exp(−λt)g2(x⁰(t), y⁰(t))dt≥ ⁺^∞

t_∗

exp(−λt)g2(x2(t), y2(t))dt−ε (3.4) with ε = 0for Deﬁnition 3.2 and with ε >0 for Deﬁnition 3.3.

(18)

3.3 Stability Properties of Dynamical System

Let us formulate the property of stability for dynamical system (1.23).

Lemma 3.1 Let u(t) : [0,+∞) →[0,1], v(t) : [0,+∞)→ [0,1] be measurable controls and (x₁(·), y₁(·)),(x₂(·), y₂(·)) be two trajectories of the system (1.23) generated by these controls from diﬀerent initial positions

(x1(0), y1(0)) = (x1, y1), (x2(0), y2(0)) = (x2, y2) Then

|x1(t)−x2(t)| ≤ |x1−x2|exp(−t)

|y₁(t)−y₂(t)| ≤ |y₁−y₂|exp(−t) (3.5) Moreover, there exists a constant C >0 depending only on coeﬃcients of matrixes A and B such that the following estimates take place

|J_k(x₁(·), y₁(·))−J_k(x₂(·), y₂(·))| ≤ C

1 +λmax{|x₁−x₂|,|y₁−y₂|}, k = 1,2 (3.6) Proof. Consider, for example, diﬀerential equations for x1(·) andx2(·)

˙

x₁(t) =−x₁(t) +u(t), x₁(0) =x₁

˙

x₂(t) =−x₂(t) +u(t), x₂(0) =x₂ Subtractingthe second equation from the ﬁrst we obtain

∆ ˙x(t) =−∆x(t), ∆x(0) =x₁−x₂

∆x(t) =x1(t)−x2(t) Hence

∆x(t) = ∆x(0) exp(−t) Analogously, we can obtain

∆y(t) = ∆y(0) exp(−t) For the diﬀerence of functionals we have the estimate

|J₁(x₁(·), y₁(·))−J₁(x₂(·), y₂(·))| ≤

+∞ 0

exp(−λt)|CA(x1(t)y1(t)−x2(t)y2(t))−α1(x1(t)−x2(t))−α2(y1(t)−y2(t))|dt≤ C

₊_∞

0

exp(−λt) max{|x₁(t)−x₂(t)|,|y₁(t)−y₂(t)|}dt≤ Cmax{|x1−x2|,|y1−y2|} ⁺^∞

0 exp(−(1 +λ)t)dt= C

1 +λmax{|x1−x2|,|y1−y2|}

Finally we give the following corollary.

Corollary 3.1 For integral functionals with ﬁnite horizonT, 0≤T <+∞the estimate similar to (3.6) takes place

| ^T

0

exp(−λt)g_k(x₁(t), y₁(t))dt− ^T

0

exp(−λt)g_k(x₂(t), y₂(t))dt| ≤ C

1 +λmax{|x₁−x₂|,|y₁−y₂|} (3.7)

(19)

4 Construction of Nash Equilibria

4.1 Auxiliary Antagonistic (Zero-Sum) Games

In order to construct equilibrium feedback controls we use the approach proposed in the theory of diﬀerential games (see, for example, [Kleimenov, 1993]).

Let us consider auxiliary antagonistic (zero-sum) diﬀerential games Γ₁ and Γ₂ with the functionals J₁ (2.7) and J₂ (2.8) respectively. In the game Γ₁ the ﬁrst population tries to maximize the functional J₁(x(·), y(·)) usingfeedback controls U = u(t, x, y, ε).

The second population has the opposite aim, it tries to minimize this functional using feedback controlsV =v(t, x, y, ε). Conversely in the game Γ2 the ﬁrst population aims for minimization of the functionalJ₂(x(·), y(·)) and the second population wishes to maximize it.

By the symbols w₁(x, y) and w₂(x, y) we denote value functions of auxiliary antagonistic games Γ₁ and Γ₂. It is known (see [Krasovskii, Subbotin, 1988], [Krasovskii, 1985]) that optimal feedback controls (control synthesis) U_i =u⁰_i(t, x, y, ε), i= 1,2 and Vj = v_j⁰(t, x, y, ε), j = 1,2 of the ﬁrst and the second population in this antagonistic game can be constructed on the information of value functions w_k(·), k = 1,2.

Strategies u⁰₁(t, x, y, ε) and v⁰₂(t, x, y, ε) can be interpreted as strategies which have positive nature (we shall call them ”positive” strategies) because they are aimed for maximization of their own quality functional. Let us mention that these strategies are cautious (guaranteed) feedback controls. Strategies u⁰₂(t, x, y, ε) and v₁⁰(t, x, y, ε) can be considered as strategies of ”punishment” because they minimize the payoﬀ functional of another population.

4.2 Equilibrium Feedback Controls

Let us construct now the pair of feedback strategies which forms Nash equilibrium by past- ing together ”positive” and ”punishment” strategiesu⁰_i(t, x, y, ε) andv_j⁰(t, x, y, ε), i, j = 1,2 of two populations.

Let (x0, y0) ∈ [0,1]×[0,1] be an arbitrary initial position, ε > 0 be an accuracy parameter and (x(·), y(·))∈X(x₀, y₀, u₁(·), v₂(·)) be a trajectory generated by ”positive”

strategies u⁰₁(t, x, y, ε) and v₂⁰(t, x, y, ε). Let Tε >0 be such a moment of time that

₊_∞

Tε

exp(−λt)|g_i(x(t), y(t))|dt < ε

By the symbols u_ε(t) : [0, T_ε) → [0,1], v_ε(t) : [0, T_ε) → [0,1] we denote step-by-step realizations of strategiesu⁰₁(t, x, y, ε), v⁰₂(t, x, y, ε) such that the correspondingstep-by- step motion (x_ε(·), y_ε(·)) satisﬁes the condition

t∈max[0,Tε](x(t), y(t))−(xε(t), yε(t))< ε

It is aﬃrmed that the followingpair of feedback controls U⁰ = u⁰(t, x, y, ε), V⁰ = v⁰(t, x, y, ε) pasted with the help of ”positive” strategies u⁰₁(t, x, y, ε), v⁰₂(t, x, y, ε) and

”punishment” strategies u⁰₂(t, x, y, ε), v₁⁰(t, x, y, ε) forms an ε-equilibrium situation in the sense of Nash

U⁰ =u⁰(t, x, y, ε) =

u_ε(t) if (x, y)−(x_ε(t), y_ε(t))< ε

u⁰₂(t, x, y, ε) otherwise (4.1)

(20)

V⁰ =v⁰(t, x, y, ε) =

vε(t) if (x, y)−(xε(t), yε(t))< ε

v⁰₁(t, x, y, ε) otherwise (4.2)

Let us remind that programming controls u_ε(t), v_ε(t) are realizations of ”positive”

strategiesu⁰₁(t, x, y, ε),v₂⁰(t, x, y, ε). In other words the “acceptable” trajectory (x_ε(·), y_ε(·)) is generated by “positive interests” of populations. The numberε can be interpreted as a parameter of “reliance” of populations to each other or a level of “risk” which populations allow in the game.

5 Value Functions of Diﬀerential Games

5.1 Value Functions for Games with Inﬁnite Horizon

Let us consider now the auxiliary antagonistic (zero-sum) game Γ₁ the dynamics of which is described by equations

˙

x=−x+u

˙

y=−y+v (5.1)

and the payoﬀ functional is determined by relation J₁(x(·), y(·)) =

+∞ 0

exp(−λt)g₁(x(t), y(t))dt=

₊_∞

0

exp(−λt)(C_Ax(t)y(t)−α₁x(t)−α₂y(t) +a₂₂)dt (5.2) The aim of the ﬁrst population is to maximize the functional J1 (5.2) on trajectories (x(·), y(·)) of the system (5.1) by disposingof control parameterU =u(t, x, y, ε). The aim of the second population is opposite: to minimize the functional J₁ (5.2) on trajectories (x(·), y(·)) of the system (5.1) by disposingof control parameter V =v(t, x, y, ε).

Accordingto formalization proposed in [Krasovskii, Subbotin, 1988] the antagonistic game (5.1),(5.2) has the value function (x₀, y₀)→w₁(x₀, y₀)

w₁(x₀, y₀) = sup

U

inf

(x(·),y(·))∈X(x0,y0,U)J₁(x(·), y(·)) = infV sup

(x(·),y(·))∈X(x0,y0,V)

J1(x(·), y(·)) (5.3) Here symbolsX(x₀, y₀, U), X(x₀, y₀, V) denote trajectories of dynamical system (5.1) generated by feedback controls U =u(t, x, y, ε) and V =v(t, x, y, ε).

5.2 Properties of Value Functions

Let us indicate some properties of value functionw₁ : [0,1]×[0,1]→R.

Property 5.1 Letw₁(t, x, y)be the value function for the diﬀerential game with dynamics (5.1) and payoﬀ functional

J₁(t, x(·), y(·)) =

+∞ t

exp(−λs)g₁(x(s), y(s))ds (5.4) x(t) =x, y(t) =y

Then value functions w1(x, y) and w1(t, x, y) are connected by relation

w₁(t, x, y) = exp(−λt)w₁(x, y) (5.5)

(21)

Property 5.2 Let w1(t, T, x, y) be the value function for the diﬀerential game with dynamics (5.1) and payoﬀ functional

J₁(t, T, x(·), y(·)) =

T t

exp(−λs)g₁(x(s), y(s))ds (5.6) x(t) =x, y(t) =y

Then value functions w₁(t, x, y)and w₁(t, T, x, y)are connected by inequality max

(t,x,y)∈[0,T]×[0,1]×[0,1]|w₁(t, x, y)−w₁(t, T, x, y)| ≤ K

λ exp(−λT) (5.7) Here parameter K depends only on coeﬃcients a_ij of matrix A={a_ij}.

Property 5.3 Value function w₁ is bounded max

(x,y)∈[0,1]×[0,1]|w₁(x, y)| ≤ K

λ (5.8)

Property 5.4 Value function w₁ satisﬁes the Lipschitz condition

|w₁(x₁, y₁)−w₁(x₂, y₂)| ≤ K

1 +λ(|x₁−x₂|+|y₁−y₂|)≤

√2K

1 +λ((x₁ −x₂)²+ (y₁−y₂)²)¹² (5.9)

Proof. Let us choose arbitrarily ε >0. Determine a moment of time T, 0≤T <+∞ from the relation

exp(−λT)K < ε i.e.

T > 1 λlnK

ε

Consider value function w₁(0, T, x, y). Accordingto Property 5.2 we have

|w1(x, y)−w1(0, T, x, y)| ≤ K

λ exp(−λT)≤ Kε λ

LetU⁰ and V⁰ be feedback controls realizingexternal extremum in relations which determine value function w₁(0, T, x, y), i.e.

w₁(0, T, x, y) = maxU min

(x(·),y(·))∈X(x,y,U)J₁(0, T, x(·), y(·)) = min

(x(·),y(·))∈X(x,y,U⁰)J1(0, T, x(·), y(·)) = minV max

(x(·),y(·))∈X(x,y,V)J₁(0, T, x(·), y(·)) =

(x(·),y(·))∈Xmax(x,y,V⁰)

J₁(0, T, x(·), y(·))

(22)

Consider the diﬀerence

w₁(0, T, x₁, y₁)−w₁(0, T, x₂, y₂) =

(x(·),y(·))min∈X(x,y,U⁰)J1(0, T, x(·), y(·))− max

(x(·),y(·))∈X(x,y,V⁰)J1(0, T, x(·), y(·)) = J1(0, T, x0(·), y0(·))−J1(0, T, x⁰(·), y⁰(·))≤ J₁(0, T, x₁(·), y₁(·))−J₁(0, T, x₂(·), y₂(·)) +ε

Here (x₁(·), y₁(·)) and (x₂(·), y₂(·)) are ”Euler splines” which are close enough to trajectories (x₀(·), y₀(·)) and (x⁰(·), y⁰(·)) realizingcorrespondingextremum.

Accordingto the stability property of dynamical system (see Corollary 3.1) we have the followingestimate

J₁(0, T, x₁(·), y₁(·))−J₁(0, T, x₂(·), y₂(·))≤ K

1 +λ(|x1−x2|+|y1−y2|) Let us note that the last estimate does not depend on T.

Combiningall estimates together we obtain one-sided inequality for the Lipschitz condition (5.9). Changing places of equal ”maxmin” and ”minmax” in the previous arguments we come to the complementary estimate of the Lipschitz condition. Thus, condition (5.9) is proven.

Remark 5.1 Properties 5.1 - 5.4 are valid also for the value function w2

w₂(x₀, y₀) = sup

V

inf

(x(·),y(·))∈X(x0,y0,V)J₂(x(·), y(·)) = infU sup

(x(·),y(·))∈X(x0,y0,U)

J2(x(·), y(·)) (5.10) of the second auxiliary diﬀerential game.

6 Value Functions and Minimax (Viscosity) Solu- tions of Hamilton-Jacobi Equations

6.1 Hamilton-Jacobi Equations

The most principal properties of value functions are so-called properties of stability (u and v stability [Krasovskii, Subbotin, 1988]) which express the principle of optimality (suboptimality, superoptimality) of dynamical programming. At points where value function w1(x, y) is differentiable these properties convert to the first order partial differential equation of Hamilton-Jacobi type which is called Bellman-Isaacs equation or the basic equation for optimal control problems. For our optimal guaranteed control problem (differential game) (5.1),(5.2) the corresponding Bellman-Isaacs equation has the following form

−λw(x, y)− ∂w

∂xx− ∂w

∂yy+

0max≤u≤1

∂w

∂xu+ min

0≤v≤1

∂w

∂yv= 0 (6.1)

(23)

Remark 6.1 Equation (6.1) does not depend on time t. It is an equation of stationary type.

It is easy to see that

0max≤u≤1

∂w

∂xu= max

0,∂w

∂x

0min≤v≤1

∂w

∂yv= min

0,∂w

∂y

(6.2) Therefore, equation (6.1) can be rewritten in the form

−λw(x, y)−∂w

∂xx−∂w

∂yy+ max

0,∂w

∂x

+ min

0,∂w

∂y

= 0 (6.3)

6.2 Generalized Derivatives, Diﬀerential Inequalities

Usually value function w₁(x, y) is not differentiable everywhere. It satisfies only the Lipschitz condition (5.9), i.e. it is only almost everywhere differentiable accordingto Rademaher theorem.

It is shown in the theory of minimax (viscosity) solutions [Subbotin, 1980, 1991], [Crandall, Lions, 1983, 1984] that value function w₁ must satisfy generalized diﬀerential inequalities at points where it is not diﬀerentiable (measure of this set is equal to zero).

These inequalities generalize Bellman-Isaacs equation and express the optimality principle of dynamical programming in inﬁnitesimal form.

In order to write the principle of dynamical programming in inﬁntesimal form let us introduce the notions of directional derivatives and conjugate derivatives for functions which satisfy the Lipschitz condition.

Let function w(x, y) : [0,1]×[0,1]→R satisfy the Lipschitz condition.

Deﬁnition 6.1 Lower and upper derivatives of functionwat a point(x, y)∈(0,1)×(0,1) in a direction h= (h₁, h₂)∈R² are determined by relations

∂₋w(x, y)|(h) = lim inf

δ↓0

w(x+δh₁, y+δh₂)−w(x, y) δ

∂+w(x, y)|(h) = lim sup

δ↓0

w(x+δh₁, y+δh₂)−w(x, y)

δ (6.4)

Deﬁnition 6.2 Lower and upper conjugate derivatives of function w at a point (x, y)∈ (0,1)×(0,1) are determined by equalities

D^∗w(x, y)|(s) = sup

h∈R²

(s, h −∂₋w(x, y)|(h)) D_∗w(x, y)|(s) = inf

h∈R²(s, h −∂₊w(x, y)|(h)) (6.5)

(24)

It is proven (see, for example, [Subbotin, 1980, 1991], [Crandall, Lions, 1983, 1984], [Subbotin, Tarasyev, 1985], [Dolcetta, 1983], [Adiatulina, Tarasyev, 1987]) in the theory of minimax (viscosity) solution that value functionw₁(x, y) is uniquely determined by the pair of diﬀerential inequalities which connect conjugate derivatives with the Hamiltonian of dynamical system. Let us give this result for diﬀerential game (5.1),(5.2).

Theorem 6.1 For a Lipschitz continuous function w: [0,1]×[0,1]→R to be the value function of differential game (5.1),(5.2) it is necessary and sufficient that the following differential inequalities hold for all (x, y, s)∈(0,1)×(0,1)×R²

D^∗w(x, y)|(s)≥ −λw(x, y) +H(x, y, s) (6.6) D_∗w(x, y)|(s)≤ −λw(x, y) +H(x, y, s) (6.7)

Here the symbol H(x, y, s) denotes the Hamiltonian of dynamical system (5.1)

H(x, y, s) =−s₁x−s₂y+ max{0, s₁}+ min{0, s₂}+g₁(x, y) (6.8) s= (s1, s2)∈R²

g₁(x, y) =C_Axy−α₁x−α₂y+a₂₂

Remark 6.2 Inequalities (6.6),(6.7) turn into Bellman-Isaacs equation (6.3) at points where function w is diﬀerentiable.

Remark 6.3 Differential inequality (6.7) expresses the so-called property of u-stability of the value function w which implies the existence of directions of nondecrease. Similarly, differential inequality (6.6) expresses property of v-stability which means that there exist directions of nonincrease for the value function w. Thus, relations (6.6),(6.7) can be interpreted as infinitesimal form of the dynamical programming principle.

6.3 Piecewise Smooth Value Function

The prevalent situation is the piecewise smooth construction for the value function w. In this case smooth components of the value function must satisfy Bellman-Isaacs (Hamilton- Jacobi) equation (6.3) and on surfaces of continuous contraction of these smooth components differential inequalities (6.6),(6.7) must hold. Realization of differential inequalities (6.6),(6.7) on surfaces of contraction is essential. There exist numerous examples demonstratingthat there exist piecewise smooth functions which satisfy Hamilton-Jacobi equation at points of their differentiability but these functions are not the value function because they don’t satisfy relations (6.6),(6.7).

For piecewise smooth functions directional derivatives and conjugate derivatives can be calculated in the framework of nonsmooth and convex analysis. Let us give corre- spondingformulas. Assume that for function w the followingequalities are valid in some neighborhood O_ε(x_∗, y_∗) of point (x_∗, y_∗)∈(0,1)×(0,1)

w(x, y) = min

i∈I max

j∈J ϕ_ij(x, y) = max

j∈J min

i∈I ϕ_ij(x, y) (6.9) w(x_∗, y_∗) =ϕij(x_∗, y_∗), i∈I, j ∈J

A Differential Model for a 2x2-Evolutionary Game Dynamics

A Diﬀerential Model for a

2 × 2-Evolutionary Game Dynamics

A. M. Tarasyev

WP-94-63 August 1994

A Diﬀerential Model for a

2 × 2-Evolutionary Game Dynamics

A. M. Tarasyev

WP-94-63 August 1994

Foreword

Contents

A Diﬀerential Model for a

2 × 2-Evolutionary Game Dynamics

A. M. Tarasyev

Introduction

1 The Model of Game Dynamics

1.1 Dynamics, Payoﬀs, Player’s Preferences

1.2 Properties of Dynamical System

1.3 Reduced Dynamical Systems

2 The Model of ”Short-Term” and ”Long-Term”

Payoﬀs

2.1 ”Short-Term” Payoﬀs

2.2 ”Long-Term” Payoﬀs

3 Nash Equilibria in Dynamical System

3.1 Feedback Controls, Trajectories of Dynamical System

3.2 Nash Equilibria

3.3 Stability Properties of Dynamical System

4 Construction of Nash Equilibria

4.1 Auxiliary Antagonistic (Zero-Sum) Games

4.2 Equilibrium Feedback Controls

5 Value Functions of Diﬀerential Games

5.1 Value Functions for Games with Inﬁnite Horizon

5.2 Properties of Value Functions

6 Value Functions and Minimax (Viscosity) Solu- tions of Hamilton-Jacobi Equations

6.1 Hamilton-Jacobi Equations

6.2 Generalized Derivatives, Diﬀerential Inequalities

6.3 Piecewise Smooth Value Function