Theoretical Background
3.4 markov chains
Stochastic processes capture the probabilistic notion of a dynamic system evolving in time. Classical time-discrete dynamical systems are represented by mapsfthat evolve their next statexn+1 =f(xn)deterministically depend-ing solely on their respectively current statexn(the very definition of state).
Arguably, a Markov chain is what comes closest to a map for time-discrete stochastic processes, as its state Xn+1 depends solely on the last state Xn (rather than the whole past of the process).
3.4 markov chains 25
3.4.1 Markov chains on countable spaces
Definition3.23([123,127,128]). LetSbe a countable set, and letX={Xn:n∈N} be a discrete-time stochastic process with values inS. Such a processXis aMarkov chainif its random stateXn for eachn > 0depends on the past only through the previous stateXn−1 (Markov property), that is
P{Xn=j|X0 =i0,. . .,Xn−1 =in−1}=P{Xn=j|Xn−1 =in−1} for alln > 0,j,i0,. . .,in−1 ∈S. For allx,y∈S, letP(x,y) denote the transition probability from statexto statey,
P(x,y) =P{Xn=y|Xn−1=x}, such thatP(x,y) > 0 andP
z∈SP(x,z) = 1 for x,y ∈ S. Recursively define the n-step transition matrixas
Pn(x,z) = X
y∈S
P(x,y)Pn−1(y,z)
withP0 defined as the identityP0(y,z) =δyz, so that for allx,y∈S,Pn(x,y) = P{Xn=y|X0=x}. The probability measure µ(x) = P{X0 =x} is the initial distributionof the chain.
Theorem 3.24 ([128]). Let S be a countable space, and let µ : S → [0,1]be an initial probability measure onS, and further letP(x,y) be transition probabilities such thatP(x,y) >0andP
z∈SP(x,z) = 1forx,y∈S. Then there is a Markov chainXnsuch that
P{Xn=y|Xn−1 =x,. . .,X0 =x0}=P(x,y) n > 0,x,y,x0∈S. andP{X0=x0}=µ(x0)forx0∈S.
Theorem3.25([123, p.8]). LetSbe a countable space, letS0be a Polish space, let f:S×S0 → Sbe measurable and letXnbe a time-discrete stochastic process onS such that
Xn=f(Xn−1,Yn), n > 0
withY1,Y2,. . .iid random variables with values inS0independent ofX0. Then,Xn is a Markov chain with transition probabilitiesP(x,y) =P{f(x,Y1) =y}.
3.4.2 Markov chains on general spaces
Theorem 3.26 ([128]). Let (S,S) be a Polish space (in fact,S could be any space endowed with a countably generatedσ-algebraS). Letµbe a probability measure on S, and letP(x,B) be a probability kernel for allx ∈ S,B ∈ S. Then there exists a discrete-time stochastic processXnsuch that forn > 0,B0,. . .,Bn∈S
P{X0 ∈B0,. . .,Xn∈Bn}
= Z
x0∈B0
· · · Z
xn−1∈Bn−1
µ(dx0)P(x0,dx1)· · ·P(xn−1,Bn) andP{X0∈B0}=µ(B0)forB0∈S.
26 stochastic processes
Definition 3.27 ([128]). Such a discrete-time stochastic process Xn is called a Markov chain on (S,S) with transition probability kernel P(x,B) and initial dis-tributionµ. Recursively define then-step transition probability kernelas
Pn(x,B) = Z
S
P(x,dy)Pn−1(y,B) x∈S,B∈S
with P0(x,B) = δx(B), so that for all x,y ∈ S we have P{Xn=y|X0=x} = Pn(x,y).
3.4.3 First passages and returns
The following definitions are formulated for Markov chains on general Polish spaces. The definitions easily transfer to Markov chains on a count-able space by regarding single elements y instead of Borel sets B, where applicable.
Definition 3.28 ([127, 128]). Let Xn be a discrete-time stochastic process on a Polish spaceS, and letB∈S. Theoccupation numberηBis the random number of (possibly infinite) visits ofXtoB:
ηB= X∞ n=1
δXn(B).
The event that the process visits the setB∈Sinfinitely often after starting atx∈S has the probability
Q(x,B) =P{ηB=∞|X0 =x}.
Forn > 0define thefirst-passage-time probability (kernel)fn :S×S→ [0,1]
from state x∈ S to set B∈ S as the probability thatnis the smallest ifor which Xi∈Bgiven thatX0=x:
fn(x,B) =P{Xn∈B,Xn−1 ∈/ B,. . .,X1 ∈/B|X0=x}
with f1(x,B) = P(x,B). Furthermore, for n > 0, let Fn : S×S → [0,1]be the probability (kernel) that the process starting at x ∈ S visits a set B ∈ S between times1andn, inclusive:
Fn(x,B) = Xn i=1
fi(x,B).
Thefirst return time τBis the random time after0when the process first entersB (or when it first returns toB, ifX0 ∈B):
τB=min{n > 0:Xn∈B}.
Given that the process starts inx, the probability distribution ofτBis P{τB=n|X0 =x}=fn(x,B).
Forx∈SandB∈S, define thereturn probabilitiesas the probability to return to B(in finite time) when starting inx:
L(x,B) =P{τB<∞|X0 =x}= X∞ n=1
P{τB=n|X0 =x}=F∞(x,B)
3.4 markov chains 27
Definition3.29([128]). LetXnbe a Markov chain on a Polish spaceSwith n-step transition probability kernelPn. Define the auxiliary probability kernelU:S×Sas
U(x,B) = X∞ n=1
Pn(x,B) (x∈S).
We have for all x ∈ S,B ∈ S the expected number of returns to B after starting atxasE[ηB|X0 =x] =U(x,B).
3.4.4 Irreducibility
Irreducibilityof a Markov chain guarantees that the chain eventually visits all regions of its state space:
Definition 3.30 ([128]). Let S be a Polish space. A Markov chain Xn on S is ϕ-irreducible if there is a measureϕonSsuch that for allx∈S,B∈S:
ϕ(B)> 0⇒L(x,B)> 0.
Theorem3.31([128]). LetXnbe a Markov chain on a Polish spaceS. The following statements are equivalent: X is ϕ-irreducible. ϕ(B) > 0 ⇒ U(x,B) > 0 for all x∈S,B∈S.
Theorem3.32([128]). LetXbe aϕ-irreducible Markov chain on a Polish spaceS for some measureϕ. Then there exists an “essentially unique maximal” irreducibil-ity measureψonSsuch that
1. Xisψ-irreducible.
2. ψ(B) =0⇒ψ{x∈S:L(x,B)> 0}=0for allB∈S.
3. ψ(S\B) =0⇒B=B0∪N:ψ(N) =0,P(x,B0) =1for allx∈B0 (B0 is absorbing).
Definition 3.33 ([128]). A Markov chainX isψ-irreducible if it isϕ-irreducible for some measure ϕ and if the measureψ is a maximal measure according to the preceding theorem. Define the family of sets of positiveψmeasure as
S+={B∈S:ψ(B)> 0}.
The setS+ is the same for different maximal irreducibility measures, and hence,S+ is well-defined. [128] For a countable state space S, the maximal irreducibility measure is the counting measure.
3.4.5 Transience and recurrence
Recurrence is a weak notion of stability of a Markov chainX. A recurrent chainXvisits every set of positive measure infinitely often. Contrarily, a tran-sient chain visits bounded sets only a finite number of times, and eventually leaves any such set. Specifically, we consider recurrence and transience in terms of the occupation number random variableηB.
28 stochastic processes
Definition3.34([128]). LetXbe a Markov chain on a Polish spaceS. A setB∈Sis uniformly transientif there exists an upper boundM <∞such thatU(x,B)6M for allx∈B. A setB∈SisrecurrentifU(x,B) =∞for allx∈B. A setB∈Sis transientif there is a countable cover ofBby uniformly transient sets.
Definition3.35([128]). LetXbe aψ-irreducible Markov chain on a Polish spaceS. The chainXisrecurrentif every setB∈S+ is recurrent. The chainXis transient ifSis transient.
Theorem3.36([128]). LetXbe aψ-irreducible Markov chain on a Polish spaceS. ThenXis either recurrent or transient.
Definition3.37([128]). LetXbe a Markov chain on a Polish spaceS. A setB∈S isHarris recurrentifQ(x,B) =1for allx∈B. The chainXisHarris recurrent if it is ψ-irreducible and every set B ∈ S+ is Harris recurrent (or equivalently, it holds thatL(x,B) =1for allx∈S).
Hence, Harris recurrence is stronger than recurrence: Theexpectednumber of visits to a recurrent set is infinite, while a Harris recurrent set is visited infinitely oftenalmost surely.
Theorem 3.38 ([128]). Let X be a recurrent Markov chain on a Polish space S. Then
X=H∪N
with an absorbing and nonempty setHand a transient setNwithψ(N) =0. Every subset ofHinS+ is Harris recurrent.
The theorem implies that the restriction of a recurrent chainXtoHdiffers to the original chain only by a ψ-null set. At the same time, the restriction to H yields stronger stability results in terms of Harris recurrence. For a countable state spaceS, the setNis empty: a recurrent chain on a countable state space is also Harris recurrent.
3.4.6 Stochastic recursive sequences
Stochastic recursive sequences generalize the notion of Markov chains to discrete-time stochastic processes. Rather than by a sequence of iid random variables, they are driven by an arbitrary random sequence:
Definition 3.39 ([129, p. 507]). Let S and S0 be two Polish spaces. Let ξn be a sequence of random elements on S0. Let f be a deterministic measurable function S×S0 → S. A time-discrete stochastic process Xnon Sis a stochastic recursive sequence driven by the sequenceξnifXnsatisfies the relation
Xn=f(Xn−1,ξn), n > 0 withX0 independent ofξn.
As Borovkov [129, p. 17] points out, each Markov chain is a stochastic recursive sequence driven by iid ξn. Furthermore, there is a notion of reno-vatingevents of the processXnfrom which on only the driving sequenceξn determines the evolution of the process rather than the states Xnbefore the event. The notion of renovating events is weaker than renewals, but never-theless allows to infer long-term behavior and ergodic properties. [129]