While there is ample research on model checking stochastic models against PCTL and CSL formulae, there is a dearth of methods and techniques supporting the interpretation of the model checking results. In this dissertation we, concurrently with other researchers, are addressing this issue by defining the notion of a
coun-2.5. Conclusion 13
terexample for various stochastic models, describing methods to efficiently compute them, and discussing their representation and interpretation.
Chapter 3
Dependability Analysis
Contents
3.1 Markov Chains . . . . 15 3.2 Markov Decision Processes. . . . 18 3.3 Stochastic Model Checking . . . . 21
3.1 Markov Chains
Markov chains are used to model dependability and performance aspects of systems by describing the system behaviour as a stochastic process. System states and tran-sitions are associated with probability and time consumption. The simplest kind of Markov chains are discrete-time Markov chains(DTMCs). A DTMC can be con-sidered as a probabilistic transition system consisting of states and the transitions between them. The discrete time model means here that the system fires exactly one transition every time tick. Each transition (s, s0) is labelled with a numerical value referred to as transition probability. This indicates the probability of this transition to be fired when the current state of the system is s. Atomic proposi-tions are used to formalise characteristics of the states. In other words, atomic propositions express simple known facts about the system states. The set of atomic proposition is denoted asAP. A DTMC is formally defined as follows:
Definition 1. A labelled discrete-time Markov chain (DTMC) D is a tuple (S,ˆs,P,L), where S is a finite set of states, sˆ∈S is the initial state, P:S×S−→
[0,1]is a transition probability matrix, satisfying that for each states, P
s0∈S
P(s, s0) = 1, and L:S−→2AP is a labelling function, which assigns to each state a subset of the set of atomic propositions AP.
For any state s, we interpret L(s) as the set of valid propositions in the state.
For each pair of states s and s0, P(s, s0) gives the probability to move from s to s0. A move from s to s0 is possible if and only if P(s, s0) > 0 holds. In this case we call (s, s0) a transition. succ(s) is the set of successor states of s, i.e., succ(s) = { s0 ∈S| P(s, s0)>0 }. A state sis called absorbing if P(s, s) = 1 and consequently,P(s, s0) = 0 for all other states s0 6=s.
16 Chapter 3. Dependability Analysis
DTMCs are widely used in the modelling and analysis of stochastic systems based on a discrete time abstraction because of their conceptual simplicity. For mod-elling and verifying ofreal-timepropertiescontinuous-time Markov chains(CTMCs) can be used. While each transition of a DTMC corresponds to a discrete time step transitions of CTMCs occur in dense real time. A CTMC is formally defined as follows:
Definition 2. A labelledcontinuous-time Markov chain (CTMC)Cis a tuple (S,s,ˆ R,L), where S is a finite set of states, ˆs∈Sis the initial state, R:S×S−→
R≥0 is a transition rate matrix and L : S −→ 2AP is a labelling function as in Definition 1.
The transition probability matrixP, which we defined for DTMCs, is replaced by a transition rate matrix R. For any pair of states sand s0,R(s, s0) is the time rate for moving from stos0. More precisely, R(s, s0) is the parameter of a negative exponential distribution describing the probability to move from s to s0 within t time units. A move from s to s0 is possible if and only if R(s, s0) > 0. In this case we call (s, s0) a transition. succ(s) is the set of successor states of s, i.e., succ(s) = { s0 ∈ S | R(s, s0) > 0 }. We use Λ(s) to refer to the total exit rate of s, i.e. Λ(s) = P
s0∈SR(s, s0). A state sis called absorbing if Λ(s) = 0. For simplicity matter we assume that a CTMC does not contain any self loops, i.e. R(s, s) = 0 for all states.
We use the notion of aMarkov chain as a generic term for either a DTMC or a CTMC.
Example 1. Figure 3.1 illustrates a DTMC modelling a system which becomes ready (state s1) after finishing some initialisation routine (state s0). When the system is ready, it may receive a task to process (state s2) and return to be ready after finishing to process the task. The system may fail at any state (states s3, s4 ands5) causing a system crash (states6). Figure3.2illustrates a CTMC modelling a system which is very similar to the one modelled by the DTMC given in Figure3.1.
The only difference here is that the time is interpreted to be continuous.
Figure 3.1: An example for a DTMC. Figure 3.2: An example for a CTMC.
3.1. Markov Chains 17
3.1.1 Paths and Probability Measures
Intuitively, a path is a concrete execution of the system, i.e., it is encompassing a sequence of state transitions. Systems considered in the domain of model checking are usually reactive systems. A reactive system does not terminate and deliver some computational results. Instead, it maintains an ongoing activity responding to stimuli from the system’s environment. Hence, paths are assumed to be infinite.
On the other hand, we often need to refer to finite path prefixes. Hence we use the notions of infinite path, or simply path, for a full path and finite path, for a finite path prefix. Formally, letD= (S,s,ˆ P,L) be a DTMC, then
Definition 3. A (infinite) path through D is an infinite sequence s0, s1, s2. . . with, for all i, si ∈ S and P(si, si+1) > 0. A finite path is a finite prefix of an infinite path.
For a finite or an infinite path σ = s0, s1, . . ., we use len(σ) to denote the length of σ determined by the number of states that are touched along σ. Note that for an infinite path π, len(σ) is equal to ∞. For a natural number k such that 0 ≤ k < len(σ), σ[k] refers to the (k+ 1)-st state of σ, namely sk. With σ(k) we denote the k-th prefix ofσ. This is the prefix of σ consisting of the firstk transitions, namelys0, . . . , sk. The termf irst(σ) refers to the first state in σ. Ifσ is finite, then last(σ) denotes the last state ofσ. We useP athsD to denote the set of all infinite paths in D. For any state s, P athsD(s) refers to the sets of infinite paths which start at s.
For a DTMC D = (S,ˆs,P,L) and a state s0 ∈ S, the probability of paths originating at s0 is measurable by a probability measure P rDs0. The underlying σ-algebra is formed by the cylinder sets which are induced by finite paths in D starting at s0. Each finite path s0, . . . , sn induces a cylinder set cyl(s0, . . . , sn) = {σ ∈P athsD(s0)|σ(n)=s0, . . . , sn}. The probability of this cylinder set is defined as follows:
P rDs0(cyl(s0, . . . , sn)) =
n−1
Q
i=0
P(si, si+1) (3.1) WithP rD we denote to the probability measure of paths inDstarting at the initial state, i.e. P rD =P rDˆs.
For a CTMC C = (S,ˆs,R,L), a path through C comprises the time delays in each state along the path in addition to the sequence of state transitions:
Definition 4. A (infinite) path through C is an infinite alternating sequence s0, t0, s1, t1, s2. . . with, for all i, si ∈ S, R(si, si+1) > 0 and ti ∈ R≥0. A finite path is a finite prefix, ending with a state, of an infinite path.
All notions for paths in DTMCs, defined above, are defined in the same way for paths in CTMCs. Further, we define σ@t for a finite or an infinite pathσ in a CTMC and a time pointtas the state occupied by the system at time pointtwhen the system runs along the path σ.
18 Chapter 3. Dependability Analysis
The probability of paths in C starting at some state s0 ∈S is measurable. We define the probability measure P rCs0 on P athsC(s0) following [BHHK03]. Let s0, . . . , sn be a sequence of states of C such thatR(si, si+1)>0 for alli∈ {0, . . . , n− 1}. Further, let I0, . . . , In−1 be a sequence of non-empty intervals in R≥0. Let cyl(s0, I0, . . . , In−1, sn) denote the cylinder set consisting of all paths of the form σ =s0, t0, . . . , tn−1, sn. . .withti ∈Ii for alli∈ {0, . . . , n−1}. The measureP rCs0 is defined on the smallest σ-algebra on P athsC(s0) formed by all cylinder sets for all state sequences starting at s0 and sequences of time intervals as described above.
Intuitively, P rCs0(cyl(s0, I0, . . . , In−1, sn)) gives the probability of visiting the states s0, . . . , snin the indicated order and staying at each statesi a period of timeti∈Ii
for 0 ≤ i < n. The measure P rCs0 is formally defined by the following induction, with P rCs0(cyl(s0)) = 1, as: transi-tion outgoing from state sn in the interval In. The probability that the tran-sition (sn, sn+1) will be chosen is given by R(sΛ(sn,sn+1)
n) . Altogether, the term
R(sn,sn+1) Λ(sn) ·R
In
Λ(sn)·e−Λ(sn)·t·dtgives the probability of firing the transition (sn, sn+1) within the interval In. Like for DTMCs, we use the abbreviationP rC for the prob-ability measure of paths in C starting at the initial state, i.e. P rC =P rCsˆ.