In this section, we show the overall goal of runtime verification and examine different approaches in this research area. Verifying a systems means to show that it conforms to its specification (Do we build the product right?) [7, p. 101f.]. This is in contrast to validation where it is checked that a system provides the intended functionality (Do we build the right product?) [7, p. 101f.]. In general, the latter question needs to be answered by experts of the domain the system is intended for together with software engineers. Verification relies on the, ideally validated, requirements defined for the product (its specification). The verification itself can be done in many different ways, e. g., by code reviews or using tests (c. f. [91]). Of high interest for practical usage are approaches that can be automated. On a first sight, the verification of a system by proving its correctness would lead to the highest certainty of correctness. However, such a verification is generally hard to achieve, because of its complexity [72, p. 901].
To reduce the complexity, several possibilities exist. For example, the system can be abstracted by using models of it that focus on the properties to verify. These models can then be analyzed by a verification tool either statically or dynamically.
Static verification techniques, like model checking [18, 31], try to verify requirements off-line by analyzing the execution paths of a program. An overview of the complete model checking process is shown in Fig. 2.8 on page 22. First, the design to be verified is modeled in an appropriate modeling language. The requirements, i. e., the specification, is formalized using temporal properties. A (bounded) model checker tries to traverse the reachable states and verifies the given temporal properties. If a property is not fulfilled,
2.4 Runtime Verification
a counterexample in form of a sequence of states is generated [8]. A lot of research in this area tries to reduce the state space that has to be examined, to overcome the so called “state space explosion” [40, p. 15][72, p. 902].
In contrast to model checking, the goal of runtime verification (also called runtime monitoring) is not to verify all possible execution paths, but to verify requirements while the program is executed [19], thus providing more rigor in testing [16]. A detailed comparison of model checking and runtime verification can be found in [50]. Using runtime verification as an integral part in the development process combines the spe-cification and the implementation to together form the complete system. In [16] this is called Monitor Oriented Programming (MOP). By combining the specification and the implementation, see Fig. 2.9 on the next page, it is obvious, that a runtime verification approach is more coupled to the end product, since the more abstract specification must be aligned to the executed system in order to monitor it (see the directed edge between the system node and the Runtime Verification node in Fig. 2.9 on the following page). If the abstract specification would be inconsistent with the monitored part of the system, this would lead to an error during execution. This is in contrast to model checking, where the abstract model of the verified system is more decoupled from the concrete implementation, requiring complementary techniques to verify the correctness of an implementation [40, p. 15]. If a model checker finds a counter example, i. e., the specification does not hold, an additional simulation step is required to locate the error inside of the system, whereas during runtime verification the application can stop at the location where the error occurred. The main drawback of runtime verification, compared to model checking, is that the properties to verify must be executed either manually or by using specialized test drivers, whereas a model checker automatically evaluates all possible execution paths, leading to a higher degree of automation.
2.4.1 Specifying Properties
Essential to runtime verification are the properties to verify, since they define the spec-ification of a system. To be able to evaluate these properties, a formal specspec-ification language that can be computed must be used. Most of these languages include elements to express temporal properties, i. e., relations between different (sequential) states of a program. This is in contrast to an invariant, which only needs access to a single state [70]. Examples of such formalisms, which are described next, areLinear Temporal Logic (LTL) [70] and Extended Regular Expressions (ERE) [46]. LTL introduces oper-ators to express temporal relations between program states. For example, the operator X applied to a formula φ is used to express the fact that φ must hold in the neXt sate. Similar to invariants is the operator G, which requires a property φ to be valid Globally, i. e., in all states. Some authors further separate between future time linear logic (FTLTL) and past time linear temporal logic (PTLTL). The former states proper-ties about the future execution of a system, whereas the latter goes back in time. In [53]
Figure 2.8: Overview of Tasks and Artifacts for Model Checking (taken from [40])
Figure 2.9: Overview of Tasks and Artifacts for Runtime Verification (based on [40])
2.4 Runtime Verification
it has been shown that PTLTL has the same expressiveness as FTLTL.
The high popularity of regular expressions inside the developer community, lead to the usage of Extended Regular Expressions for runtime verification. Since a program execution trace can be seen as a string of states, regular expressions fit well for defining properties on them. EREs extend regular expressions by adding operators for com-plement (¬) and intersection (∧) to the three regular expression operators union (∨), concatenation (·), and repetition (∗) [46]. These two additional operators, especially the complement operator, ease the specification of patterns for traces that are not al-lowed [80]. As an example consider the safety property “it should not be the case that in any trace of a traffic light we see green and then immediately red at any point” [80].
Following [80] this can be expressed in “the natural and intuitive way” by the ERE
“¬((¬∅)·green·red·(¬∅)), where ∅ is the empty ERE (no words), so ¬∅ means “any-thing””.
2.4.2 Classification of Runtime Verification Techniques
Runtime verification can be done using different techniques. For example, it can be done by directly reacting on changes inside the running system or it can be done “post mortem” by analyzing recorded execution traces. In the following, we provide an excerpt of the relevant categories for our work, as they are described in [76] and [16]. Common to all techniques is that they need to inject some kind of monitoring code inside the running application. Though, they have different consequences for the runtime behavior of the monitored application, e. g., the introduced overhead, and differ in the possibilities to react on encountered violations of the specification.
On-line and Off-line
On-line runtime verification approaches verify properties while the system that is verified is executed, whereas off-line approaches analyze the recorded traces “post mortem” after the system has finished execution.
If runtime verification is done on-line, the user gets the benefit of immediate feed-back. The monitor can force the executed system to pause and can provide detailed information about the current state. The major drawback of on-line verification is the introduced overhead, since at relevant points during execution, the verification engine needs to evaluate the properties to validate. Depending on the size of the system and the specification, this can introduce noticeable delays.
To overcome possible delays in the execution of a system under test, off-line approaches minimize the introduced overhead by recording traces of the execution. Using an ad-equate logging technique, this leads to much shorter delays. These recorded execution traces are used after the system under test has executed to rebuild the program exe-cution and to verify the specified properties. If a property is violated, only the trace
information is present to the user, which makes the reconstruction of the failing situation more complicated than using an on-line approach where the system under test stops in the failing state.
Note that sometimes, e. g., in [76] on-line approaches are calledsynchronous(the vali-dation engine reacts on a received event and returns control to the monitored application after validation) and off-line onesasynchronous(the monitored system sends events and immediately continues execution afterwards).
Trace-Storing vs. non Storing
Obviously, two possibilities to work on execution traces exist. First, the complete trace can be stored to allow for a verification algorithm to compute on it. And second, a monitor can change the snapshot of the running system if it receives an event and discard this event after an incremental update of snapshot and required validation tasks have been done. The former approach is useful, if a validation algorithm has a reduced complexity when it has access to the complete trace. One example are algorithms to efficiently evaluate extended regular expressions [76]. The latter approach, which is not storing the complete trace but keeps a synchronized snapshot, reduces the overall consumption of memory and further allows to compute values without the need to loop through the trace.
Predictive vs. Exact Analysis
While the exact analysis of recorded traces can only detect concrete violations of prop-erties, algorithms that try to predict possible anomalies by analyzing valid execution traces exist also. Use cases for them are, e. g., the prediction of possible deadlock or datarace situations [76]. In this thesis, we focus on exact algorithms, therefore we leave out a more detailed discussion.