Partition-Oriented Test Generation
Stephan Weißleder
Humboldt-Universit¨at zu Berlin, Institut f¨ur Informatik Rudower Chaussee 25, 12489 Berlin, Germany
weissled@informatik.hu-berlin.de
Abstract:In this paper, we deal with input partition-oriented, model-based test suite generation. We extend existing control-flow-based coverage criteria, describe corre- sponding tool support, and present an exemplary evaluation based on mutation analy- sis. All explanations are substantiated by an example of a freight elevator.
1 Introduction
Testing is most important to validate system behaviour. Each test run consists of a se- quence of input stimuli that are fed into the system under test (SUT). The correspond- ing observable behaviour is then compared with the expected behaviour. In model-based testing, the expected behaviour is specified in models [UL06]. There is tool support for automatic generation and execution of model-based test suites [Tel, Lei, Wei].
In the field of software testing, the quality of test suites and the test effort are the most im- portant issues. Coverage criteria are widely accepted to measure the quality of test suites.
Various coverage criteria have been defined on the code level [Lig02, Bei90]. In model- based testing, many of them are transferred to the model level. Although many coverage criteria are connected by the subsumption relationship, many of them are unconnected.
Some unconnected criteria can be combined in the test generation process [WS08a].
Mutation analysis is often used to measure the fault detection ability of test suites [Par05, ABLN06, WS07]. The effect of mutation analysis on the fault detection ability of real faults is still under examination. There are case studies about the fault detection abilities of test suites for real faults, hand-seeded faults, and faults generated by mutation opera- tors [ABL05]. In this paper, we correlate mutation analysis and input partitions. We show that known control-flow-based coverage criteria are insufficient to cover all input partitions and that the fault detection ability of the corresponding test suites can be improved. For that, we extend existing control-flow-based coverage criteria and compare the results with mutation analysis. We exemplify our approach with a part of a freight elevator model.
This paper is structured as follows. The next section contains the reoccuring example of a freight elevator. In Section 3, the motivation for extending control-flow-based coverage criteria is presented. Section 4 contains an evaluation of corresponding test suites with mutation analysis. Conclusion and discussion are provided in Section 5.
2 Example: Freight Elevator
In this section, we present an example behaviour specification of a freight elevator. A freight elevator allows to add and remove weight, to press a button for a target floor, and also to select prohibited floors if authorized (e.g. by key or id card). Figure 1 shows part of a UML state machine describing the behaviour of a freight elevator. The events removeWeight(w) andaddWeight(w) trigger transitions whose effects change the actual weight of the freight elevator. The eventpressButton(b,r)triggers the movement of the ele- vator according to the pressed button (b) and the authority/rank of the operator (r). The lev- els above basement are accessible for all operators; the levels below are only accessible for operators with a rank higher thanminRank. Depending on the actual weight, the freight el- evator refuses to move (if overloaded), moves slowly, or moves fast (if empty/minWeight).
idle
button pressed
start moving
movefast
moveslow [actualWeight > minWeight]
[actualWeight = minWeight]
[actualWeight <= maxWeight]
[actualWeight > maxWeight]
pressButton(b,r) [(b <> currentFloor) and (b > basement or r > minRank)]
/setButton(b)
insertWeight(w) / addToWeight(w) removeWeight(w) / substractFromWeight(w)
reachFloor reachFloor
context: ElevatorControl::substractFromWeight(w : Integer) post: actualWeight = actualWeight@pre - w
context: ElevatorControl::addToWeight(w : Integer) post: actualWeight = actualWeight@pre + w
Figure 1: Behavioural model of a freight elevator.
3 Extending Control-Flow-Based Coverage Criteria
In this section, we extend control-flow-based coverage criteria. The foundation of this are our experiences while implementing improvements for the prototype ParTeG [Wei]
regarding the combination and evaluation of different coverage criteria [WS08a].
3.1 Problem
Mutation analysis is often used to measure the quality of test suites. There are case studies that support the importance of this measure [ABL05]. The set of sufficient mutation op- erators (cp. [OLR+96]) changes conditions of a correct SUT by exchanging relational op- erators, boolean operators, or absolute values. Therefore, mutation operators often create faults near the boundaries of input partitions (cp. [Mye79]). Hence, to detect a maximum number of mutants, test suites need to cover a maximum number of input partitions and
apply boundary-based coverage criteria to them [KLPU04]. We showed in [WS08a] that control-flow-based coverage criteria can be used to generate input partitions for abstract test cases. Already simple considerations show that the known control-flow-based cov- erage criteria are insufficient to enforce a coverage of all input partitions (leaving aside considerations about an infinite number of input partitions). In the following, we show two of these considerations, which concern the inconsistency and the incompleteness of control-flow-based coverage criteria for (in-)equations. We useMC/DC [CM94] exem- plarily for all control-flow-based coverage criteria. This criterion is widely accepted and its satisfaction is even required in safety-critical systems (standard RTCA/DO-178B).
Control-flow-based coverage criteria are based on predicate values and result values of logical expressions. We consider the conditionb<>currentFloorof the example’s guard condition [(b<>currentFloor) and (b>basement or r>minRank)]. The expression b<>currentFloor is semantically equivalent to the expression (b <currentFloor) or (b>currentFloor). However, coverage criteria like MC/DC create different sets of test goals (cp. [IL04]) for each expression (cp. Figures 2 and 3). Consequently, test goals of control-flow-based coverage criteria depend on the syntactic representation of logical expressions. Therefore, control-flow-based coverage criteria are considered inconsistent for different syntactic expressions of semantically equivalent expressions.
b <> currentFloor Result Test goal for MC/DC?
True True Yes
False False Yes
Figure 2: Test goals of MC/DC for the first inequation.
b < currentFloor b > currentFloor Result Test goal for MC/DC?
True True Contradiction No
True False True Yes
False True True Yes
False False False Yes
Figure 3: Test goals of MC/DC for the disjunction of both inequations.
Furthermore, control-flow-based coverage criteria do not necessarily create input parti- tions that cover the complete input value space. We consider the test goals for the expres- sion[(b<>currentFloor) and (b>basement)]as part of the afore used guard condition.
For the casecurrentFloor>basement, Figure 4 shows three corresponding test goals for MC/DC. For each test goal, just one test case is necessary to satisfy it. Consequently, only representatives of three input partitions are used. However, Figure 5 shows that there are four input partitions (A1-A4) for the casecurrentFloor>basement. A test case satisfying the first test goal (see Figure 4) can use representatives from input partitionA2orA4. Test cases for the second and the third test goal use representatives from partitionA1andA3, respectively. Consequently, there will be only one abstract test case for partitionsA2and
A4. From this it follows that one of these partitions, e.g. A2, will not be used for con-
crete test case generation. Consequently, mutants that result from changes inA2will be
undetected (unkilled). This insufficiency directly results from the inability of control-flow- based coverage criteria to deal with certain inequations. Note that avoiding inequations is no solution: the negation of an equation results in the same issues. Consequently, we propose to extend coverage criteria according to the used expressions.
b <> currentFloor b > basement Result Test goal for MC/DC?
True True True Yes
True False False Yes
False True False Yes
False False Contradiction No
Figure 4: Test goals of MC/DC for the pressed buttonb(for the casecurrentFloor>basement).
button currentFloor
basement
A1 A2 A3 A4
Figure 5: Input partitions for the pressed buttonb(for the casecurrentFloor>basement).
3.2 Solution
We presented former algorithms of automatic test case generation based on UML and OCL in [WS07, WS08b]. The current version of the test generation algorithm first derives test goals from the selected coverage criterion (cp. [WS08a]). The basic search algorithm col- lects each guard condition on potential paths of the state machine. It searches backwards, evaluating all postconditions, and transforming the guard conditions in order to derive in- put value partitions (cf. Figure 6). The solution to the presented problems is to resolve the inequations with ”<>” to two inequations with ”<” and ”>”, respectively, and create dif- ferent test goals for them. The same solution is applied to the negation of equations. This results in a superset of the actually needed test goals for coverage criteria like MC/DC.
We call this extension of coverage criteriaequation-extension (eq-ext). For our example, the test generation algorithm for equation-extended coverage criteria defines 4 test goals according to the input partitions shown in Figure 5.
test sequences with undefined input parameters
influence of input para- meters on system attributes
conditions about system attributes
conditions about input parameters to satisfy conditions about system
attributes
test sequences with input parameters close to the partition boundaries derived
from conditions
Step 1 Step 2 Step 3
test sequences with undefined input parameters
Figure 6: Basic test suite generation algorithm.
4 Evaluation
In this section, we present an evaluation of the equation-extended coverage criteria. We used mutation analysis to compare the fault detection ability of test suites that satisfy MC/DC, Multiple Condition Coverage (MCC), and the corresponding equation-extended coverage criteria, respectively. We combined the control-flow-based coverage criteria with random value selection and value selection according to Multi-Dimensional [KLPU04].
Figure 7 shows the corresponding comparison. As a result, test suites that satisfy equation- extended coverage criteria detect (kill) more mutants than the test suites that satisfy the original control-flow-based coverage criteria. This is no weakness of the selected coverage criterion MC/DC - we get the same results for MCC. The differences between the num- bers of detected mutants may seem small. However, there are studies that underline the importance of the last few percent of coverage satisfaction (in this case: mutant detection coverage) [ABLN06]. Consequently, each small improvement is feasible and desirable.
Covera
Coverage Criteriage Criteria RaRandomndom MuMulti-Dilti-Dimensmensionalional MC/DC
MC/DC 33 / 41 mutants (9 test cases) 40 / 41 mutants (17 test cases) MC/DC (eq-ext)
MC/DC (eq-ext) 34 / 41 mutants (10 test cases) 41 / 41 mutants (19 test cases) MCC
MCC 33 / 41 mutants (10 test cases) 40 / 41 mutants (19 test cases) MCC (eq-ext)
MCC (eq-ext) 34 / 41 mutants (12 test cases) 41 / 41 mutants (23 test cases) Figure 7: Mutation analysis of coverage criteria.
5 Conclusion and Discussion
The mentioned problems show the need for investigating coverage criteria and the pre- sented solution is quite intuitive. The equation-extension of each coverage criterion re- sults in a higher fault detection ability of the corresponding test suite. Furthermore, the extension is automated and results only in a few more test cases.
However, there is still the need for discussion. For instance, the selection of MC/DC can be discussed. However, even the use of Multiple Condition Coverage (MCC) results in the same conclusions. Furthermore, the selected boundary-based coverage criteria can be discussed. They just require value selection inside of input value partitions [KLPU04].
Myers proposes to select concrete values inside and outside of value partitions [Mye79].
Myers’ approach could help in the presented example. In cases when there are two ne- glected adjacent value partitions, however, this would not help. Furthermore, it would lead to too many test cases because concrete values around boundaries of two partitions would be selected twice. Myers’ approach also bears difficulties concerning the expected system behaviour for test input values that are outside of input value partitions. We al- ready stated that avoiding the mentioned type of inequation (”<>”) would not solve the presented problems. The same problems arise for negations of equations (”=”), which are necessary to satisfy coverage criteria like, e.g. MC/DC. Finally, one should also discuss if there are more possible and feasible extensions of control-flow-based coverage criteria.
Acknowledgements.This work was supported by grants from the DFG (German Research Foundation, research training group METRIK).
References
[ABL05] J. H. Andrews, L. C. Briand, and Y. Labiche. Is mutation an appropriate tool for testing experiments? InICSE ’05, pages 402–411, New York, NY, USA, 2005. ACM.
[ABLN06] J. H. Andrews, L. C. Briand, Y. Labiche, and A. S. Namin. Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria. Software Engineering, IEEE Transactions on, 32:608–624, 2006.
[Bei90] B. Beizer.Software Testing Techniques. John Wiley & Sons, Inc., 1990.
[CM94] J. J. Chilenski and S. P. Miller. Applicability of Modified Condition/Decision Coverage to Software Testing. InSoftware Engineering Journal, 1994.
[IL04] I-Logix.Rhapsody Automatic Test Generator, Release 2.3, User Guide, 2004.
[KLPU04] N. Kosmatov, B. Legeard, F. Peureux, and M. Utting. Boundary Coverage Criteria for Test Generation from Formal Models. InISSRE ’04, pages 139–150. IEEE, 2004.
[Lei] Leirios. LTG/UML. http://www.leirios.com.
[Lig02] P. Liggesmeyer. Software-Qualit¨at: Testen, Analysieren und Verifizieren von Software.
Spektrum Akadamischer Verlag, 2002.
[Mye79] G. J. Myers.Art of Software Testing. John Wiley & Sons, Inc., New York, USA, 1979.
[OLR+96] A. J. Offutt, A. Lee, G. Rothermel, R. H. Untch, and C. Zapf. An experimental determi- nation of sufficient mutant operators. ACM Transactions on Software Engineering and Methodology, pages 99–118, 1996.
[Par05] A. Paradkar. Case studies on fault detection effectiveness of model based test generation techniques. InA-MOST ’05, pages 1–7, New York, NY, USA, 2005. ACM Press.
[Tel] Telelogic. Rhapsody Automated Test Generation. http://www.telelogic.com.
[UL06] M. Utting and B. Legeard. Practical Model-Based Testing: A Tools Approach. Morgan Kaufmann Publishers Inc., 2006.
[Wei] S. Weißleder. ParTeG (Partition Test Generator). http://parteg.sourceforge.net.
[WS07] S. Weißleder and B.-H. Schlingloff. Deriving Input Partitions from UML Models for Automatic Test Generation. InLNCS Volume on Models in Software Engineering (MoD- ELS 2007), 2007.
[WS08a] S. Weißleder and B.-H. Schlingloff. Quality of Automatically Generated Test Cases based on OCL Expressions. InICST, April 2008.
[WS08b] S. Weißleder and D. Sokenou. Automatic Test Case Generation from UML Models and OCL Expressions. InTesting of Software - From Research To Practice (associated with Software Engineering 2008), February 2008.
