4.4. Test Suites
In the last section we presented the refinement of theTestModelconcept of the MBT meta-model for reactive component systems using the UML. In this section, we present the cor-responding refinement for theTestSuiteandTestCaseconcepts. Recalling that a test case represents a subset of the overall behavior defined by the test model, our test cases represent a subset of the behavior of our UML based test models. The behavior of these test models is described by several, parallel executed automatons which communicate using a method based or signal based communication paradigm. To create a formal definition for test suites and test cases using metamodels, we first want to introduce the involved design decisions:
Self-Contained and Reliant Tests
Test cases can be designed to include sufficient information so that the original test model is not needed. On the other side, they also can be designed to include as little as possible information, but requiring the original test model. In the first approach, the test cases are self-contained, which means that they include all behavior necessary to execute them without the need of the original test model. This kind of test cases are handy if we want to avoid giving our test models to other parties. However, these test cases usually need more disc (or memory) space, and changes to the test model may imply the regeneration of the whole test suite. In the second approach, the test case contains minimal behavior, but isrelianton the original test model. The expected behavior of the SUT has to be reconstructed by using the original test model. This kind of test cases consume less disc (or memory) space, and allow changes (to some degree) to the test model without the need to regeneration. However, it also forces us to reveal our test models to other parties.
Non-Determinism
Another design decision is to which extent test cases should handle non-deterministic choices of the SUT. If non-deterministic choices of the SUT are not handled, we may recognize a difference between the output of the SUT and the expected output of the test case, although the SUT is correct. A test case might falsely detect a fault, so an inconclusive outcome would be the correct verdict in this case (Fraser, 2007). One design approach is to expect specific decisions of the SUT, i.e. scheduling of components, and fail if the expectations are not fulfilled. Another approach is to annotate places in the test case where non-deterministic
behavior is possible to avoid a falsely detecting a fault. Finally, test cases can be designed to have a tree-like structure, so that alternative behavior can be specified for branching points.
Attribute Values
Our UML based test models can contain attributes in the UML Class diagrams. A design de-cision is to whether storeconcreteorsymbolicvalues for attributes in the test cases. Aconcrete value means that we expect a single correct outcome of the SUT for this attribute. In con-trast, asymbolicvalue defines a set of possible value outcomes. We can use symbolic values in two ways: First, we can use them to derive several test cases containing concrete values.
The concrete values could be chosen by enumerating the set, if possible, or by heuristics like choosing edge values (Utting and Legeard, 2006). Second, we can use symbolic values to allow non-determinism in the SUT (see above).
Concrete values improve the self-containment of test cases, so that consumers of these test cases do not need to derive concrete values in order to be able to execute them. Symbolic values allow to derive more than one test case by selecting concrete values for the symbols.
However, with symbolic values, the test cases are not self-contained anymore, and the con-sumer of the test cases has to perform additional steps to derive executable test cases.
Supporting all possible combinations of the above design decisions in one formalization would lead us to a complex metamodel, crowded with unnecessary details for some use cases. The formalization of reliant test cases, for example, does need less concepts than one for self-contained tests, since the missing information can be derived using the original test model. We therefore decided to create a basic formalization for reliant test cases, and show step by step how extensions to this basic formalization can lead us to self-contained test cases. In addition, we will show an extension for non-deterministic test cases.
The basic refinement of the MBT metamodel for UML is shown in Fig. 4.6. The most in-teresting elements are theUMLInputandUMLOutput, with their specializations for method and signal based inputs (stimuli) and outputs (observations). In addition, every UMLTest-Stepreferences an scheduled UML component through the relationscheduledComponent.
With this basic refinement, it is possible to express reliant test cases, which only contain a series of inputs and outputs.
Figure 4.7 shows a reliant test case for the Online Storage Example using a UML se-quence chart. This test case contains only signals between the involved components (User,
4.4. Test Suites 71 :Client
uiSendFile() requestSendFile()
transferApproved() transferCompleted()
:User :Server
fileContingent=10 transferCompleted=false
clientAuthenticated=false discFull=false
fileContingent=9
clientAuthenticated=true transferCompleted=true
Legend:
x=42 expected value 42 of attribute X
Figure 4.9.:A test case for the Online Storage Example containing signal inputs/outputs and attribute values. In contrast to the reliant version (Fig. 4.7), the original test model is not required for deriving attribute values during test execution.
industry makes intense use of automata8 for design, verification, and implementation of car components. The automaton is typically transformed into programming code, where the information of states and transitions might get lost. However, if the system preserves the information of states and transitions during runtime, we might interpret the current active states and transition as anotheroutputof the system. A reasonable way to test such a system using MBT is to include the information about states and transitions in test models.
Our approach to use UML Statecharts for test models allows testing such systems. With reliant test cases, we can derive the expected active states and transitions of the SUT dur-ing test execution. However, for a self-containdur-ing test, we need to include the structural information in our test cases. Fig. 4.10 shows our extension to the basic formalization of reliant test cases. We created two sub-types of the elementUMLOutputfor the configura-tion (def. 4.4) and active transiconfigura-tions. Both of them reference the respective UML types for states and transitions. With this addition, we can enhance our test case for the Online Stor-age example with structural information (Fig. 4.11). Test cases containing details using all the above extensions (signal input/output, attribute values and structural information) can finally avoid using the original test model, since all information is contained in the test case itself.
To broaden the view on the various presented metamodels, figure 4.12 shows an overview of the current state of our MBT refinement. We extended parts of the common MBT