Types of Testing

In the following, we describe the types of testing considered in this thesis. They are confor-mance testing, interoperability testing, and their combination.

2 Prerequisites 12

SUT

Conformance Testing User IUT

Conformance Testing

Figure 2.2: Conformance testing

2.2.2.1 Conformance Testing

Conformance testing is “testing the extent to which an Implementation Under Test (IUT) satisfies both static and dynamic conformance requirements” [36]. This means that confor-mance testing is generally used to check whether an implementation follows the require-ments stated in a standard.

In conformance testing, one IUT is tested with functional black-box tests to check if the IUT is conform to a standard. Figure 2.2 schematically depicts this test setup. The IUT is embedded in the SUT, which is a testing environment that also includes parts that are required by the IUT to provide its service or functionality to the user. Conformance testing usually requires the development and implementation of sophisticated testing tools, e.g., based on TTCN-3 [32]. Such tools support the simulation of the environment, which is needed for a proper execution of the IUT.

2.2.2.2 Interoperability Testing

Interoperability is assessed through interoperability testing, which is the “activity of proving that end-to-end functionality between (at least) two communicating systems is as required by the base standard(s) on which those systems are based” [36]. In interoperability test-ing, all participating systems are usually tested and assessed for interoperability against a qualified equipment [36], which is illustrated in Figure 2.3. AQualified Equipment (QE) is a reference implementation, which is a fully functional implementation of one or more standards. But the determination of a reference implementation for interoperability testing is difficult; because it needs to be assured that the reference implementation implements the standards correctly. However, each participating system implementation should be able to interoperate with all the others, not only with the reference implementation. Systems should rather be tested for interoperability against each other. Therefore, we updated the definition of interoperability testing described in [36]. We removed the QE and, therefore, avoid its determination. Each tested system in interoperability testing is called an EUT.

The collection of all EUTs is called the SUT [36]. Figure 2.4 depicts the interoperability test setup. Using this approach, interoperability testing provides a feasible way to assess if two or more systems are able to communicate or interoperate, i.e., to understand exchanged data.

13 2.2 Software Testing

User Equipment 

Under Test (EUT) Qualified 

Equipment (QE) User

Figure 2.3: Interoperability testing with a qualified equipment [36]

SUT

The EUTs interoperate via an abstractMeans of Communication(MoC). It is generally assumed that the communication services used between EUTs are compliant to underlying standards. Interoperability testing is usually driven manually because of the proprietary nature of end user interfaces.

Interoperability tests are applied at interoperability events, where vendors test the ability of their systems for interoperation with systems provided by other vendors and based on the same standards. The basis for each interoperability event is a previously agreed upon interoperability test specification. During the event, implementations of different vendors are plugged to each other and assigned to test sessions. The test sessions are executed in a parallel manner and have usually a specific time limit. Within this limit, it is attempted to execute as many applicable tests as possible. Examples for such interoperability events are the Plugtests^TM [30] events that are organized by ETSI. Depending on the concrete interoperability event, customers of the vendors and research partners are also allowed to attend for observing the test sessions.

2.2.2.3 Interoperability Testing with Message Checks

Comparing interoperability testing to conformance testing, each of them has its advantages and drawbacks. Conformance tests alone cannot guarantee system interoperability espe-cially for the application layer. Even if the IUT passes the conformance tests, it does not automatically prove that the IUT is interoperable with other systems implementing the same standard, because the standards may contain implementation options and leave space for in-terpreting requirement specifications, which can lead to interoperability problems [144].

The benefit of interoperability testing is that it can verify a correct service provision to end users. However, it may require a complex setup, e.g., aUniversal Mobile Telecommunica-tions System(UMTS) network including the configuration of all involved nodes and does not ensure adherence to standards.

2 Prerequisites 14

Figure 2.5: Interoperability testing with two EUTs and message checks

In our approach, we use interoperability testing in combination with conformance testing so that it is possible to check the conformance to a standard related to the interoperation and, in addition, to check if the EUTs are interoperable [119, 122]. This approach extends interoperability testing with the monitoring of the communication among the EUTs and updates the approach described by [36] by removing the QE as well as defining the SUT. In the remainder of this thesis, we call this combination of testing interoperability testing with message checks. This means that during the execution of interoperability tests, messages are recorded within test execution traces at (possibly standardized) interfaces between different EUTs by monitors to analyze the compliance of the recorded messages to the standards.

This allows the verification of the correctness of protocol procedures, while the assessment of interoperability takes place. Message checks also provide a basis for fault analysis. In contrast to traditional conformance testing, message checks assess requirements that are only related to the interoperation. An interoperability test setup combining interoperability tests with message checks is depicted in Figure 2.5. The end-to-end functionality is assessed from the end user points of view while the message checks take place at the intermediate interfaces. Although this approach is not a replacement for conformance testing, it offers an economic alternative to gain insights about the conformance of equipment participating in an interoperability test to a standard.

Interoperability tests with message checks are also described in the literature using differ-ent terminologies. The main idea of combining conformance testing with interoperability testing has been presented, e.g., by [140, 146, 148]. Viho et al. [146] provide a formal framework for interoperability testing. They present a general architecture of interoper-ability based on lower and upper testers as defined by the international ISO/IEC multipart standard 9646 OSIConformance Testing Methodology and Framework(CTMF) [74]. The CTMF standards define the upper tester and the lower tester strongly related with the OSI model. Interoperability testing with message checks can be used independent of the OSI model.

If interoperability tests with message checks are applied in interoperability events, the validation of standards can be performed in addition to the assessment of interoperability.

The results of the tests including interoperability issues as well as discrepancy of the applied standards are reported to the responsible technical committee. This feedback is then used to improve the standards.

15 2.2 Software Testing

Base Standard Base Standard

Test Specification Development Process Identification and Cataloging of 

Requirements 1.

Implementation Conformance  ( )

2. Statement (ICS) Specification