Summary of the Approaches

To validate the model, we must validate all elements and their relationships. This means that for a specific quality model, we must validate artifacts, entities, stake-holders, activities, quality factors and impacts. Of these, the core of the model are

46

4. Summary of Results quality factors and impacts. Therefore, in the following, we largely focus on these two elements during the evaluation³.

In empirical software engineering, researchers often differentiate three general ap-proaches (see e.g. [WRH⁺12]): surveys, case studies, and controlled experiments.

In the following, we describe each method on its own, providing a description how this technique can be applied to validate a quality factor.

4.2.2.1. Validation of Quality Factors in Expert Interviews

The first approach to validate quality factors are surveys, e.g. in the form of expert interviews⁴. Surveys allow collecting information from people describing their knowledge or behaviour [WRH⁺12]. Given a certain context, such as a domain, a field, or a specific company, we can employ interviews to validate a large set of quality factors by collecting this information from adequate, knowledgable stakeholders.

Our approach consists of four steps:

1. Select experts: The first step, subject selection, aims at selecting a representative set of experts for each of the potential stakeholders.

2. Inspect artifacts & entities: The second step validates the artifacts and entities, e.g. by iterating through project artifacts. Since in practice this can reveal a plethora of artifacts and entities (see e.g. [MWL⁺12] for such studies), reduction of artifacts together with local experts can clarify the scope and greatly reduce the required effort.This involves questioning which artifacts exist, which artifacts are in fact used, and how they relate to each other.

3. Inspect stakeholders & processes: The third step aims at understanding the role and process model of the project. Therefore, the goal of this step is to develop a complete list of stakeholders and their most important activities.

4. Validate impacts: The goal of the fourth step is to validate the quality factors and their impacts. In particular, in this step, we must evaluate the following questions:

• Are all relevant quality factors and their impacts present? If not, we need to extend the quality model and consider also extending the survey.

• Are all quality factors and their impacts relevant? A quality factor and its impacts arerelevant if it measurably affects the activities of a stakeholder.

In case we have irrelevant factors, we must remove them from the model.

• Are all quality factors and their impacts accurate? A quality factor and its impacts areaccurateif the presence of the quality factor at the entity has the described effect on the activities. When answering this question, we could find counter-examples, in which an impact does not hold. In these cases, we can decide to either remove the quality factor completely, or correct it by splitting up the entities, activities, or even the quality factors. For an example of such an analysis, see the UI design details excerpt in Publication A.

Please note that this process may not necessarily be executed sequentially, but rather iteratively. For example, Step 3 could reveal new stakeholders, which need new expert selection etc.

3 Even though stakeholder analysis itself can be a difficult and sometimes cumbersome activity.

For further details, Pacheco and Garcia [PG12] summarize the state of the art in stakeholder identification in their systematic literature review.

4 Surveys can come in the form of interviews or questionnaires [WRH⁺12]. Even though each of these two methods enables to apply a different set of techniques, the advantages and disadvantages for validating quality factors largely apply for both. In the following, we focus on interviews, since it allows more fine-grained discussion and analysis.

4.2. RQ 2: How Can We Create Valid Quality Models?

4.2.2.2. Validation of Quality Factors in Case Studies

A case study investigates a certain study object in real life in its original context and in detail, usually applying multiple techniques [WRH⁺12]. Consequently, one can apply case studies in a constructive manner to understand a quality factor in context or discover new quality factors. Again, the approach follows the elements of the quality model:

1. Select a field and one or multiple cases: As a first step, we need to select the field under analysis and cases, where the data (see Step 4) is available.

2. Inspect artifacts & entities: (See expert interviews).

3. Inspect stakeholders & processes: (See expert interviews).

4. Validate or mine quality factors: In this step, we analyze which problems or defects in the case originated from artifacts and caused negative consequences for corresponding activities. To observe these, one can look through the produced artifacts, e.g.:

• Review protocols that indicate the effort that was invested during QA of the RE artifacts

• Incorrect test cases or incorrect test results that show that the test engineer misunderstood the RE artifacts

• Requirements change requests or defects in bug tracking systems that can be traced back to RE artifacts, to understand defects in the RE artifact that are discovered during development or further activities

• Changes that have been performed on the RE artifacts in order to under-stand maintenance efforts

Lastly, in line with the suggestions by Wohlin et al. [WRH⁺12], we suggest to triangulate the results with further sources, such as interviews.

4.2.2.3. Validation of Quality Factors in Controlled Experiments

As a third approach, we can analyze quality factors and their impacts in (controlled) experiments. In experiments, we systematically manipulate individual variables to understand the impact of a certain treatment on further variables [WRH⁺12].

Consequently, we can apply experiments to analyze the hypotheses we have about a single quality factor and its impacts on activities. Following the terminology of [WRH⁺12], the experiment design is as follows:

Define treatment: The treatment consists of the different aspects of the quality factor.

Therefore, we usually compare specifications where a quality factor is present (Treatment group, e.g. passive voice requirements) against those where a factor is not present (Control group, e.g. active voice requirements).

Select dependent variables and metrics: The dependent variables are metricsV of the activities (e.g. understanding a specification), according to the impact. This means, that for impacts on effectiveness, we measure the number of errors made, for efficiency we measure the time that is required to execute the activity.

Select subjects: Since the quality model builds upon the quality in use, and the dependent variables are the number of errors or the time required for a certain task, we need human subjects to execute this task. We argue that, as far as possible, the subjects should resemble the real stakeholders of the artifact to prevent confounding factors skewing the resulting data.

Define hypothesis: For the independent variableV and the groups {Treatment, Con-trol}

H0:V(Treatment)≤V(Control) HA:V(Treatment)> V(Control)

48

4. Summary of Results

Select data analysis: Hypothesis testing can then be performed according to standard testing procedures (see e.g. [WRH⁺12] or [WMMY12]).

All in all, experiments allow analyzing single quality factors in detail.