Research Gap

In the following, we summarize the research gaps. We look at quality definitions first, before discussing QA afterwards.

2.5.1. Research Gap: Quality Definitions for RE artifacts

To summarize this section, there are various viewpoints onto quality, namely:

Generic theories: Various authors have worked on approaches to define RE quality in a systematic way. While these approaches are helpful to define quality factors, their generic approach does not take into account that requirements engineering is just a means and not an end. For one example, Pohl [Poh93]

assumes that formalization is a goal of the RE process. We disagree with this view, since a formal model that is never used nor understood has no purpose in a software engineering project.

Standards and generic quality characteristics: Unfortunately, the current standards for RE quality are rather a list of characteristics than a structured approach towards quality. Although IREB contains some hints towards thinking about RE artifact usage, this is not yet reflected in the quality model. In particular, the strong differences between the currently most relevant standards show how arbitrary the existing characteristics are, and that we do not have a systematic approach for deciding which characteristics are relevant, when and why. In addition, the meaning of various characteristics is too abstract and unclear.

Specific quality factors: The same holds for approaches for specific, more concrete quality factors, such asnegative statements. Existing approaches are defining lists of characteristics without an adequate concept of quality as a foundation.

In consequence, the reasoning behind quality factors remains unclear, which makes the question whether or not to make use of the quality factors dubious in practice.

All in all, we have generic theories, sets of characteristics, and concrete quality factors. While some approaches provide a holistic view on quality, none of these takes the usage of the RE artifacts and the context into account. Therefore, we still have a limited understanding of what high quality RE artifacts are in its usage context.

2.5.2. Research Gap: QA for RE Artifacts

In order to assure that RE artifacts are of high quality in practice, various QA techniques can be applied. A detailed analysis of existing methods for QA of RE artifacts can be found in Publication G. As explained, we can differentiate into constructive and analytical, and automatic and manual approaches. Of these, automatic analytical methods have the potential to address the problem described in Section 1.1. In the following, we summarize the state of the art in application of automatic methods for analytical quality assurance and describe open research gaps.

For this, we summarize related work from an evaluation, a quality definition, and a technical perspective¹⁰.

10 Please refer to Publication G for the detailed analysis of existing methods for QA of RE artifacts that lead to this summary.

2.5. Research Gap

First, one gap in existing automatic QA approaches is the lack of empirical evidence, especially under realistic conditions. Only few of the introduced contributions were evaluated using industrial requirements artifacts. Those who do apply their approach on such artifacts, focus on quantitative summaries explaining which finding was detected and how often it was detected. Some authors also give examples of findings, but only few works analyze the accuracy of their automatic approaches in depth, especially in the vague domain of ambiguity. When looking at the characteristics that are described in ISO 29148, we have not seen a quantitative analysis of precision and recall. Furthermore, reported evidence does not include qualitative feedback from engineers who are supposed to use the approach, which could reveal many insights that cannot be captured by numbers alone. However, we postulate that the accuracy of quality violations very much depends on the respective context. This is especially true for the vague domain of natural language where it is important to understand the (context-specific) impact of a finding to rate its detection for appropriateness and eventually justify resolving the issue.

Second, the existing approaches are based on proprietary definitions of quality, based on experience or, sometimes, simply on what can be directly measured. The ARM tool [WRH97] is loosely based on the IEEE 830 [IEE98] standard. However, as the recent literature survey by Schneider and Berenbach [SB13] states: “the ISO/IEC/IEEE 29148:2011 is actually the standard that every requirements engineer should be familiar with”. We are not aware of an approach that evaluates the current ISO 29148 standard [ISO11b] in this respect. As the analysis of existing works in Publication G shows, for most language quality defects of ISO 29148, there has not yet been a tool to detect these quality defects. To all our knowledge, for neither of these factors, there is an differentiated empirical analysis of precision and recall.

Yet, many other quality models (most notably from the ambiguity handbook by Berry et al. [BKK03]) and quality violations could lead to Requirements Smells, as far as they comply with the definition given in the next section.

Finally, taking a more technical perspective, our Requirements Smell detection approach does not fundamentally differ from existing approaches. Similar to previous works, we apply existing NLP techniques, such as lemmatization and POS tagging, as well as dictionaries. For the rules of the ISO 29148 standard, no parsing or ontologies (as used in other approaches) were required. However, to detect superlatives and comparatives in German, we added a morphological analysis, which have not yet seen in related work.

In summary, we need more evidence on automatic QA for requirements artifacts via systematic studies in terms of distribution, precision, recall, and relevance, as well as by means of a systematic evaluation with practitioners under realistic conditions.

30

CHAPTER 3

Research Design

This chapter outlines the design of this thesis. In particular, based on the problems described in Chapters 1 and 2, we derive a thesis statement and research goals.

Afterwards, we detail these goals into research questions. For each research question, we design applicable methods and summarize the contributions. Lastly, we explain how these contributions relate to other works, to which the author also co-contributed.

3.1. Problem and Thesis Statement

In the introduction, we described the problems of incompleteness, inadequacy and imprecision of the current state of the practice in RE artifact quality. In the previous chapter, we described that the state of the art in RE artifact quality research is not able to address this problem. We summarized this in the following problem statement:

Problem Statement: We have a limited understanding of what high quality RE artifacts are and need more efficient methods to control RE artifact quality in practice.

Thesis Statement: In our work, we address these problems through an activity-based understanding of quality and automatic detection of quality factors. In summary, we argue that an activity-based quality model enables to more precisely and completely model RE artifact quality for a given context. Moreover, to apply these quality models and therefore to improve RE artifact quality in practice, a combination of automatic and manual methods can help to increase efficiency, speed and consistency of RE artifact quality control.

3.2. Research Challenges and Research Questions

The problem and thesis statements raise two challenges, which we refine into two research questions each (see also Fig. 3.1). The first part sets the theoretical basis, whereas the second part targets more efficient solutions for quality control.

3.3. Methods and Contributions

Challenge 1: We need a precise and valid understanding of what high quality RE artifacts are in a specific context.

RQ 1: How can we precisely define quality for RE artifacts? To precisely discuss what high quality requirements are, and to have a basis for quality assurance, we need a model to systematically reason about RE artifact quality. In other words, we need a model that explains which factors of an RE artifact define it as a high-quality artifact. This model must be adaptable to various contexts, and must allow to accept or refute a factor, based on an systematic argumentation.

RQ 2: How can we create valid quality models? The language to define such a quality model is not sufficient. We furthermore need applicable methods to verify that the models are valid. We need approaches that enable to either build valid models from scratch or differentiate the correct from the wrong factors.

However, as explained in Chapter 1.1, to just understand and define quality in a precise manner is not sufficient since manual QA lacks efficiency in practice.

Challenge 2: We need more efficient methods to control RE artifact quality in prac-tice.

RQ 3: How can we efficiently ensure quality factors? In practice, even if there is an established quality model, projects struggle to ensure that their RE artifacts adhere to this quality model. Therefore, we furthermore need an efficient method to support requirements engineers keeping the desired goal of artifact quality. To answer this, we propose an approach called automatic requirements smell detection.

RQ 4: What are the benefits and limitations of requirements smell detection? To validate that the requirements smells approach in fact achieves the stated goals, we validate the advantages and limitations of such an approach. In particular, we are interested to understand in which cases automatic smell detection cannot support requirements engineers.

3.3. Methods and Contributions

In Fig. 3.1 we provide an overview of the approaches and contributions of this thesis. The contributions are structured along the two aforementioned problems.

Each problem is addressed through first, an analysis and design phase leading to a constructive approach (RQ 1 and 3), and, subsequently, the approach’s evaluation phase (RQ 2 and 4). Each research question is answered by one or multiple contributions, which we explain in depth in the following. Furthermore, Fig. 3.1 shows how the contributions relate to the Publications.

RQ 1: How can we precisely define quality for RE artifacts?

Method: We suggestactivity-based RE artifact quality models (ABRE-QMs) to precisely define quality in a given context. Based on a quality-in-use viewpoint, an ABRE-QM defines quality as a set of quality factors of entities that have an impact on activities in the software development process. We furthermore provide an approach to define ABRE-QMs and discuss a research roadmap.

Contribution 1. The notion of Activity-based RE artifact Quality: RE artifact QA requi-res a precise understanding of quality. As explained in Chapter 1.1, to this end, quality models are often incomplete, inadequate and imprecise in their

32