4. Summary of Results 41
4.2. RQ 2: How Can We Create Valid Quality Models?
4.2.3. Summary of Results
We executed each of the aforementioned methods in a different case. In the following, we report on the individual results. We start with a summary of the validation of a quality model in interviews. Afterwards, we briefly explain the results of a case study on quality factors for RE artifact maintenance. Finally, we summarize our experiment on the impact of the quality factor passive voice on understanding RE artifacts.
4.2.3.1. Case 1: Validation through Expert Interviews
This section summarizes the results reported in Publication A.
Goal. The goal of this study, as formally described in Tab. 4.1, was to evaluate whether an ABRE-QM representation of requirements quality definitions helps team members in a software project.
Table 4.1.:Research Goal of Case 1 Evaluatean activity-based quality model
with respect tothe validity of impacts, as well as benefits and drawbacks to validate a company-specific quality definition
from the point of view of project team members
in the context of use cases in a industrial business information system development setting.
Study Design. We performed the study at the insurance company Munich Re. As study subjects, we identified a leading requirements engineer and an experienced developer. To create the model, we took the company’s 28-pages use case guidelines and translated each of the roughly 50 rules into an ABRE-QM representation. In a first workshop with the experts, we validated artifacts, entities, stakeholders and activities present in their software development environment. In a second workshop, we iterated through each quality factor and impact and discussed the validity of this impact. After the validation, we openly discussed the benefits and drawbacks of such a representation.
Results. For most of the impacts, the translation was straightforward to conduct.
However, for a few quality factors, such as presence of UI details as well as the use of subflows, the explication of arguments revealed tradeoffs for quality factors.
For example, the presence of UI details in RE artifacts has various impacts on the further activities. Fig. 4.3 shows a summary of the impacts, both positives impacts on understanding and testing, but also negative impacts on maintenance and use case execution. In Publication A, we discuss this tradeoff in depth. In another example, Munich Re enables the reuse of flows in use cases throughsubflows, a construct that is equivalent to a function call in programming languages. The validation explicated that this concept has advantages during maintaining and reuse-activities, but disadvantages in reading and understanding-activities. The
4.2. RQ 2: How Can We Create Valid Quality Models?
Req. EngineerImplementerTesterUser
Implement UC All Stakeholders
Create UC MaintainUC Find UC Understand UC PerformUC
Use Case
Basic Flow Name
Preconditions
Postconditions
StepUI Design Details Rationale: A visual representation of aUI supports the stakeholder in understanding, because it makes the Use Case more concrete.
+
Rationale: Visual elements changemore often than the way how a user interacts with the system. Therefore, UC with UI detail must bechanged more often.-Rationale: The tester can associate the UC directly to the UI which makes running the test case easier.
+
Rationale: A user might have to use asuboptimal UI, because it was determined early inthe process.
-
Run Test Derivate Test IdeaImpacts Stakeholder & Activities
Artifacts & Entities
Figure4.3.:ThisfigureshowsanABRE-QMexcerptforUIDesignDetails.TheexcerptshowshowABRE-QMcanbeusedtosystematicallyanalyzequalityfactorsthroughitsimpacts.
50
4. Summary of Results quality model thus shows this tradeoff and could enable a cost-based evaluation, as is known from activity-based quality models in other disciplines such as source code [DJLW09, Dei09, WLH+12] or system tests [HJE+14, Hau16].
After the workshops, the experts expressed that they see two advantages of such an approach: First, making the impacts explicit and discussing tradeoffs serves as a validation of their own implicit quality understanding, as well as their guidelines.
Debating the tradeoffs, in their opinion, could help improving the guidelines. Second, the experts mentioned that the ABRE-QM could help improve the completeness of the guidelines: Since very few of the impacts lead directly to testing, they discussed whether their use cases were optimized for testers or if additional rules are required.
In summary, the results indicate that interviews enable to discuss a larger set of quality factors. For example, in this case, we could discuss a whole quality model in two workshops. This method also enables us to explicate some of the arguments and counter arguments for or against some quality factors. However, when facing unknown or contradicting quality factors, we need more precise and less subjective methods.
4.2.3.2. Case 2: Generating Factors for Maintainability through Change Analysis The majority of costs in the life cycle of a project do not go into the creation of software, but into its maintenance [Boe81, Gla98]. While quality factors for software maintenance have been studied for source code [DJLW09, Loc13], quality factors for maintenance of RE artifacts is less known. This section summarizes Publication C.
Goal. For this study, we differentiate between semantic changes of RE artifacts, which change the system described, and syntactic changes, which change the de-scription of the system. The goal of the study (as formally described in Tab. 4.2) is to analyze these changes to gain a deeper understanding of maintenance in RE artifacts. This allows us to derive quality factors for unnecessary, cumbersome, and risky activities during RE maintenance.
Table 4.2.:Research Goal of Case 2 Analyzechanges in contents of use case artifacts
with respect tofrequency, location, change types, and level of risk from the point of view of requirements engineers in practice
in the context of a large business information system in maintenance.
Study Design. In order to understand how to design RE artifacts for good main-tainability, we analyze: How many of the changes in maintenance are semantic changes (i.e. changes of the system described) versus syntactic changes (i.e. changes how the system is described)? In addition, we analyzed, during these changes, what were quality factors of the RE artifact that made changing the artifact (subjectively) difficult?
To answer these questions, we analyzed 15 months of changes in the use cases of an industrial software project in maintenance phase. We developed a typology of changes, based on the differentiation between semantic and syntactic changes and quantified the distribution within this typology. We furthermore identified and quantified related changes. We discussed the results with practitioners and analyzed their perception of difficult changes.
4.2. RQ 2: How Can We Create Valid Quality Models?
Results. The study shows that roughly every second change in the RE artifact did not change the functionality of the system itself, but only the documentation of the system within the RE artifacts. Since at least the syntactic changes could be avoided if carefully designed upfront, we argue that these efforts show the relevance of maintainability for RE artifacts. Furthermore, certain use cases were especially prone to changes, due to their unfortunate structural organization. We discovered that the maintenance of alternative flows in use cases is of special relevance for maintenance, since these are the parts that frequently change. Lastly, a qualitative and quantitative analysis aiming at a deeper understanding of problematic changes identified the quality factor of a changing terminology as well UI design details causing many changes. Practitioners reported that interconnected use cases negatively influenced maintenance, since keeping consistency in these use cases makes their changes particularly difficult and risky.
All in all, the results indicate the potential of case study research to identify previously unknown quality factors in a relatively unknown field. However, we have to carefully analyze whether the results are specific to this individual project. In addition, in case study research, we cannot control the variables, thus preventing us to draw conclusions about what would have happened if the project had documented their RE artifacts differently. Therefore, this method enables us to explore on unknown territory, but we have to take results with a grain of salt.
4.2.3.3. Case 3: Experimental Validation of a Quality Factor
In a third case, we analyzed a very common quality factor for RE artifacts in depth:
passive voice. Passive voice is widely considered a problem in RE [Kof07, ISO11b, Wie05]. Nevertheless, passive voice is widespread in RE artifacts in practice (for examples see [MoD11, Bos07, BHH+03, MCH+12]). Consequently, the question arises: Does passive voice really cause problems in understanding RE artifacts?
Therefore, we inspected passive voice as one quality factor, in order to understand its impact onunderstanding activities (see Fig. 4.4). This section summarizes the results of Publication D.
Stakeholders & Activities
Artifacts & Entities Artifact
Sentence
Passive Voice
Understand
Identify Actors Identify Domain
Objects Identify Domain Object Relations
Quality Factors
& Impacts
?
Reader
Figure 4.4.:In our experiment we analyzed the impact of passive voice sentences in RE artifacts onto the reader. This figure shows the corresponding ABRE-QM.
52
4. Summary of Results
Goal. The goal of this study, as formally described in Tab. 4.3, is to understand whether passive voice in RE artifacts causes problems for readers in understanding.
Table 4.3.:Research Goal of Case 3
Characterizethe impact of passive voice sentences in RE artifacts on un-derstanding
with respect to the quality of the artifacts produced from the activities find actors,identify domain objects,identify domain object relations
from the point of view of the reader of an RE artifact
in the context of an analysis of requirements from real projects by graduate and undergraduate students in computer science.
Study Design. For this study, we took a set of industrial RE artifacts and extracted requirements that contained sentences in passive voice without agents (by-phrase).
We then asked our study subjects to provide us with information about the sentence, including the main actor, the main domain objects and their relations (in form of a domain model). We then compared the results of the treatment group with the results of a control group, who received the same requirements translated into active voice.
Results. The results were inconclusive regarding whether or not passive voice hinders identification of actors: Although the active voice group performed better, these results were not statistically significant. The identification of domain objects was equally difficult for both groups. However, the passive voice group performed significantly worse when asked to identify how the domain objects relate. Many participants in the treatment group were even unable to relate objects at all (see Fig. 4.5).
This experiment indicates that taking a quality-in-use perspective allows us to more precisely reason about quality factors. In addition, experiments can be used to inspect effects of certain quality factors in depth. However, experiments are a very costly endeavour, both in terms of time and personell. In addition, it might not be possible to study all effects (e.g. long-term-effects) in such an artificial environment with reasonable time and money.