We explored stakeholders’ perceived usefulness of user feedback, use cases for automated feedback analysis, and possible integrations into their workflow. We applied qualitative research by reviewing seven papers from related work and by conducting an interview study with twelve stakeholders. In total, our qualitative analysis encompasses the evaluation of 90 stakeholders from the industry. We summarize the findings of this chapter.
User feedback usefulness. Stakeholders value user feedback but rarely use it in their decision-making process. They argue that both explicit and implicit user feedback comes in amounts that are unfeasible to analyze manually. They are not aware of tools that analyze user feedback automatically and, therefore, rely on the manual analysis. One stakeholder said that for analyzing 1,000 app reviews, their team spent two full days, but as research shows, popular apps re-ceive about 4,000 app reviews and about 32,000 tweets daily. Still, many of the stakeholders in our qualitative research state that they try analyzing the feed-back manually. They realized that user feedfeed-back could lead to quicker bug fixes, increase the requirements quality, and can contain information for finding inno-vative features. Stakeholders do not consider their feedback useful but also the feedback their competitors receive to identify gaps in the market.
So far, some stakeholders have tools to collect user feedback, but most do not have any tool support that helps them to get meaningful information such as ideas for new requirements. Depending on the stakeholder’s role, they either seek for more general data representation about the overall app performance or more detailed summaries hinting toward the actual pain points and requests.
Use cases for automated user feedback analyses. Stakeholders stated that feedback is most useful if automated approaches filter it so that only relevant feed-back is visible. Additionally, if irrelevant feedfeed-back is filtered and only feedfeed-back describing, e.g., problem reports and feature requests, is available, the amount of feedback is sometimes still too high. Therefore, they wish for further aggregations like grouping similar feedback.
Typically, the majority of user feedback comes from non-technical users. There-fore, if stakeholders solely analyze explicit user feedback, they do not have access
to interaction and context data, which are essential to understand described prob-lems. Alternatively, if stakeholders only have access to implicit feedback, they can understand “what” users did but now “why”. They do not know the reasoning of the user. Our study shows that combining explicit and implicit user feedback allows for generating new and improving existing requirements. In the ideal case, stakeholders have filtered explicit feedback available that describes, e.g., a prob-lem and have access to the interaction and context data that led to the probprob-lem.
Therefore, the combination of both feedback types can lead to quicker bug fixes.
Nonetheless, some stakeholders stated that they would not necessarily trust automated approaches. For example, a stakeholder stated that they would like to see the filtered feedback, too, to understand if they miss important informa-tion. They want to correct wrongly categorized feedback and include it in the analysis. Therefore, it is essential to give stakeholders a control mechanism in the tool to correct and improve the underlying algorithms.
Integration into workflows. The available IT infrastructure is different or even unique for most companies. Therefore, there is no single integration scenario that fits the workflow of every stakeholder. Most stakeholders in our interview study wish for a web-based standalone tool that they can access from anywhere at any given time. Such a tool helps stakeholders to feel in control and to feel informed about their app’s performance in the market.
Requirements Intelligence
It always seems impossible until it’s done.
Nelson Mandela
Contribution. This chapter contributes with the introduction of requirements intelligence, a framework based on stakeholder needs. The framework leverages the analysis of explicit and implicit user feedback. We detail the analysis activities and the integrated interactive visualization, which visualizes the analysis results.
4.1 Motivation
In the previous chapter, we detailed on stakeholders’ needs and expectations regarding automated analyses of explicit and implicit user feedback. We found that they need automated tool support to collect and to filter the vast amounts of user feedback they receive. The stakeholders further highlighted the need to identify the specific features (functional requirements) that their and their competitors’ users address. The stakeholders want to get an overview of their app’s performance and detailed insights into the user feedback and the addressed features in a web-based tool.
Following the insights of Chapter 2 and the results of Chapter 3, we propose requirements intelligence, a framework to continuously collect, preprocess, filter, transform feedback to requirements, and to interactively visualize explicit and implicit user feedback. First, we define the term requirements intelligence in Section 4.2. Then, we present the requirements intelligence framework and its activities in Section 4.3. After that, we introduce our machine learning pipeline
that we employ within the activities of the framework in Section 4.4. Section 4.5 summarizes the chapter.