6.2 Evaluation
6.2.2 Results
As shown in Figure 6.1, we discuss the results of each document mainly in line with the enhancement of internal communication and external
communi-6.2| Evaluation 117
cation. We use the definition in [Pik+08]. Internal communication highlights the communication within the software development team, whereas external communication in our context means the communication between develop-ment team and product owner, scrum master, safety expert and the customer in our context. The internal communication and external communication are further investigated in sprint planning meetings, daily scrums, development, and sprint review meetings. We use quantitative data to show the general effect on communication. The safety culture has also been analysed by quan-titative data. Afterwards, detailed observations are derived by qualitative data.
6.2.2.1 Safety Story
Internal discussion in sprint backlog refinement
External discussion in sprint backlog refinement
Internal discussion in development
External discussion in development
Internal discussion in sprint review meeting
External discussion in sprint review meeting
Safety culture +2
+1 0 -1 -2 positive
effect
negative effect
STORY EPIC STORY EPIC
STORY EPIC
STORY EPIC
STORY EPIC
STORY EPIC
STORY EPIC
no effect
Figure 6.2: Effect on Communication of Safety Story and Safety Epic As shown in Figure 6.2, the safety story pattern in our context shows positive values, together with little fluctuation between each respondent.
Only the external communication in the sprint review meetings and safety culture reveal some doubts.
Positive effect on communication:In the sprint planning meetings, some developers said:"The safety stories specify a good structure with a clear pur-pose, which is a basement for the internal discussion."Furthermore, they said:
"The safety stories reflect safety concerns and wishes in a clear manner, which allows a constructive safety discussion with safety expert."In daily scrum:"It helps the developers driving an initial idea."As the safety story pattern focuses on "what" rather than "how", it does not interfere the concrete developing
tasks and motivates various ideas and discussions in the development team.
Moreover, the developers prefer more communication with the safety expert to clarify their execution tactics. To compare with the functional user story pattern, a safety story is mostly produced by an objective safety analysis or norm requirements. A purpose-go-first expression can better reflect safety requirements’ properties.
The safety culture shows a positive value by using the safety story pattern.
The developers thought that: "Safety stories are the most detailed unit of safety features."Without a clear format of the most detailed unit, it is difficult to integrate safety culture into concrete development.
Negative effect on communication: In our project, we used Jira as the issue tracking tool. Some developers mentioned that:"Several safety stories seem still too long to be caught in the backlog page."The safety story pat-tern is explicit. However, safety background knowledge in the development team, which helps writing safety stories, could help to hinder some excessive detailed explanation. The comment block in Jira could be fully utilised.
The product owner mentioned also that:"There is no specific place in Jira for a safety product backlog". A Jira plug-in could be helpful in the future.
Moreover, based on the safety story pattern, a constructive discussion in the development team is still the main purpose. People should not neglect an efficient communication and only rely on the existing safety story pattern.
Finally, some developers reflected that: "The commitment of requirements between the safety expert and the development team has been improved, yet the commitment of accepting each requirement has not been focused on."The need for an acceptance criterion has been mentioned, yet an explicit pattern is still lacking [Rub12]. Thus, an acceptance criteria pattern would help to improve the external communication in the sprint review meeting.
General impact: Except the effect on communication, the safety expert gave also the feedback that: "The safety story pattern did help to have a commitment between the safety expert and the product owner."Before the implementation of the safety story pattern, there were a lot of conflicts and
6.2| Evaluation 119
interrelations between functional user stories and safety stories. With an explicit "what" focused safety story, the functional user stories not only have less conflicts to safety stories, but also can be linked with each safety story in Jira. Last but not least, the safety story pattern shows an easy adaptation into other safety analysis techniques, such as STPA in our context.
6.2.2.2 Safety Epic
As shown in Figure 6.2, all the values about safety epics are on a relatively positive level, whereas "external discussion in sprint review meeting" and
"safety culture" have more variance.
Positive effect on communication: In a sprint planning meeting, we got the feedback that: "We have more material support for a safety plan and for starting a discussion." It also helps to clarify confusing safety stories between safety expert and development team. Since some safety stories seem similar, a lack of a clear description about the system background will cause misunderstandings. Furthermore, because of the increasing amount of requirements, safety stories are necessary to be grouped, especially in complex system, such as Internet of Things (IoT). It helps the discussion in each sub-group and enables a clear identification of the safety scope in different sub-systems. The pre-planned safety epics ensure that the safety consideration has been put into the agenda, which motivates the develop-ment team to think about safety from the beginning of the project.
Negative effect on communication:However, during the daily scrum, sev-eral people have not reviewed the safety epics again. Some developers said:
"The safety epics do not affect the daily development in a critical manner, be-cause they show only a rough idea and are not really helpful to the internal discussion."The safety stories have been tailored into detailed tasks during development. The team members focus more on the development of detailed tasks. That causes some doubts on communication in development. In the sprint review, the safety epics seem also not frequently used. "Everything
in the review is mostly focused on the safety stories."However, for a better understanding of the status and planning for the next sprint, we need indeed some discussions about the status of safety epics rather than a temporary discussion in the review meetings. To add a time slot on sprint review meet-ings for safety epics’ discussion could keep the information of safety epics spread throughout the development team, the same as in the sprint backlog refinement.
General impact: From the product owner and safety expert viewpoint, a structured safety epic helps the priority management for the backlogs. The separation of safety requirements depending on the various sub-systems and different sources could help managers decide which safety stories should be put into the current sprint. Furthermore, a clear, structured, concrete safety goal has been integrated into system development. A safety epic helps the spread of safety culture in the middle-level, which fills the gap between detailed safety tasks and the generally overall system safety goals.
6.2| Evaluation 121
Table 6.1: Effect of Safety Documents
Documentation Positive effect on communication Negative effect on communication General Impact
Safety story
1. A clear purpose 2. An explicit structure 3. A quick overview 4. Reflect safety concerns and wishes clearly 5. Help building initial idea for further discussion
6. A "what" focused mode motivates more external communication 7. A "what" focused mode does not interfere the development
1. In Jira, some safety stories are still too long to be caught in the backlog page 2. For a better external
communication, the description block in Jira could be used
3. No place in Jira for safety product backlog 4. A structured communication based on the initial idea is more important 5. An acceptance criteria format is needed
1. More clear for safety expert and development team to have a commitment when writing them 2. No difficulty to adapt to
an other safety analysis method (STPA) 3. Safety culture has been enhanced
Safety epic
1. More material support in sprint planning meeting
2. Group safety stories 3. Great help for complex system (IoT) development
4. Help the discussion about safety stories
1. During development, the use of safety epics is low 2. In sprint review, no planned time slot for safety epics’ discussion 3. Difficult to inform
the development team when a safety epic is updated
1. Help managing the priorities 2. Give a clear structure of the integrated safety
3. Spread safety culture in the middle-level
Agile safety plan Need further research
1. A huge gap between the high-level plan and concrete development 2. The developers prefer a direct communication with the safety expert
rather than looking through the plan 3. Possible to be replaced by other documents
1. A clear overview and evidence of the process
2. Delivered safety report 3. Help solving conflicts between safety goals and functional goals 4. Backup knowledge for development team
1226|DocumentationinS-Scr
6.2.2.3 Agile Safety Plan
Negative effect on communication: Regarding communication, most of the students cannot give a value for evaluation. We summarise the causalities as follows: First, during the sprint, few developer looked at the agile safety plan. They mentioned that: "At the beginning, we got a basic idea and description of the safety plan, but later it did not have a major influence on our development."Second, there is still a huge gap between the safety plan and detailed development tasks. As one of the interviewee said that:"We could have a clear overview of the safety in the project, but have no idea about how to integrate this into our daily work."Third, the developers prefer more communication with the safety expert rather than review the agile safety plan when needed. The interviewee said:"The plan is well structured and with not so many pages, but when a problem occurred, we would like to talk to the safety expert directly."One of the original ideas in preliminary S-Scrum was using an agile safety plan to help developers solving unclear safety issues.
We added STPA technique’s information to it. However, the feedback shows communication is a more effective way. This seems not to be contradictory to our main purpose.
The agile safety plan supports a planned safety process and the certifi-cation body and to some extent strengthen the communicertifi-cation with the safety expert and/or the certification [MSL16]. Our intention of adding an agile safety plan and integrating parts of STPA information is because of a weak planning level in preliminary S-Scrum and the conflicts in priority management. Thus, although we failed the investigation of communication effectiveness from the agile safety plan in our case, the general impacts could also help the organisation deciding if an agile safety plan is necessary for its project. For example, using an agile safety plan as a deliverable part of the safety-critical product; capturing the important discussion, decision, or agreement between safety expert and development team to have a clear recollection; helping new team members come up to speed quickly with safety knowledge; and when there is a regulatory requirement for some certain safety-related documents [Rub12].
6.2| Evaluation 123
General impact: We derived the values of an agile safety plan from the interviews: For the scrum master, an agile safety plan provides an overview and evidence of the safety process. For the product owner, it is helpful for a continuous planning. For the customers, they are more clear and confident about the safety assurance of their products, and could have additionally a delivered safety report for the invisible safety artefact. For the safety expert, it is easier to have a commitment to the priorities of safety stories and func-tional user stories for each sprint. For development team, they mentioned that:"Because of a plan, the developers have a complete safety knowledge as a backup, when safety expert is unavailable.". A summarised result is depicted in Table 6.1.