Is a Design Rationale Vital WhenPredicting Change Impact?

(1)

Is a Design Rationale Vital When Predicting Change Impact?

Iulia Alexandrescu 25.11.2003 Hauptseminar für Überfachliche Grundlagen „Design Rationale“ WS03/04

Prof. Bernd Bruegge, Ph.D., Allen Dutoit, Ph.D.

25.11.2003 Iulia Alexandrescu

2

Why DR?

l Potential benefit for system changes in architectural evolution

Ë change impact analysis: the rationale for the architecture is vital when analyzing impact of new or changing requirements

l Negative effect of software development expansions Ëcosts of documenting the systemÁË must be balanced to the potential gain

l Without understanding the system architecture: change requests are difficult/ impossible/ errors

l Strong grip of the architecture required for product-family or product-line reuse Ë lead-time very important

3

Why DR?

Shorten Development Lead-time

l Access to market introduction window and chance of market dominance

l Decrease the risk of market lockout l Market survey information still valid at market

introduction

l Important for shareholders:

Ë stock value drops when announced product is later than expected.

Ë too late to market -> annihilation of potential market ->software investments wasted

4

Why DR?

Software Maintainability

l Regular release of updates and new features of existing software (e.g. mobile phone industry) l Know the existing system by working with the

system or talking to the initial developers l Difficult when complicated system or many

developers

l Establish communication over time Ë DR documentation: understand the present system architecture

5

Approaches to DR Documentation

l Retrospective DR

- written after development of the system first generation - does not delay the market introduction of the system first

generation l Narrative DR

- produced before the first release of a system - risk that the DR is biased by the DR author l Weaknesses in some approaches to DR:

- DR needs substantial training to be used effectively - weak association between a DR document and the actual code

(designers usually dislike making additional documentation) - some style guide and a structure necessary for a good DR

6

I. Controlled Experiment on Software Architecture Evolution- Lars Bratthall, Björn Regnell, Dept. of Comm. Systems Lund Univ.; Enrico Johansson, Ericsson Mobile Comm., Lund, Sweden

l Objective:Ë shorten development lead-time l Solution: Ë facilitate architectural changes by

providing a DR documentation - why the architecture is built as it is

l Hypothesis: Ë faster and more correct changes if such information is available during change impact analysis.

l Controlled experiment:

- How effective is DR as a support for change impact analysis?

(2)

7

The Experiment

l Object of study: a textually represented DR combined with some system static architectural description at various aggregation levels

l Purpose: measure the DR effectiveness Ë how well and how fast change requests are performed

l Context: 2 systems: a local telephony switch control software and a car cruise control system

l Participants: experienced developers coming to a new system, without previous knowledge of it:

- 7 Senior industrial designers and 10 software engineering Ph.D. students and faculty members - One group had access to a DR, one did not

8

The Experiment

Hypotheses and Realization

l Main hypothesis: people design differently when having access to a DR

l Level of significance chosen beforehand to avoid fishing for particular results

l Levels of experience with real-time systems and modeling language used to randomize participants Ë ability to quickly identify errors in distributed real-time systems

l Subjects exposed to 2 or 3 systems, and a number of change requests for each system

9

The Experiment

Tools

l SDL = The model language: describes the code graphically using extended finite state machines, which can be hierarchically grouped.

l SDL described to the participants (slides and handouts) Ë reasonable understanding of the code modeling language

l All knew what to do and how to fill in the forms l Participants were assigned change requests in

random order (realistic requests for new services) l The DR author has had no knowledge of the change

tasks to come Ë reflected reality

10

The Experiment

Threats to Validity

l (1) Conclusion validity - relationship between treatment and outcome.

l (2) Internal validity - - matters that may affect an independent variable’s causality, without the knowledge of the researcher.

l (3) Construct validity - whether we measure what we believe we measure Ë design threats and social threats.

l (4) External validity - generalization of the findings to other contexts and environments than the one studied.

11

The Experiment

Summary of the Results

l Quantitative: - improvement in correctness and speed for one of the systems, with access to a DR document - inconclusive results for the other system ËFurther studies recommended.

l Qualitative: DR speeds up changes and improves correctness

l DR appreciated more for the more complex system B than for the less complex system A

l No participants claimed that access to a DR was harmful l Effectiveness of the DR decreases as the system gets

better known

l Participants liked having access to a DR.

l Faster and better work with access to a DR, than only with source code and requirements specification

12

DR Improvement Suggestions

l Original designer should write why each aggregation level is broken down into lower ones, and describe the purpose of each component.

l Strong association between the DR and the code

l Limited size of the DR.

l On a system level, only 4 standardized headings:

Ëorganization of system into files Ëuse of language constructs Ëmain dynamic architectural principles Ëclues to understanding the system.

(3)

13

Suggested Approach – Advantages

l Information content proven to be useful in architecture level design.

l Maintain flexibility in a system during its evolution through architecture understanding

l Requires only little training, good for large systems, easily used with existing case tools

l Writing a DR for a component at a particular aggregation level can be prompted by a code entry tool Ë higher likelihood of producing the documentation.

l Documentation follows code structure Ë appropriate DR more likely to be found during changes

l DR documentation tightly associated with the code Ë easily maintained when code changes

14

Conclusions

l Access to DR (in the suggested way) - positive impact:

‡ on lead-time and quality

‡ when experienced designers need to predict where changes must be performed

‡ on an unknown real-time system.

l Participants felt the need for a static architecture overview and sequence diagrams.

l Further experimentation needed to find “the best”

model for various purposes.

l Projects where lead-time is important - DR during design:

cheap, effective way to facilitate future system evolution, without delaying initial system release.

15

II. Representation and structure in the re-use of design rationale by novice analysts

- Georgios P. Iliadis, Loughborough University, UK

l Theme: Use of design rationale to manage breakdowns that take place as part of software design activities.

l Objective:

- investigate the potential of DR to transfer expertise and process-related information to novice analysts/designers (a cash machine)

l Questions:

Ë which representation is most suitable to a typical design problem

Ë the role of the structure of the argumentation material in this context

16

The Experiment

Design and Hypothesis

l 2 independent variables:

DR representation

Ë tabular DR - suitable to making sense of DR syntactic aspects Ë narrative DR - useful in eliciting semantics out of a DR Ë graphical DR - for combination of multiple (related) fragments Ë no prediction on the overall performance

DR structure - 2 types of the narrative QOC form:

Ë leading the reader from options to criteria Ë leading from criteria towards options.

Ë in a reviewing/evaluating mode, the latter is more suitable Ë increases subject performance.

l 2 dependent variables:

Ëcorrect answers fl‡ response time

17

The Experiment

Realization

‡ Subjects: first-year undergraduate students in Information &

Computing, or Computing & Management

‡ Training session

- tutorial on all forms of DR using QOC and a short exercise

‡ Practical session

- students were handed the tasks one at a time - Videotaped: the camera timer used for response time measurements

- Preference questionnaire and a short debriefing session at the end

18

The Experiment

Results and Conclusion

l Tabular form = the easiest for novice analysts to grasp l Graphical form = convenient to work on (very expressive) l Narrative form = a bit awkward to browse, but quite useful on focus.

l QOC elements semantics are intricate as a first-time subject; should be elaborated extensively in DR teaching.

l A narrative format is overall harder to comprehend.

l Tabular form - the most suitable for the task; compact l Graphical form - the most effective.

l Preferences:

- The tabular DR by far the most preferred - The narrative and graphical ‡ equal scores

(4)

19

III. An Empirical Evaluation of Design Rationale Documents

- Laurent Karsenty ARAMIIHS, Toulouse, France

l Objective: evaluate how useful DR documents are when a designer needs to reuse a previous design.

l Considered:

Ë integration of the DR with the information currently processed by designers (integrating DR with blueprints) Ë introduction of an iterative approach to the DR capture, going beyond the design period

Ë the need to promote collaboration between designers

l Experiment in mechanical engineering design – redesign of a 9 months airspace project

l Valuable for other fields ‡ view on the approach for capturing DR, not the application domain.

20

The Experiment

l Questions:

1) Do designers confronted with an unknown design need to know the DR?

2) How do designers use DR documents?

3) Do we succeed in capturing the DR designers look for?

l Subjects: 6 experienced, professional designers (3 engineers and 3 technicians)

‡ external to the project

‡ were asked to understand and to assess a past design

‡ provided with documents describing the solution and the DR (documents constructed using the QOC method)

21

The Experiment

Realization

l 2 kinds of information:

1) describing the solution ‡ mostly blueprints

2) describing the reasons for design decisions ‡ paper-documents l Designers were free to use blueprints and DR as they

chose

l Evaluation meetings:

1) Description of the tasks, and presentation of the available documents.

2) Beginning of the tasks, without time constraints.

3) At the end, a few questions on designers’ opinions on the usefulness of the DR

4) Evaluation meetings lasted about 1 hour each, and were taped.

22

The Experiment

Results and Conclusion

Ë2 Ways of using the DR: opportunistic use, and extensive use (paper better than computer)

1) more than half of the designers' questions concern DR 2) some designers extensively use the DR document as a

support for understanding and assessing a previous design 3) 41% of the designers' DR questions answered with the

QOC-based document

ËDR documents useful, at least for designers who use it as a support to their reasoning, but not sufficient.

Ëtraditional approaches for capturing DR exhibit some limitations ‡ Possible adaptations necessary

23

Analysis of the Experiments

l Experiments showed:

- DR is beneficial to software development and has positive effects on marketing if the appropriate DR is used - DR should be used together with schemes and other visual

descriptions of the system

- DR good for future developers of the system, but not appreciated by actual developers (time for documentation, delay of system development, lack of interest in

documentation – not their responsibility(?), possible threat to the actual job…)

- A good DR easily used by people with no DR experience Ë positive effect on future development of the system

24

Improvement Suggestions

Ë Exp 1:

l Participants indicated that they needed a static architecture overview and sequence diagrams

l Further experimentation is needed to find out what is “the best”

model for various purposes.

Ë Exp 2:

l More emphasis is to be placed on the second part of a re-use situation i.e. the problem-solving part, as that is more directly associated to breakdown situations.

Ë Exp 3:

l Analyses do not give a complete answer to how useful DR documents are, especially as they did not investigate the potential difficulties for accessing pieces of information composing the DR.

l Traditional approaches for capturing DR exhibit some limitations.

Possible adaptations of these approaches are necessary.

(5)

25

Validity

l The conclusions of the experiment were very realistic – validity threats dealt with and eliminated.

l Solid conclusions for each type of experiment:

‡ real-life projects as starting points

‡ probable change requirements and system improvements/ upgrades

‡ randomization of participants assured

‡ subjects were appropriate for the tasks