5. Quality, Dependability and Validation
5.1 Layered Consideration of Errors
usage level extern (perceptible)
program level intern (latent)
logical level dto.
physical level dto.
Error
Propagation of undesired events layer to layer in a system
5.2 Quality Requirements / Development Activities
Maintainability (Can I repair it?)
Flexibility
(Can I change it?)
Testability
(Can I check the performance?)
Portability (Can I use it on another
machine?)
Reusability (Can I reuse parts of it?)
Interoperability (Can I connect it to another system?
Product Operation Prod
uct T ran
sition Produ
ct Revision
Correctness (It does what I want?)
Reliability (Do this always like this?)
Efficiency (Does it run on HW fast?)
Integrity (Is it safe? / protective / protected, ...)
Usability (Can I use it good?)
5.3 Dependability
Counter measures / Tools (Means)
Measures
Faults Errors
Malfunctions Failures
Procurement
Assessment
Reliability
Availability
Safety Security Undsired States/
Events (Impairments)
Deviations
. . .
. . .
. . .
Dependability
(nach J.-C. Laprie, 1989)
„Property of a computer system such as reliance can be justifiable placed on the service it delivers.“
readiness for usage
continuity of service
non-occurence of catastrophic consequences onto environ- ment
non-occurence of unauthorized disclosure of information
non-occurence of improper alteration of information
aptitude to undergo repairs/
evolution
availability reliability
performability assurance
security assurance
safety confidence integration maintainability
5.4 Software Testing
Analysis (Static)
Formal Methods
Non-Formal Methods
Testing (Dynamic)
(Correcntness Proof, Symbolic Testing, Algebraic Methods,
(Dynamic)Assertions, ...)
(Review-Oriented:
Review/Walkthrough/
Inspection, ...)
Semi-Formal/Empircal Methods (Hybrid Methods)
Implementation-Oriented Testing
Grey-Box Testing
Specification-Oriented Testing (Black-Box Testing /
Functional Testing)
(White-Box Testing /
Structural Testing)
Software Testing Strategies
Objective (Method Selection) Component Level Component order
Coverage Test Debuging
Modul Test Subsystem Test Integration Test System Test ...
Bottom-Up Test Top-Down Test Sandwich Test Control Flow-Oriented
(C0-,C1-, ..., C-Test)
Acceptance Test / Release Test Regression Test
Mutation Testing / Back-to-Back-Test Data Flow-Oriented
(dd-, dk-, kk-, ... -Test) Operational Profile ...
≈
Cause-Effect Analysis
Equivalence Class Test / Partition Test Boundayr Value / Domain Test
Statistical Test / Random Testing ...
5.4.1 C-Test Example
– 2 decisions, 1-time execution:
{abe, ace} → 21 = 2 paths
– 2 decisions, 2-times execution:
{abe, ace, abdbe, abdce, acdbe, acdce} → 22 + 21 = 6 paths
– 2 decisions, 3-times execution:
{...} → 23 + 22 + 21 = 14 paths
– General : p := i
p : Number of Paths d : Number of Decisions n : Number of Repetitions
con1
con2
sta1 sta2
J J N
N
∑
= n id 1
b c
d
e a
25 repetitions:
3
25+ 3
24+ ... 3
1= 1,2*10
12 ≈1 Billion path
START
END T higher
than 150 °C Pressure
relief valve has been switched on?
Pressure greater than 10
bar?
Switch Reserve Cooling System
on Pressure
increases
T increases
N
J J
J
N
N
Exercise: Control Flow Graph
Regular Expression?
5.4.2 Representation of Michael Jackson Specification using Regular Expressions
Example of a Jackson Tree
x
a*
b c d e
f o g o
h i j k l
r o
m o n o o o p o q o
s o t o
p = t = λ (the empty word).
Realization of the Tree constructs as Regular Expressions Operations
x
a b c
y
a o b o c o z
b *
Sequence
(a b c)
(a+b+c)
Selection
a*
Iteration
Establishment of a regular expression for the example:
a = bcde c = f+g g = hijkl i= m+n
k = o+p; p= λ l = q+r
r = s+t; t = λ
a = b(f+(h(m+n)j(o+ λ ) (q+(s+ λ )))) de
x = a*
Generation of test paths using the regular expression:
– Specification (restricted to a single execution of the loop) x = a* with a = b(f+(h(m+n)j(o+ λ ) (q+(s+ λ )))) de
– One can obtain all paths of the restricted expression by its expansion (single execution).
• bfde
• bhmjode
• bhmjoqde
• bhmjosde
• bhmjode
• bhnjose
• bhnjode
• bhnjsde
• bhnjde
• bhmjsde
• bhmjde
• bhnjoqde
• bhnjqde
5.4.3 Data Flow-Oriented Testing
Operations on paths
d: defined, created, initialized, etc. (imported) k: killed, undefined, released (released)
u: used for something (benutzt)
c: used in a calculation (in arithmetic Operation(en)) p: used in a predicate (in logical Operation(en))
dd: suspicious1 kd: normal ud: bug? normal?
dk: ?bug?2 kk: ?buggy?3 uk: normal
du: normal ku: bug3 uu: normal
*: don´t care (nothing of interest happens)
*k: ?anomalous? k*: normal
*d: normal d*: ?anomalous?
*u: ?anomalous? u* normal
(1= suspicious : verdächtig; 2= bug : Fehler; 3= buggy : verdächtig)
5.5 Strength of the Test Criteria
>
>
All *c uses All*c/some*p >
All paths >All du paths > All uses > All defs (d´s) > Decisions > Basic Blocks All*p/some*c >
All *p uses
branch
statement
x
> y (x Stronger y) : <=> {a|a: test case produced by x} ⊃ {b|b: test case produced by y}
5.6 Testing Aids
– Automatic Test Driver (ATD) – Test Languages
– Test Case Generator
Static Test
Test Driver / Comparator
Test Case Generator
Test Cases
Test Results
Structure Report Test
Report
Test Inputs
System Under
Test
Test Program
Test Instrumentor
Structure Information
Type Declarations
Test Tool
Test Environment
ALL TESTCASES SUCCEEDED STATEMENT EXECUTED : 100%
BRANCHES TRAVERSED : 100%
Report of a Passing Test
VERIFY FAILURE IN TEST CASE 2 AFTER TERMINATION
(SUB1: K.EQ.8.AND.SUB1: N.EQ.29) OUTPUT ERROR ON RECORD 3 OF LUN 10 IN TEST CASE 3
STATEMENTS EXECUTED ...
...
Report of a Failed Test
5.7 Non-Formal Testing Methods for Software
– Non-formal: not based on rigorous mathematical methods
– Formalized: oriented by conventions of empiricism / experience – Non-formalized formal test method
-Formalized Reviews, -Inspections,
-Walkthroughs for
• Correctness
• Consistency
• Completeness
• Portability
• etc.
General Description: Inspection-Oriented Testing and Analysis Methods
5.8 Test Termination
Fault Costs Quality Costs
Optimum Quality Level
Quality Assurance System Costs Costs
Test Course – Cost-Related
Costs or Number of Faults
Cost per Discovered Fault
t Cumulative Number
Of Discovered Faults
Test Termination
Test Course - Fault Detection
5.9 Reliability - Prediction / Estimation of the Fault Potential
t o t
Fault Rate
Instantaneous Value of
Fault Rate Target Value
Observation Time
Previously Observed Data Predictions
t 1
Prediction Period
Approximation
Fault Reporting, Projection into the Future
Some Well-known Software Reliability Models (I) (Overview)
– Jelinski-Moranda-Modell,
– Bayesian Jelinski-Moranda-Modell, – Schneidewind-Modell,
– Geometric-Modell,
– Generalized Poisson-Modell, – Goel-Okumoto-Modell,
– Musa-Okumoto-Modell, (Basic Execution Modell) – Yamada Delayed S-Shape-Modell,
– Littlewood-Modell,
– Littlewood Nonhomogeneous Poisson-Prozess-Modell, – Littlewood-Verall-Modell,
– Keiller-Littlewood-Modell,
– Brooks-Motley-Modell,
– Duane-Modell.
5:10 Instead of Concluding Remarks
The "magic" triangle
Compliance Costs
Meeting Deadlines
Quality Level
decrease increase
shorten extend
decrease increase
