Basic Aspects of an Extended Reliability Analysis of a HAFTLAN (Homogeneous Auxiliary Fault Tolerant Local Area Network)

deposit_hagen

Mathematik und

Informatik

Informatik-Berichte 55 – 06/1985

Winfrid G. Schneeweiss

Basic Aspects of an Extended Reliability Analysis of a HAFTLAN (Homogeneous

Auxiliary Fault Tolerant Local Area Network)

BAsrc AsPECTS • FAN ExTENDED RELIABILITY ANALYSIS oF A HAFTLAN (HOMOGENEOUS ÄUXILIARY FAULT TOLERANT LocAL ÄREA NETWORK)

Winfrid Schneeweiss, Hagen

Abstract :

This is the basic report of an investigation of the fundamental reliability characteristics of HAFTLANs for single fault tolerance in connection with 1-diagnosability. It is shown how fault trees or equivalent graphs can be found for such systems and how reliability characteristics such as availability and failure frequency can be deduced. Furthermore, some thoughts on system self-diagnosis and on well organized routing even in cases of certain component failures are communicated.

*

--, _' ') ' I , _ / : \ . , , , /

----~; _;,! :., ' ;

page:

0 Notation 3

1 The HAFTLAN concept for tolerance 4

2 Synthesis of a fault tree or a success tree for a HAFTLAN 7

3 Availability of a HAFTLAN 13

4 Failure frequency, MTBF, and MTTR of a HAFTLAN 16

5 HAFTLAN self-diagnosis 18

6 Routing in a HAFTLAN 26

7 References 29

Ü NOTATION

a,b, ...

indices of sub-systems (units, components)

c 0,c 1,... coefficients of the multilinear form of system unavailability

A

availability

O

duration of the tests t . .

l , l +

E{Y} expection of the random variable Y

<p Boolean function (of fault tree)

i

,k,.t,m,n indices

µ

VS

Pr{a}

s

t . .

l ,J t . . ^k

l , J '

fa

i

ure rate

1/MTBF repair rate ;

1/MTTR

number of HAFTLAN nodes (micros)

number of work-computers directly attached to HAFTLAN mean failure frequency of system

probability of the random event a index for system

test of sub-system j by sub-system i

comparison test of units i and

with unit k as comparator T . . ,T . . k (binary) result oft.

resp. t . . k

l,J l,J, 1, l,J,

X fault tree binary state indicator variable

/\

V

negation of X

binary state indicator variable of component k, resp. system S indicator variable of broken connection between nodes i and j u n a v a

l ab

l

ty

logical AND (conjunction); /\ is often ornitted

logical OR (disjunction)

1 THE HAFTLAN CONCEPT FOR SINGLE FAULT TOLERANCE

HAFTLAN is supposed tobe able to integrate several office or labora- tory mini or micro computer networks and single computers to forma LAN, but even more to upgrade an existing LAN to enhance its availa-1)

bility. HAFTLAN is regarded as a low cost alternative to upgrading existing operating systems of several computers for the purpose of achieving computer network fault tolerance. For micro-based LANs HAFTLAN could serve as a low-cost back-up facility.

Due to its homogeneous structure (low-cost micros and low-cost links or buses of the same type) HAFTLAN should be suitable also for ana- lytical analysis with respect to reliability, system self-diagnosis and data transport including queueing. Fig. 1 shows a typical HAFTLAN

(square nodes) together with the system of "work-computers" (round nodes) i t serves.

Fig. 1

--- --- ---

7 ---00'

\

~

\ /

\ I

\

\

\

\

\

\

\

\

\ I I

I /

/ I

I / /

Typical HAFTLAN for single fault tolerance, excepting nodes for work computers (circles).

1) The by now well-established abbreviation for local area

In the subsequent reliability analysis i t is assumed that a HAFTLAN fails as soon as any work-computer can no longer cornrnunicate with any other or if HAFTLAN is disconnected;> In this context i t is understood that work-cornputers do not assist comrnunication between HAFTLAN nodes. 1)

To insist on HAFTLAN's connectedness means to insist on its capability of self-diagnosis. Hence, if e.g., in the case of a desired cornmuni- cation between nodes 6 and 7 of fi0. 1, node 1 is defective, and one cannot be sure if node 2 is operative or not, this should be regarded as a failure of the cornmunication between the nodes 6 and 7.

'.I'he (fat) ring structure of HAFTLAi.-J (excluding the links to the work- cornputers) is typical for single fault tolerance and, furthermore, according to the PMC model [ 1 ], typical for the 1-diagnosability of HAFTLAN. The case of using a pair of buses to which all the micros of HAFTLAN are attached by double links each is shown in Fig. 2, where the links to the work computers have been ornitted.

Fig. 2 HAFTLAN with a double bus for cornmunication.

1) Other criteria for "failure" would also make sense; however, they are not pursued here.

From a reliability point of view the more critical failures of the two buses of fig. 2 will be global failures affecting allthe attached nodes. Hence fig. 3 indicates a proper structure as a starting point for a standard reliability analysis.

Fig. 3 HAFTLAN with buses modelen as nodes (6) for reliability analysis.

Aspects of the distributed Operating system of a HAFTLAN are not dis- cussed in this report.

2 SYNTHESIS OF A FAULT TREE OR A SUCCESS TREE FORA HAFTLAN Let

X . .

= ~

1.-+J !

o,

if there is no path from node i to node j

else. ( 1 )

Then the fault tree function

rp

8

nodes 1 to n

8 and serving nW work-computers n

8+1 to n

8+nw is nH+nw-1 nH+nw

xs ⁼

^V V

x_T

h ^V

V

l-+J i<j

i=n8+1 j=i+1

i,jE{n

8+1 , . . . ,n

8+nw}

where XH indicates the disconnectedness of the HAFTLAN.

'l'o find X. l...,J . i t is necessary to disregard all the work-computer nodes except nodes i and j. What remains is a standard 2-terminal network connection problem.

1 )

Alternatively one can construct a success tree as the graph of the negated fault tree function

X . .

1.-+J

i:ound by de Morgan's law.

( 3)

Fora systematic reliability analysis of systems of any realistic size i t is, first of all, necessary to give names (numbers) to all those parts (modules,subsystems) which are tobe treated as separate entities

1) Nota standard name for the graph of a Boolean function describing the conditions under which the given systern works properly.

in the subsequent analysis. As done aboveone can give the numbers to nH to the HAFTLAN cornputers and the numbers nH+1 to

those work-cornputers which are directly connected with the actual HAFTLAN. Links get as indices the numbers of their adjacent nodes (cornputers viz. rnicros). Hence, for fault tree analysis, we have indicator variables

for the HAFTLAN nodes ,

• a subset of {X X } for links

1 n +1 '· · · ' n n + n

'H h' H W

between the work-computers and HAFTLAN, and

• x 1 , 2 'x2, 3' · · · 'xnH, 1 for the HAFTLAN links, unless the topology of HAFTLAN differs

structure.

frorn a simple ring to

In rnany cases i t will Xk remain, where k

=

be practical to rename the X. .

]. , J such that only

A _k =

~

l . . .

J.,J

1, for failure of component k

O , else . ( 4)

Usually, first all the single X. . of ( 2) and then x

5 are found as a

] . • J

disjunctive normal form (DNF). Hence the system fault tree is typically that of fig. 4.

The total number of fault tree input variables is the sum of - the number of HAFTLAN nodes and links (2 nH)

- the number of links from work-computers to HAFTLAN (2 nw).

L:

:?ig. 4 Typical HAFTLAN fault tree (corresponding to a DNF of

f).

The X. . and

X. .

l.-J l. .... J

Fig. 5 gives an exarnple extracted from fig. 1. Boxes contain component indices .

Fig. 5

.--•..,,6 2

....___.___,

.___~1,6 1 1----+---,1,5 5 4,5 4 ,4 .,___.,_~

3

Reliability block diagram for paths (through the HAFTLAN of fig. 1) between nodes 6 and 8.

'

·-·-·-·- --- ^---

^...

' ' _'

/ / /

1 - - - - . . - - - t

g

1

i 1 /2 '

/ /

'

Fig. 6 : Typical network for an X . . . (Interrupted lines for nincuts.)

l-+ J

For the fault tree of such networks one finds frorn fig. 6 x. .

=

l-+J a c a e g ^.i:: g

( 5)

Alternatively, from the minimal ties (paths)

v x x_x _{C d} g v xbx xfx C - g v xbx C x e v xcxdxexf ^{( 6)}

Tne r.h.s. of {6) is langer than that of (5) .Hence, not only for

reasons of numerical accuracy of reliability parameters tobe derived Erom X. . viz X. . , in this exami::-,le the fault tree is preferred since

l-+J l-+J L::

i t is easier to handle.

APPENDIX

To check the correctness of (5) and (6) we quickly show that their r.h.s. 's are the negations of each other. From (5), by de Morgan's law

A (

X

a

V

-

e ^{( 7)}

To keep intermediate expressions short i t is wise to "multir,ly" such pairs of brackets first where the absorption rule

X. v X.X.

=

l l J l

and the idempotence rule

X.X.

=

l l l

can frequently oe applied. For instance

'Y1

x

^-

=

^-

- - - -

Y2

^xf

⁼

- -

^-

"f

xf

- - - -

=

xf

Furthermore,

- - - - - -

v

x xf x

x xdx

.

( 8)

( 9)

X X ^V X X

C e C g ( 10)

xdxf ^V xdxg ( 1 1 )

X e ^V Xf)

X cxd ^{( 1 2)}

( 1 3}

?inally

-

=

l+J

which equals (6), q.e.d ..

( 1 4)

comrnent : In an application of (14) to fig. 5, prior to further

processing,one could make use of the idempotence law (9), which indi- cates that X , . . . ,X a g are not stochastically independent of each other.

3 AVAILABILITY OF A HAFTLAN

The availability,or rather its one'scomplement the unavailability,is best found from a form of the fault tree output function ~ that is written without Boolean operators and where on the way to a polynomial

form nowhere the idempotence rule X~= X

l i (9a)

would be applicable. For instance,in the well-known 2-out-of-3 system case the original Cf is

and one form useful for availability calculations is [2]

'I'he reason for preferring such forrns of

9'

( 1 7)

to system unavailability

u

( 1 8)

(A proof follows easily from E{X}=Pr{X=1}.) For instance,in the above 2-out-of-3 case

( 1 9)

If also negated variables are allowed, the Shannon decomposition

procedure is one of the best to get rid of Boolean operators [3]. For indicator variables of the type (4) the decomposition

( 20) can also be written with "+" instead of "v". This follows immediately frorn the defining equation of the disjunction (OR) operator for

indicator variables

( 21 ) Clearly, for

=

-

l : yjyk

=

For instance,for ~ of (15) with i

=

(22) Since

x

x

x

x

( 21 a)

(which is a trivial exarnple of (20)), the final

f

( 23)

By (18) or a trivial extension [4] systern unavailability is

( 1 9a) which equals (19) since A.

= -

=

l l l

As a further, less trivial exarnple we transforrn (5)

dence

+ xaxb[Xd + x_x_(X +XXX) j

a r g g c e

=

+ x xbx x x a c e g + x xbx xdx xt.x a c e g + xaxbxd

( 5a)

( 5b) 1~otice that this result must not be applied to the case of fig. 5. It

is indispensible that (9) be applied to (5a) first.

1) The same and an alternate derivation of a sum of products form according to an algorithm by Abraham are treated in [3] for (14).

4 FAILURE FREQUENCY, .MTBF, and .MTTR OF A HAFTLAN

Wherever the Markov model applies, system failure frequency and related parameters like MTBF (mean time between failures) and MTTR

(mean time to repair) can be calculated, at least in principle, though for large systems with many states the computational effort can be formidably high.

Therefore, here we use results of the basic theory of stochastic point processes [5]. Unfortunately this means (for realistic systems) that the components must be stochastically independent in their states good viz. bad. For practical work i t is best to start with system unavaila- bility

u

5 given in polynomial form,where besides the component unavai- labilities U. also the availabilities A. are allowed; where A.=U.=1-U.:

l l 1. l l

m n.

1.

US

=

I

^u .t. .

l,J

·.1.·ilen system f ailure frequency v

5 is ( see also [ 4])

m

= I

i=1

wnere

c.[(

1. _J= . 1

if if

u

.l., J

n. l

L

( 25)

( 26)

For instance,by (19a) for the 2-out-of-3 system

( 27)

Note that "mixed" polynomials with Uk and Ak are usually much easier to determine than "pure" ones; typically those with Uk only. However, for really good numerical approximations pure polynomials without Ak are clearly preferrable since, usually, the Ak ~ 1, whereas the Uk << 1.

S HAFTLAN SELF-DIAGNOSIS

The extremely important system self-diagnosis of a HAFTLAN can be executed according to a variety of methodologies or "philosophies".

Well applicable are

~equential diagnosis, since in most HAFTLAN applications repairs should be easily possible [1]

- comparison-tests using a neighboring component as a comparator [8]

- PMC-rnodel self-diagnosis [1].

Some details of the application of these three basic rnethodologies are

•Joing tobe discussed now. This discussion airns at being sufficiently 8elf-contained to render an immediate study of the fundamental

references [1] and [8] unnecessary.

Note that, as adapted to fault tree indicator variables, the test indicator variables are defined here as follows :

'1¹ • •

=

l,J

T . . k

=

if unit k, the test comparator, is down.

SEQUENTIAL DIAGNOSIS

The by now classical PMC [1] paper discusses at considerable detail the sequential k-diagnosability of ring structured systems. Theorem 5 of [1] says : "A single-loop system is sequentially (2m+tc) - fault diagnosable if the number n of units i t contains satisfies

n ^~ 1 + (m+1) 2 + "- (m+1) m integer, tcE{0,1}". (28)

For the HAFTLAN of Fig. 1 with n=nH=S nodes, (1) yields m=1, tc=O,such that sequential 2-diagnosability results. This can be verified in a very elementary manner as follows; see fig. 7.

Fig. 7

o} ^b}

The 2 basic double-faults of a 5-unit loop (ring).

(Defective nodes with double circles.)

If one finds a test sequence (see fig. 7a or b) with T. . +1

1. ' 1. = O, Ti+1 ,i+2 = O, Ti+2,i+3

=

then the test graph edge (vi+2,vi+3) shows to a defective unit. If one

1) All indices mod 5

cannot find a syndrome with the property (29), then one should look for one,where for some i (fig 7 b only)

T. _{l , l} . +1

=

There (vi+l'v-i+ 2 ) shows to a defective unit.

1) All indices mod 5

( 30)

COMPARISON TESTS

Comparison tests yield values of Boolean test indicator variables T . . k if units i and j are pairwise compared with the help of unit k

l,J,

which is an immediate neighbor both to u. and u. [8]; see fig. 8.

l .J

- - - /fi,j,I< - - -

"'

( immaterial) fest edge

Fig. 8 Test by comparison with a common neighbor as comparator.

Ti,j ,k ::::: T. . k.

J

'l'

For rings of less than 5 units this test does not work. For 3 units this is immediately obvious frorn fig. 9 in case T

21311=1.

Fig. 9

T, ,

1'\_ .,.,----

/ I

-- ... , / T1

= 1

',, 1 1

I

--r--- __ ,,,

T

=0v 1

I I

Cornparison tests in a ring of 3 units, where unit 1 is faulty.

(Solid lines for the cornmunication links.)

For 4 units, again with unit 1 faulty, fig. 10 shows the communication graph (solid lines) and the test graph (broken lines). From

T2,4,3=0 ' ^T

=

2, 4 , 1

one cannot decide which of the two units 1 or 3 is faulty, unless T 2, 4 , 1

=

Fig. 10

2

' ' ' _'

^,"'

'

'

,,

' /

,.,

/ ' ,

'

/

'

'

/ _,,""--

/ ... .... ....

' '

... ^{.... '\}

... ....

3

4

Comparison tests in a ring of 4 units, where unit 1 is faulty.

For 5 units {see fig. 1) fig. 11 shows the communication graph and the test graph.

Fig. 11

unit

Comparison tests in a ring of 5 units. The dotted lines are the edges of the test graph.

Ti.ssuming a comparison test to yield

0 if X. _J.

=

T . . k J.,J,

=

*

,

0 v1 , if Xk

=

Erom fig. 11, in the case of one faulty unit, the syndrorreis

T 1 , 3 , 2 = 1 ' T 2 , 4, 3 =O ' T 3 , 5 , 4 =O ' T 4, 1 , 5

=

Hence the simple decoding rule for the syndrome is : "Given the cyclic sequence of test results as in (32), look for two consecutive O's and the neighboring 1 's ! They are all correct results. The cornparator unit of the fifth test is faulty".

?MC-MODEL SELF-DIAGNOSIS

In case self-diagnosis is organized according to the well-known PMC model [11 node i tests node i+1; specifically node nH tests node n1 The test of node i comprises also the test of link (i-1,i) and of its links to work-computers.¹>

If the output T. ·+1 of the test t. ·+, of node i+1 by node i equals

l ₁l l 1l

1, node i+1 is assumed tobe bad , if T . .

1

=o

1,1+

be good. However, if node i is bad, its conclusions can be wrong. So we have two cases o f t . ·+

1 to consider, given that at most one node

l , l

is bad :

a) Node i is bad and erroneously "thinks'' node i+1 tobe bad too (see fig.12a).

b) Node i is good and node i+1 is bad (see fig. 12b).

In fig.12 the T. ·+

1 are written on top of the test edges.

J , J

0

a) b)

Fig. 12: Parts of the testgraph of a 1-diagnosable HAFTLAN.

In both cases the "leftmost" 1 gives the correct test result. Now i t is up to the HAFTLAN operating system to find this out and to communi- cate this to all the nodes of HAFTLAN "cutting" out the bad node first.

In this context high operational speed is usually desirable, since a

second node might fail soon after the first one. If a tested node

needs the testing one during the whole test, only about half the tests can be done simultaneously (assuming equal nodes with equal test

routines). Fora duration D of any test t. ·+

1 we have as HAFTLAN

l , l

self-test time DH

2 D 3 D

for for

even odd,

see fig. 13a) and b). Hence i t can make sense to add to a given

- - - - - - l

( 3 3)

<!···· ...•.

1 - - - - - - ⁰ I i

r----+---'---, ¹ - - - .

---.

L---:---_.J

b)

Fig.13 Pairs of nodes of ring network for individual test.

st nd rd

- - 1 , --- 2 , . . . 3 test phase;

a) even, b) odd number of nodes.

HAFTLAN another node exclusively for the purpose of achieving DH=2D.

For further details see [6].

6 ROUTING IN A HAFTLAN

Here rou ting means providing in forma tion as to which rou te a "message "¹¹ should use to go frorn one work-cornruter via HAFTLAN to another one.

Routing will generally be a multi-stage decision process in that in every·HAFTLAN node an incoming message can be told which link to pass next. This should include the possibility of detours because of

failures of HAFTLAN cornponents (links or nodes). This is a classical problem of randorn dynamic programming [7). However,in cases which are not too commplicated, use can be made of a reliability block diagram, especially if this has been drawn such that minimal paths were con- structed,starting from the source (of the rnessage) and airning at its destination.

For instance,if in the system of fig. 1 node 6 wishes tosend a message to node 7, fig. 14, the reliability block diagram for this mission,gives all the information needed for optimal routing, if

"optimal" means "looking for the shortest path first".

- - 1, 7 t---+---___J

1,2 i - - - ' ~ - - - -

1, 5 4,5 2,3

Fig. 14: Reliability block diagram useful for routing.

1) Here "message" is meant in the broadest sense, almost synonymous of information.

In fig.14 the convention for trying the shorter path first is : "If you come to a branching point of the type shown in fig.15 try that alternative whose first block is next''.

----__,~C1--- B

Fig.15 Convention for sequencing of alternatives.

So, after reaching the branching point Bin fig.15, first try the alternative beginning with component c₁, then that beginning with component

c

c

If one cannot advance any further on a path (because of a component failure) then one jumps back to the last branching point.

Fig. 16 shows this type of routing algorithrn for fig.14. Note that node 2 and link (2,7) rnust appear thricein fig. 16 lest one risk infinite looping.

It is a matter of much deeper analysis to decide where to store which parts of such routing algorithms.

Start

no

yes no Use link (1, 6)

no yes

yes Use link (1, 7)

no yes

Usenode 2

no

Use link(2

End

no

no

yes Use link( 1 2)

yes Usenode 2

yes no

yes Use link(1,S) no

no

Use link(2,3J

Report tha t ^no

pafh is viable

no

•

. .

7 REFERENCES

[O] Fruitful discussion with H. Bähring, Hagen.

[1] Preparata F., Metze G., Chien R.: On the connection assignment problem of diagnosable systems. Trans. IEEE, vol. EC-16

(1967), 848-854.

[2] Schneweiss W.: Calculating the probability of Boolean expression being 1. Trans. IEEE, vol. R-26 (1977), 16-22.

[3] Schneeweiss W.: Disjoint Boolean products via Shannon's expan- sion. Trans. IEEE, vol. R-33 (1984), 329-332.

[4] Schneeweiss W.: Cornputing failure frequency, MTBF & MTTR via rnixed products of availabilities and unavailabilities.

Trans. IEEE, vol. R-30 (1981), 362-363.

[5] Cox D., Isharn Y.: Point processes. London Chaprnan ^& Hall 1980.

[6] Schneeweiss W.: On the duration of systern self-diagnosis in local area networks. Elektron. Rechenanlagen 26 (1984), 298-304. (Full text in Gerrnan.)

[7] Schneeweiss C.: Dynamisches Programmieren (Dynarnic Programrning) Würzburg : Physica, 1974.

[8] Maeng J., Malek M.: A cornparison connection assignrnent for self- diagnosis of rnultiprocessor systerns. Proc. 11th

FTCS, IEEE-press 1981.