2 Retriev al in CBR

ValidatedRetrieval

inCase{BasedReasoning

Ev angelos Simoudis James Miller

DigitalEquipmentCorporationCambridgeResearchLab

CRL90/2December12,1990

Abstract

Wecombinesimpleretrievalwithdomain-specicvalidationofretrievedcasestoproduceausefulpracticaltoolforcase-basedreasoning.Basedon200real-worldcases,weretrievebetweenthreeandsixcasesoverawiderangeofnewproblems.Thisrepresentsaselectivityrangingfrom1.5%to3%,com-paredtoanaverageselectivityofonly11%fromsimpleretrievalalone.

cDigitalEquipmentCorporation1990.Allrightsreserved.

1

1 In tro duction

Wehavecombinedsimpleretrieval(basedonthesimilarityofsurfacefeatures)withdomain-specicvalidationofretrievedcasestoproduceausefulpracticaltoolforcase-basedreasoning.Startingwithacasebaseof200real-worldcases,wehavenarrowedourconsiderationtobetweenthreeandsixcasesoverawiderangeofnewproblems.Thisrepresentsaselectivityrangingfrom1.5%to3%,comparedtoanaverageselectivityofonly11%fromthissamecasebaseusingretrievalwithoutvalidation.Weareapplyingthesametechnologytoalargercasebaseinadierentdomain,andhavedeployedarelatedtoolwithamuchlargercasebaseforactualuseintheeld.Ourworkbeginswithareal-worldproblem:acomputermanufacturer'sdiagnosisofsystemsoftwarefailures.Inthisdomain,diagnosticknowledgeexistsinseveralforms:manuals,courses,productionrulesystems,andknowl-edgebases.Butthepredominantstartingpointincurrentuseisasetofdatabasescreatedbyrecordingsuccessfullydiagnosederrorconditions.Inordertodiagnoseanewfailure,non-expertspecialistsretrievefromadatabaseprevi-ouslysolvedcasesthatappearsuperciallysimilartothenewproblem.Theythenattempttoverifythesimilaritybyperformingtestsonthenewprob-lemandcomparingtheresultswiththoseofeachretrievedcase.Whentheybecomeconvincedthatapreviouscaseissubstantiallythesameasthenewproblem,theyexaminetheresolutionoftheoldcaseandreportit(possiblyamendedoreditedtomorecloselytthenewproblem)tothecustomer.Onlyinrarecasesareexpertsrequestedtoexamineproblems|mostareresolvedfromtheexistingdatabase|andthesolutionsarethenaddedtothedatabase.Thisexistinghumansystemisaconscioususeofcase-basedreasoning(CBR)techniqueswehaveimprovedthesystembyaddingtoitanautomatedtoolusingresultsfromAIcase-basedreasoningsystems.Inordertoproduceatoolofpracticalvaluewewereforcedtoexaminemorecloselythetaskofretrievalincase-basedreasoning.Basedonourexperienceweproposeanex-tensiontocurrentsystems,validatedretrieval,thatdramaticallyreducesthenumberofcasespresentedtothereasoningcomponent(humanorautomated)ofacase-basedsystem.Validatedretrievalreliesondomain-specicknowl-edgeabouttestsusedtocomparecasesretrievedfromthecasebasewithnewlypresentedproblemcases.Knowledgeabouttherelationshipsamongthevarioustestsiscapturedinavalidationmodelwhichweimplementasase-

22RETRIEVALINCBR

manticnetwork8].Inordertobuildourvalidationmodelwearefacedwithaclassicknowledgeacquisitiontask.Byperusingexistingdatabasesusedbyspecialistsweareabletoacquirethisknowledgewithareasonableamountofeort|andwithonlyasmallinvestmentofspecialists'time.

2 Retriev al in CBR

CBRsystemsrstretrieveasetofcasesfromacasebaseandthenreasonfromthemtondasolutiontoanewlyposedproblem.Existingsystems(1],2],3],4],9]and10])maketwoassumptionsabouttheinitialretrievalofcasesfromthecasebase:1.Veryfewcaseswillberetrievedfromthecaselibrary.2.Theretrievedcasesarerelevanttotheproblembeingsolved.Inmanypracticalapplications,retrievalaloneissucienttosolvethedif-cultpartofatask.Forexample,inourdomainofdiagnosisofcomputersoftwarefailures,specialistscaneasilyrespondtocustomerproblemsiftheycanquicklylocateafewsimilarcasesfromtheircollectivepastexperience.Forthisreason,wehaveconcentratedontheretrievalaspectofcase-basedreason-ing.InMBRTalk10],also,theessentialtaskisretrievalthe\reasoning"componentconsistsofmerelypassingtheretrievedinformationdirectlytoanoutputunit.

2.1 Related W ork

ClosesttoourownworkistheworkofKotononcasey5],aCBRsystemwhichhasbeenappliedinthedomainofmedicaldiagnosis.caseyhasajusticationcomponentwhosegoalistodeterminewhetherthecausalexpla-nationofaretrievedcaseappliestoanewproblem.Thisfrequentlyallowscaseytoavoidinvokingitscausalmodelwhencreatinganexplanationforanewcase.casey'sjusticationphaseissimilartoourvalidationphase.Butthereisanimportantdierencebetweenthesetwosystemsarisingfromdierentassumptionsabouttests.caseyreliesonpreciselytwotests(EKGandX-rays),bothofwhichareinexpensiveandnon-invasive.Bothofthesetestsareperformedpriortotheretrievalphaseandtheresultsareusedtoprovidesurfacefeaturesfortheretrievalalgorithm.Bycontrast,thereare

2.2TwoPhases:RetrievalandValidation3

literallyhundredsofteststobeperformedinourdomainsanditisfartooexpensivetoperformalloftheminadvanceofinitialcaseretrieval.Asaresult,oursystemsdevoteattentiontominimizingthenumberofteststhatareperformed.Wenotonlyperformtestsincrementallyandcachetheresults,butalsoemployknowledgeabouttheteststhemselvestoreducethenumberoftestsperformed.TheCBRsystemchef2],whosedomainisChinesecooking,alsohasajusticationcomponent.Inordertojustifyeachretrievedcase,chefusesbackwardchainingrules.Whileourvalidationmodelisnotappropriatetothisdomain,wehaveexaminedandrejectedtheuseofrulesforourvalida-tionmodels.Thisdecisionisbasedonthedicultyofacquiringtheexpertknowledgeneededtocreatelargerulesets,especiallyincomparisontothesimplicityofconstructingourvalidationmodels.Thesemodelsarecapturedbyasemanticnetworkthatrepresentsgroupsoftestsandinformationabouttherelationshipsbetweenthegroups.Furthermore,aswithcasey,chefdoesnotusetheresultsofcomparisonswithearliercasestopruneitssearchforrelevantcases.Theswale6]systemconcentratesprimarilyonmodifyingtheexplanation(containedinanexplanationpacketorXP)ofaretrievedcasetomatchanewproblem.Nonetheless,ithasasubcomponent,xpaccepter,thatjustiestheapplicationofaretrievedXPtoacurrentsituation.TheaccepterveriesanXPbydeterminingifitcanbelievetheapplicabilitychecksthatarepackagedwitheachXP.Eachsuchtestissimilartothetestsassociatedwithourvalidationmodel.Becauseofthesmallnumberofcases(eight,byourcount)xpaccepterneveraddressedtheissuesofscalewhichareourmajorconcern.Thus,swaleneverdevelopedajusticationmodeltorelatethevariousapplicabilitycheckstooneanother.

2.2 Tw oPhases: Retriev al and Validation

Ourgoalistotakeasizablepre-existingcasebasealongwithanewproblemandproduceasmallnumberofrelevantcases.Likeourhumanspecialists,oursystemsperformdiagnosisintwophases:

Retriev al:

itposesaquerytothecasebaseusingasubsetofthefeaturesthatdescribethenewproblem.

43THEVALIDATIONMODEL

Validation:

itfollowsthevalidationprocedurefromeachretrievedcasetodetermineifitappliestotheproblemathand.Thegoaloftheretrievalphaseistoextractfromthecasebasethosecasesthatappeartoberelevanttothenewcase.Sincethecasebaseislarge,andwehavebeeninterestedprimarilyinsequentialimplementations,itisimportantthatthecasebasebeorganizedinawaythatpermitsecientsearchbasedonsurfacesimilarities.Forthisreason,weorganizethecasesintoageneralizationhierarchy(usingunimem7]).Theretrievalphaseconsistsoftraversingthegeneralizationhierarchytondaclosematchtothenewproblem.Theresultofthistraversaliseitheranindividualcase(aleafnode)orasetofcases(aninternalnodeinthehierarchy,returnedasallofthecasesindexedunderthatnode).Unlikethosesystemsthatrelyexclusivelyonunimemforcaseretrieval,wedon'tnetuneunimemtoreducethenumberofcasesretrieved.Thevalidationphasethenconsiderseachoftheretrievedcasesandat-temptstoshowthatthecaseisrelevanttotheproblemathand.Associatedwitheachcaseinthecasebaseisasetoftestsandtheirresultvaluesthatmustbemetforthestoredcasetobevalid.Wecallthesetoftestsandvaluesavalidationprocedure,andeachelementofthisset(i.e.asingletest/valuepair)iscalledavalidationstep.Thetestsareappliedtotheactualproblemandtheresultsarecomparedwiththeresultsinthecase.Basedonthiscomparisonboththecurrentcaseandotherretrievedcasescanberemovedfromfurtherconsideration.Onlywhenallofthetestsforagivencasearesuccessfullymatchedagainstthecurrentproblemisthecasereportedasacandidateforareasoningcomponent'sconsideration.(Inotherdomains,itmaybepossibletoassignweightstoindividualtestresultsanduseathresholdoraveragingschemefordecidingwhetherornottorejectthecase.)

3 The Validation M odel

Thevalidationphaseofourmethodisstraightforwardiftheindividualvalida-tiontestsaresimpleandself-contained.Unfortunately,inourdomains,andprobablyinmostreal-worlddomains,thisisnotthecase.Ineachdomainwehavestudied,wehavefoundthatthetestsareinterrelatedinawaythatisnotevidentinadvance,andwehavebeenforcedtofacetheknowledgeacquisi-tiontaskhead-on.InSection3.2wedescribeamethodologyforacquiringthisknowledgeabouttests.Wehavesuccessfullyusedthismethodologytodevelop

3.1Whatisavalidationmodel?5 validationmodels:structuresthatcapturemuchofanexpert'sknowledgeinawaythatmakesiteasyforthevalidationphasetoprocessthetestsitrequires.

3.1 What is avalidation mo del?

Ratherthanrequireacompleteandaccuratedescriptionofeachtestusedbythespecialist,wecapturetheoverallstructureofthetestspaceitself.Theresultingstructure,ourvalidationmodel,consistsofrelatedgroupsoftestsandinformationabouttherelationshipbetweenthegroups.Forexample,ifwewanttoknowwhyahouseishot(theproblem),wemayrstwanttoseeiftheairconditionerisworkingbutthisrequiresustondoutifthehousehasanairconditioner.Inthisexample,thedesiredtest(istheairconditionerworking?)isrelatedtoanothertest(isthereanairconditioner?).Thisknowledge,aswellasknowledgeabouttheimportantoutcomesandimplicationsofatest,iscapturedbythevalidationmodel.Wehavechosentorepresentthisknowledgeintheformofasemanticnetworkwhosenodescorrespondtosetsoftestsandwhosearcsindicaterelationshipsbetweenthesesets.

3.2 Creating aV alidation M odel

Webuildourvalidationmodelsbyrstexaminingexistingdatabasesthatareusedbyhumanspecialists.Thesedatabasesmaybeeitherformalized(asinthecaseofourWPS-PLUS 1system)ormerelyinformalnotespreparedbythespecialistsfortheirownperusal(asinthecaseofourVAX/VMSsystem).Inourtwocase-basedsystems,theexistingdatacontainsatextualdescriptionofthestepsthatthespecialistsusedtoverifyahypotheticalexplanationoftheproblem.Inconstructingthevalidationmodel,itisourgoaltocapturetheinterrelationshipsbetweenthevalidationtests.Asaresult,wehavebuiltvalidationmodelsthatcorrespondtoaparticularcasebaseby:1.Readingthevalidationproceduresofeachcaseandbuildingalistofallthevalidationstepsusedintheentiredatabase.Intheprocessofreadingthedatabaseandpreparingthislist,theimplementordevelopsasenseoftheunderlying(butunstated)relationshipsbetweenteststhatarementionedinthedatabase.

1DEC,VAX,VMSandWPS-PLUSareregisteredtrademarksofDigitalEquipmentCorporation.

64ANEXTENDEDEXAMPLE

2.Examiningtheresultinglist,lookingforgroupsofteststhatappeartoformrelatedsets.Organizingthelistprovidesabasisfordiscussionwithdomainexperts,whohelp\debug"theproposedorganization.3.Reningthestructureofthelistthroughknowledgeacquisitionsessionswithdomainexperts.Duringthesesessions,signicantrangesoftestresultsareidentied,asareinferencesfromtheseresultsthateliminatetheneedtoperformothertests.Thatis,adependencygraphbasedontestresultsisdeveloped.4.Iteratingtheabovetwostepsafterconsultingadditionalinformationsuchasmanualsandcodedocumentation.Thestructureofthedomainbecomesclearerateachiteration.(Wehavefoundthatthreeiterationsaresucienttoproduceausefulstructuring.)Thenalvalidationmodelconsistsprimarilyofentriescorrespondingdirectlytoinformationthatappearsintheoriginaldatabase.5.Integratingthetestsetsintothestructurederivedinthepreviousstep.Thisintegrationmakesexplicittheprerequisitesofeachtest,aswellasprovidingalternativewaysofobtaininginformationordinarilyprovidedbyaparticularcriticaltestincaseswherethattestcannotbeperformed.

4 An Extended Example

Inordertounderstandthevalidatedretrievalprocess,considerthefollowingexample.Ourdomainisautomobilediagnosisandrepair,andweassumeanexistingcasebasewithitsassociatedvalidationmodel.Wearegiventhefollowingcase:

NEW CASE mak e

:MAZDA

mo del

:626

mo del year

:1985

engine typ e

:2.0LEFI

miles

:50,000

problem

:enginedoesnotstart.Theretrievalphaseusesthemake,model,problem,andapproximateyearofmanufacturetosearchthroughacasebaseofpreviousautomobileproblems.Basedonthesesurfacefeatures,weretrievethreecasestobevalidatedbeforepresentationtoareasoningcomponent:

7

CASE 1 mak e

:MAZDA

mo del

:626

mo del year

:1988

engine typ e

:2.0LEFI

miles

:10,000

problem

:enginedoesnotstart.

validation

:Thefuelinjectorwasclogged.Fuelwasnotdeliveredtothecombustionchamberfortheenginetoignite.Forthisreasontheenginecouldnotstart.

solution

:cleanedthefuelinjector.

CASE 2 mak e

:MAZDA

mo del

:626

mo del year

:1984

engine typ e

:2.0L

miles

:60,000

problem

:enginedoesnotstart.

validation

:Thecarhadafaultygaspump.Fuelcouldnotbede-liveredtothecombustionchamber.Forthisreasontheenginecouldnotstart.

solution

:Replacedthegaspump.

CASE 3 mak e

:MAZDA

mo del

:626

mo del year

:1987

engine typ e

:1.8L

miles

:20,000

problem

:enginedoesnotstart.

validation

:Aleakexistedinthegasline.Fuelcouldnotbedeliveredthroughthefuelline.Forthisreasontheenginecouldnotstart.

solution

:Fixedtheleak.Thevalidationmodelcontains(atleast)thethreeteststhatarereferencedbythesecases:\checkifafuelinjectorisclogged",\checkifthegaspumpisworking",and\checkifthereisaleakinthefuelline".Therstoftheseisactuallycomposedoftwosimplertests:atestforfuelpresentinthereservoiroftheinjectorandatestforfuelexitingtheinjector'snozzle.Ifthereisnofuelintheinjectorthenwecandeducethattheinjectorisnotatfault.Rather,theproblemliesearlierinthefuelsystem|eitherinthepumporthefuelline.ThesystemrstattemptstovalidateCase1byrepeatingthevalidationstepsfromthatcase.Thatis,wewishtotestifthefuelinjectorisclogged.Intheprocessofperformingthistwo-steptestweactuallyacquireknowledgethatisrelevanttoCases2and3:ifthefuelreservoirisnotemptywecaneliminatebothcasesifitisempty,wecaneliminateCase1.Thisrelationship

85RECENTRESULTS

isencodedinthesemanticnetworkthatrepresentsourvalidationmodelandisusedinthevalidationphase.Inthebestcase,thisvalidationmodelallowsustoreducetheworkrequiredtovalidatecasesfromfourteststotwotestsandsimultaneouslyreducesthenumberofcasestobeconsideredbythereasonerfromthreetoone(selectivityof33.3%).ThersttestisforanemptyfuelreservoirifthereservoirisfullthenCases2and3areeliminated.Wethentestthenozzleforfuelexiting.Ifnofuelleavesthenozzle,thenCase1ispresentedtothereasonerbutiffuelisleavingthenozzlewe,unfortunately,eliminateCase1aswellandleavethereasoningcomponenttoitsownresources.Theworstcaserequiresallfourtestsandprovideseitherzerooronecasetothereasoner.

5 Recen tR esults 5.1 An Op erating System: VMS

TherstsystemwedevelopedisusedforthediagnosisofdevicedriverinducedcrashesofDigital'sVMSoperatingsystem.Theknowledgeaboutsurfacefea-tureswasobtainedprimarilyfromDECinternalpublicationsandwascom-plementedbyanexpertfromtheVMSsupportteamduringthreeknowledgeacquisitionsessions.Ittookatotalof84hourstoacquirethedomainspe-cicknowledgeaboutsurfacefeatures.Basedonthisinformation,thedomainknowledgeusedbyunimeminordertoorganizethecasesintoageneralizationhierarchywasimplementedinvedays.Ittookanadditionalfourdaysofreadingvalidationproceduresinthedatabasetodevelopavalidationmodelfordevicedrivers.Inaddition,fourmoreknowledgeacquisitionsessions,lasting40hours,wereneededtoreneandimprovethevalidationmodel.Encodingtheactualvalidationmodeltookabout80additionaldays.Thetotalnumberofdaysspentonknowledgeacquisitionanddevelopmentisshownbelow:ActivityPersonDaysKnowledgeacquisition20Development85Sincethiswasourrstattempttobuildacasebaseandvalidationmodel,thesenumbersaremuchlargerthanweexpectforsubsequentsystems.Our

5.2AWordProcessingSystem:WPS-PLUS9worktodateonthesystemdescribedinSection5.2appearstoconrmthisexpectation.Thesystemwasevaluatedusingacasebaseof200casesthatwereobtainedfromnoteswrittenbyspecialists.Thesurfacefeatureretrievalphaseofthesystemwasevaluatedbypresentingeachofthe200casestotheretriever(asnewproblems)andpreparingahistogramofthenumberofcasesretrieved.unimemprovidesamechanism,knownasretrievalweights,fortuningitsre-trievalcapabilities.Aftersomeexperimentation,wediscoveredthattheuseoflargerretrievalweights(i.e.morestringentmatchingcriteria)causedthere-trievertomissmanyrelevantcasesand,inmanyoccasions,tofailtoretrieveanycasesatall.Withlessstringentcriteriathisproblemwasrectied.How-ever,manyoftheretrievedcaseswerenotrelevanttotheproblem.Withtheoptimalweighting,wewereabletoretrieveonaverage22casesperretrieval(11%).Thevalidationphase,however,wasabletoreducethisnumberofcasestoanaverageof4.5casesoutof200(2.25%).Inaddition,wepresentedthreenewcasestothesystem.Basedonsurfacefeaturesalone,weretrieved20,25,and16cases(10,12.5,and8%selectivity).Thevalidationphasereducedthisto3,5and3cases,respectively(1.5,2.5,and1.5%selectivity).Ourexpertsconrmthatthesevalidatedcasesaretheonlyonesrelevanttotheproblemspresented.

5.2 AW ord Pro cessing System: W PS-PLUS

Thesecondsystemperformsdiagnosisofcustomerproblemswiththewordprocessingcomponentofanoceautomationproduct.During15hoursofknowledgeacquisitionsessions,theknowledgeaboutsurfacefeatureswasob-tainedfromasupportengineerfortheproduct.Itthentookanadditionalvedaystoencodethedomainknowledgeforusebyunimem.Thevalidationmodelwasobtainedfromthevalidationproceduresofthecasesinthedatabase,aninternalpublication,and10hoursofknowledgeacquisitionwiththesameengineer.Whiletheworkisnotyetcomplete(only50outof340caseshavebeenencoded),ithastakenonly10daystoimplementthevalidationmodel.Thissystemisstillunderdevelopment.However,thetimewespentonknowledgeacquisitionanddevelopmentisshownbelow:

106SCOPEOFWORKActivityPersonDaysKnowledgeacquisition5Development10Thissystemwasevaluatedusingacasebaseof340cases.RepeatingthesameexperimentperformedwiththeVMScasebaseledtoanaverageof26casesperretrieval,or7.6%selectivity.Thevalidationphasereducedthistotwocases,or0.58%selectivity.Sincethevalidationmodelforthiscasebaseisnotyetfullyencoded,wehavenotpresentednewproblemstothesystem.

6 Scop eof W ork

Ourvalidatedretrievalmethodcanbeappliedinmanytypesoftasks.Thebasicrequirementsare:anexistingdatabaseofpreviouspracticalexperi-enceasetofquickteststhatservetoreducethesearchspaceatlowcostasetofmoreexpensiveteststhatcanfurtherreducethesearchspaceandanunderstandingoftherelationshipsbetweentheexpensivetests.Wehaveidentiedfourareasofpotentialinterest,butwehavelimitedourimplementationworktotherstofthese:

Diagnostic tasks

.Asshownintheexample,weusethesymptomsofaproblemasthesurfacefeaturesfortheretrievalphase.Thevalidationproceduredescribeswhichteststoperforminordertodetermineifthecaseisrelevanttothenewproblem.

Design tasks

.Thesurfacefeaturesarespecicationsthatadesignmustsatisfy.Thevalidationproceduresverifythataproposeddesignmeetsthespecication.

Sales tasks

.Thetechniquecanbeusedtohelpidentifysalesprospectsforanewproduct.Thesurfacefeaturesarethecharacteristicsofacustomersuchas:sizeofbusiness,typeofbusiness,locationofbusiness.Eachvalidationproceduredescribesthecustomer'srequirementsthatweresatisedinaprevioussale.Thevalidationmodelincludesteststhatdeterminewhetherornotacustomerneedsaparticulartypeofproduct.

11

Managemen ttasks

.Thesetasksincludeaccounting,creditanalysis,investmentdecisions,andinsuranceunderwriting.Ineachofthesear-eas,specialistscanidentifyeasilyrecognizedfeaturesintheirproblemdomain(typeofcompany,size,etc.)thatallowrapidretrievalofsimilarsituationsencounteredinthepast.Theythenhavemoredetailedteststhatcanbeapplied(debt/equityratio,paymenthistory,typeofclient,balancesheets,etc.).

7 Conclusions

Ourworkhasconcentratedexclusivelyontheissueofcaseretrieval.Acare-fulstudyoftwoapplicationsinwhichpeopleconsciouslyusecaseretrievalhasshownthatretrievalbasedsolelyonsurfacefeaturesisnotsucientlydiscriminatingforusewithlargecasebases.Itresultsinlargenumbersofcasesreturnedtothereasoner,eachofwhichmustthenbefurtherexaminedatgreatexpense.Addingavalidationphasethatusesknowledgeofdomain-specicteststoprunetheretrievedcasesdramaticallyreducesthenumberofcasesthatmustbeexaminedbythereasoner.Wehavefoundthatacquiringknowledgeaboutdomain-specictestsisaidedbyaninitialperusaloftheexistingdatabaseusedbyspecialists.Withareasonableamountofeort,andwithonlyasmallinvestmentofspecial-ists'time,thisinformationcanbecapturedinavalidationmodelrepresentedasasemanticnetwork.Wehaveusedthismethodologytoproducetwosys-tems.Oneofthesesystemshasbeensuccessfulinpractice,andtheother(incomplete)systemislikelytobeequallyuseful.Whiletheburdenofknowledgeacquisitioninourmethodologyissmallcomparedwithothermethods,itisnotnegligible.AutomatingthisworkbycombininganaturallanguagesystemtoanalyzeexistingdatabaseswithAI-assistedstatisticalcomparisonofsurfacefeaturesprovidesafertileareaforfurtherinvestigation.Furthermore,wesuspectthatacarefulstudyofsuchasysteminpracticewillrevealvalidationteststhataresucientlycommonthatitmaybereasonabletopromotethemtosurfacefeatures,leadingtoasystemwithbetterretrievalcapabilities.Thisanalysis,whichmustrstbeperformedmanuallytovalidateourassumption,isitselfanexcellentareafortheapplicationAImethods.

Ac kno wledgemen ts

127CONCLUSIONS

TheauthorswishtothankProf.DavidWaltzforhishelpwiththisresearch.Inaddition,wehavereceivedhelpfulcommentsfromMarkAdler,AndrewBlack,DaveHanssen,RoseHorner,andCandySidner.Wearealsoindebtedtotheengineerswhohavegiventheirtimeandknowledgetohelpusunderstandthetwodomainswehavestudied:theDigitalSupportEngineersatColoradoSprings,ColoradoandSpitbrook,NewHampshire.

REFERENCES13

References

1]RayBareiss,KarlBranting,andBrucePorter.Theroleofexplanationinexemplar-basedclassicationandlearning.InProceedingsofCase-BasedReasoningWorkshop,1988.2]KristianHammond.Case-BasedPlanning:AnIntegratedTheoryofPlan-ning,Learning,andMemory.PhDthesis,YaleUniversity,1986.3]JanetL.Kolodner.Reconstructivememory:Acomputermodel.CognitiveScienceJournal,7:281{328,1983.4]JanetL.Kolodner,Jr.RobertL.Simpson,andKatiaSycara-Cyranski.Aprocessmodelofcase-basedreasoninginproblemsolving.InProceedingsoftheInternationalJointConferenceonArticialIntelligence,1985.5]PhyllisKoton.Usingexperienceinlearningandproblemsolving.PhDthesis,MassachussettsInstituteofTechnology,1988.6]DavidLeake.Evaluatingexplanations.InProceedingsoftheSeventhNationalConferenceonArticialIntelligence,1988.7]MichaelLebowitz.Experimentswithincrementalconceptformation:Unimem.MachineLearning,2:103{138,1987.8]M.R.Quillian.Semanticmemory.InMarvinMinsky,editor,SemanticInformationProcessing,pages227{270.MITPress,1968.9]EdwinaRisslandandKennethAshley.Hypotheticalsasheuristicdevice.InProceedingsoftheFifthNationalConferenceonArticialIntelligence,1986.10]CraigStanllandDavidWaltz.Towardmemory-basedreasoning.CACM,29:1213{1228,1986.

14ACREATINGTHEVMSVALIDATIONMODEL

A Creating the VMS Validation Mo del

Thisappendixdescribes,indetail,themethodusedtoconstructthevalida-tionmodelfortheVAX/VMSdevicedriverdiagnosissystem.EachsubsectioncorrespondstooneofthestepsdescribedinSection3.2.Forexpositorypur-poses,ofcourse,wehaveremovedanumberofintermediatestepsandshowonlyselectedresults.

A.1 Reading the Initial Data Base

WebeginwiththedatabaseincurrentusebyCustomerSupportSpecialists.Thisdatabase(actually,aninformaltextual\notesle")consistsofabout150entries.AtypicalentryisshowninFigure1.Atthisstagewenotonlyreadtheseentries,butwetidythemupabit.Forexample,someoftheseentrieswereincompleteandweremanuallyexpandedintomultiplecasesforourcasebase,resultingin200entriestobestudied.Theprimarypurposeofthisinitialreadingistocreatealistofalloftheteststhatareexplicitlymentionedbythespecialistsintheirexplanationsoftheresolvedcases.FromthecasepresentedinFigure1weextractthreetests:1.RetrieveProcessPCB2.CheckJIBAddress3.CheckCountForthisparticulardatabase,wederivedaninitialsetofabout100tests.

A.2 Grouping the Tests

Readingthroughthisdatabaseledtoanaturaldecompositionofthetestsbygroupingthemaccordingtodatastructuresmentionedinthecases.Thus,wegroupedtestsrelatedtotheJIB(jobinformationblock)together,joiningthetests\CheckJIBAddress"and\CheckCount"fromthecaseshowninFigure1.Inthisparticulardatabase,wegroupedthetestsintoroughlysevensetsoftests.Bycontrast,inourworkwiththeWPS-PLUSdatabase,thenaturalgroup-ingwasalongfunctionallines.Whilewehavenormevidence,itseemslikelythatsuch\natural"groupingswilloccurinmostsizabledatabases.

A.2GroupingtheTests15

Versionofsystem:VAX/VMSV4.2Reasonforbugcheckexception:ACCESSVIOLATIONProcesscurrentlyexecuting:

SP=>8018E7AC00000004

8018E7B07FFE7DE4

8018E7B4FFFFFFFD

8018E7B80000026A

8018E7BC00000000!THISSHOULDHAVEBEENON

PREVIOUSLINE

!ANDVISAVERSA

8018E7C000000001

8018E7C400000005!BEGINNINGOFSIGNALARRAY

8018E7C80000000C!ACCESVIOLATION

8018E7CC00000004!WRITEACCESSPROTECTION

8018E7D000000020!VA

8018E7D480004A4EIOC$BUFPOST+017

8018E7D804040000

8018E7DC80004A6DIOC$BUFPOST+036

8018E7E0800CD108DZDRIVER+118thereasonforthisbugcheckisbecausethepideldoftheIRPhasbeenzeroed.INpostprocessingthesystemtriestogivethebueredbytecountbacktotheprocessthatdidtheio.HEdoesthisbyusingthePIDeldoftheirptoindexintothepidtablethengetsthePCB(nullprocessinthiscase)andgetstheJIBaddress.THEJIBaddressforthenullprocessis0.THEsystemthengoestotheJIB$LBYTCNToset(20)fromthebaseoftheJIB(0inthiscase)andtriestoaddbackinthebytecount.INthiscasewegetanaccessviolation.haveresponsefromVMSdevelopementthatthereis/wasabugxedin4.2ithasbeenmentionedtotryandlowerthebaudrateonterminalsasaworkaround.Figure1:AnInitialDataBaseEntry

16ACREATINGTHEVMSVALIDATIONMODEL

A.3 Rening the Test-Group Structure

Armedwiththisgroupoftests,wereturnedtothespecialistswhohadpreparedthedatabase.Withtheirhelp,welearnedthemeaningofeachofthetestswehadidentied.Thespecialistspointedoutthatournaturalgroupingsdidn'tcorrespondtogroupingsthattheywouldhavemade|primarilybecausetheyhada(non-articulated)setofabstractionsthatourgroupingsdidnotmakeclear.Byreviewingourgroupingstheywereforcedtoverbalizetheabstractionsthattheytookforgranted,andwewereabletosortourtestsinto15groupingsthatreectedthespecialists'notionofthecorrectabstractionsfortheirdomain.InthecaseoftheVAX/VMSdatabase,thespecialiststypicallysubdividedourgroupingsintothreedistinctcategories:thosethattestfortheexistenceoftheexpectedlocationinmemory,thosethattestedtheinternalconsistencyofthedatastructure,andthosethatprobethedatastructuresforspecicvalue.InthecaseofthetestsextractedfromFigure1,\CheckJIBAddress"isintherstcategory,thespecialistspointedoutfortheneedforanadditionaltestwehadnotidentiedthatperformedtheconsistencytest,and\CheckCount"isinthethirdcategory.Thus,ourinitialmodelofsevenstructure-basedtestgroupswasrened,throughinteractionwithspecialists,into15groupswithalayeredstructure.Testingalwaysproceedsfromonelayertoanother,startingwithtestsforexis-tenceofthedatastructure,proceedingtotestsontheconsistencyofthedatastructure,andnallyontoprobingspecicvalueswithinthedatastructure.

A.4 Acquiring Additional Kno wledge

Thisstepisprimarilyaniterationoftheearlierone,buttwounusualitemsaroseduringthisphasefortheVAX/VMSsystem.Therstwastherealiza-tionthatcertaintermsusedinthedescriptionswerenotreectedinanyofthetestsourgroupingswehadalreadyderived.Inthisparticularcase,therewascontinualuseofphraseslike\duringASTdelivery,"and\duringpost-processing."Examinationofadditionalmaterial(materialfromaninternalDECcoursesuggestedbythespecialistsasagoodstartingpoint)revealedthatthesereectedtheprocessingstagesofthesystemwewerediagnosing.Ourinitialbreakdownhadbeenalongdatastructureboundaries,butanor-thogonal,equallyimportantstructuringispossiblealongtemporallinesbasedonthesestages.Itisthesetwoalternativedecompositionsthatwereferredto

A.5IntegratingTestsintotheModel17

inSection3.2asthe\structureofthedomain."Furtheriterationwiththecoursematerialsandthespecialistsrevealsamuchricherstructure,consistingofmultiplelevelsofabstraction.Withthetemporaldecompositiontakenasprimary,Figures2and3showtwodierentlevelsofabstraction.Figure2istakenataveryabstractlevel,showingthefourmajorprocessingstagesandthedatastructuresusedateachstage.Figure3isa\close-up"ofthesingleprocessingstageknownas\ASTdelivery."

A.5 Integrating Tests into the M odel

Armedwithamuch-improvedmodelofthedomain,webuiltasemanticnet-workcapturingasmuchofthismodelasbearsdirectlyontheabilitytoselectandapplythetestsandtestgroups.IntheVAX/VMSsystem,ourvalidationmodelreectsbothdecompositionsalongdatastructuresandalongprocess-ingstages.Thesemanticnetworkhas15nodes,foreachtestgroup,77nodesthatrepresentknowledgeabouttheprocessingstagesofdevicedrivers,and22nodesthatrepresentknowledgeaboutdatastructures.

18ACREATINGTHEVMSVALIDATIONMODEL

LEGEND:

data structuretest test groups IRPUCB PCB valid PCB?

processing devicedriver

postprocessing

AST Delivery preprocessing

startingaddress

startingaddress valid IRP? startingaddress valid UCB?

Figure2:DeviceDriverValidationModel:AbstractView

A.5IntegratingTestsintotheModel19

AST Delivery

Break Internal PID

retrieve process PCB

null process PCB?

PCB$PID PCB Vector

first ACB ACB$ASTQBL JIB

PCB$ASTQBL PCB$JIB JIB$BYTCNT Check ACB

type

get ASTQBL Check JIB Valid Return byte count Get JIB

address

JSB to AST routine REMQ AST

Enqueue ACB in AST queue

Table

Check

address JIB? byte count

Figure3:DeviceDriverValidationModel:ASTDelivery