Bug detection was a key status indicator through
out the J\1VAX logical verification effort and thus helped to steer the team's work. Hugs were tracked carefu lly with an on-line system and ana lyzed each week to consider trends, successfu l and unsuccess
ful bug-findi ng techniques, and bug hot spots, which requi red additional attention. The bug detec
tion rate was fairly constant throughout the project at abou t 22 per month, with the excep tion of the last month in which the rate dropped to nearly zero. An analysis of the bug-detecting effectiveness of each testing tech nique shows that all test tech
niques were effective and seemed to complement each other. Table 1 shows the percentage of bugs detected by each technique. This table includes data on the ever-valuable, nonsim ulation verifica
tion tec hnique of simply reviewing, inspecting, and discussing the design and its many representations.
The decision to release the design for fabrication of first-pass chips was a consensus decision made by the verification, archi tecture, and design teams.
From a verification perspective, the design was ready for release when the bug detection rate remained at zero for several weeks and the majority of the planned tests had been i mplemented . The develop ment schedule, whereas additio nal delay in releasing the design wou ld.
Vol. 4 No. 3 Surnmer 1992 Digital Teclmical journal
Ta ble 1 Bug Detection Using Va rious Techn iques
Percent of
Techn ique Total Bugs Found
Focused tests 28
Directed, pseudorandom
exercisers 23
Review, i nspection,
observation, thought 20
Detection technique unknown 1 2
AXE 8
MAX 6
HCORE 2
Other
Results and Conclusions
Only 15 logical bugs were fo und i n the first- pass NVAX CPU chip design, a ll of which were either eas
i ly worked aroun d or did not impact normal system operation . The n an1re of the bugs fo u nd in the first
pass design ranged from stra igh tforward bugs that escaped detection tor clear-cut reasons to extremely complex bugs that required h o u rs or weeks of rig
ormJs prototype system test ing to u ncover. Some of the bugs escaped detection during s i m u l ated verifi
cation for cl assic reasons such as:
• Tes t i ng of the function was pe rformed just before release, in a hurried man ner.
• Sim u l ation performance proh i b ited r u n n i ng a cert a i n type of test case.
• A test was n o t run i n a certa i n mode due to the difficu l ty of running it i n all possible modes.
• It took an exerciser r u n n i ng on a simu l ator a long time to encou n ter the conditions that wou ld evoke the bug.
• A test was i nadvert e ntl y dropped from the set of exercisers that were run contin uously
Det a i l s abou t five of the more i nterest i ng hugs fou nd in the first -pass design fol low. Included is i n fo rmation about how the hug was detected, a hypothesis on why the bug e luded detection before first-pass c h ips were fabricated , and les
sons lea rned from the detection and e l i m i n ation of the bug.
Digital Tecbuica/ journal liJI. ·I .Vo ..i S111nml!r l')')l
Logical Veri(ica tiun of the NVAX CPU Ch ip /Jeslj{ll
1 . One simple bug was detected by r u n n i ng the HCORE test su ite o n the pro totYpe system with the floati ng-point unit ( F- box) ll isable d . This bug could have been fo u nd i n the same way through simu latio n , but the test suite was n o t run as a fin a l regression test w i th the F-box d isabled . I n ge nera l , focused tests l i ke HCORE were n o t run wit h varied chip/system configu rati ons. The ver
ification team concl uded that a l l foc used tests sho u ld be r u n with d ifferent chip/system con fig
u rat i o ns. At the m i n i m u m , a configuration that disables all possible fu nctions s hou ld be H::sted.
2. Another hug was discovered because the Cl' l l c h i p generated spurious writes to memory in the protot ype system. The exercisers prolx1bly did generate the conditions necessary to evoke thi s bug; however, the spurious wri tes wen t un no
ticed . It is extremely diffi cult to verify that a machine does everyt h i ng i t is supposed to do and noth ing more. Ad ditional assertion checkers or m o n i tors in t h e models might detect such bugs in the fu t ure.
3. A third bug was evoked when a prototype system exerciser executed a translation bu ffer inva l idate a l l ( TBIA) i nstruct i o n u nder certa i n con d i t i o ns. On a rea l system, the TBI A instruc
tion is used o n ly by the operating system. I n our verificat ion effort, the TIIIA instruction was l i ttle used by the exercisers that were simu lated.
Operations that are performed o n ly by the oper
ating system should not be underem phasized in exercisers.
4. One first -pass bug was related to the halt in ter
rupt, which i s used o n l y d u ring debugging oper
ations. The h a l t i nterrupt receivecl m i n i m a l testing a n d was not tested a t a l l i n a n y t y p e of exerciser. Discovering this bug was especia l l y ann oying beca use a similar bug h a d escaped detection by t he i nitial logical verification effort fo r a previous VAX i m rlementation. This turn of events reinforces the be.l ief th at there is va l u e i n reviewing t h e escaped b u g l ists from other proj
ects. Also, dur i ng the veri ficat i o n effort, there seemed to be a natural, but erro neous, tendency to u mlertest fu n ctions used i n frequent l y or not at a l l during normal system operation. Such fu nctions sometimes require extra ::tttent i o n , because they may b e qu ite complex a n d may have been given less carefu l thought d ur ing the design process.
NVAX-rnicroprocessor VAX Systems
5. A state bit that needed to be i ni t ia l ized on power
up was not. This problem was not iced during i n i t ia l ization simu lation but erroneously ratio
nalized as being acceptable. Design assu mptions and assertions about i n itial ization should be ver
ified through simu lation or other means.
Overa l l , the 1\f\/AX Cl'll chip logical verification effort was a su ccess. The pseudorandom testing strategy detected several complex and subtle logi
cal bugs that otherwise probably wou ld not have been detected by simu lation. The extensive simula
tion performed on the schematics-derive(! model of the chip prov ided a h igh degree of confidence in the design .
The goals of producing h igh ly functional first
pass chips and bug-free, second-pass chips were both met. Neither the bugs in first-pass chips nor their work-arounds impeded prototype system debugging in any sign i ficant way. and first-pass chips with work-a rounds were used in prepro
d uction, field-test systems. The verification team corrected the 15 first-pass design bugs fo r second
pass chips, which were shipped to customers i n revenue-producing systems.
Acknowledgments
The NVAX logical verificati o n effort was performed by a team of engineers from the SEG microproces
sor verificati o n group . Members of this tea m included Wa l ker Anderson, Rick Calcagn i , Sanjay Chopra, and Joh n St. Laurent. 1\f\/A.,.'( archi tects M i ke Uhler and Debra Bernstein provided extensive tech
n ical d irec t ion and assistance to the verification rea m . The SEG CAD group helped by i ts cont inual
46
development and support of too ls. 'l'he CHA GO simu lator wou l d not have been possible without the significant co ntribu ti o ns of Kev i n Lack l . Will Sherwood provided h igh-qua l itv, top- level gu id
ance th rough a l l phases of the project. 'fhe system development groups performed system - level simu
lations and rigorous p rototype test i ng. The VAX Arch itecture G ro u p A..'<:E/�LAX team, once again, prov ided a nd supported a n effect ive verification tool. Lastly, the success of the project and the final qual i ty of the NVAX chip l ogical design are as much a tribute to the work of the '\f\IAX arch i tecture a nd design teams as they are to the work of the verifica
tion team.
References
1 . G Uhler e t a l . , "The NVAX ami NVAX+ High
performance VAX Microprocesso rs," Digital Teclmical Jou rnal, vol . 4. no. :) (Sum mer 1 992, this issue): 11 - 23.
2. D. Donchin et a l . , ·'The N VAX Cl'l f Chip:
Design Cha l lenges, Methods. and C A D Too ls,"
Digital Tecl.mical Journal, vol . 4. n o. 3 (Summer 1992, this issue): 24- .37
3. R. Calcagni and W Sherwood , " VAX 6000 Model 400 CPU Chip Set Functional Design Ve rification," Digital Technical Journal, vol. 2. no. 2 (Spring 1990) 64 - 72.
4. D. Bhamlarkar, "Architectu re J\<lanagement for Ensuring Software Compatibil ity in the VAX Fam ily of Computers," 1/:'f:F Computer ( february 19H2): 87- 9 3
vbl. 4 No . . l St1111111er 1992 D igital Tecbuical jounwl
Lawrence Chisvin Gregg A. Bouchard Thomas M. Wenners