In support of developing com mon metric.� across the in ternetworking router i ndustry, t he Internet Engineering Tas k Force ([ ETF) has set up a Benchmarking Methodology Working G roup, which has devc;Ioped definitions for router pert<>r
mance.' Three key metrics defined by t h is group provide the background for our d iscussion of Digital's rou ter software design.
78
• 'fhroughput-the maxi n1um (for�varding) rate at which none of the offered frames ( packets) are dropped by the device (i.e., packets per second)
• Frame loss rate- the percent of frames (packets) that should have been forwarded by the network device (router) while u nder a constan t load but which were not forwarded due to l ack of resources (i.e., percent packets Jost)
• Latency-for store-and-forward devices (i .e . , routers), the t i me in terval beginning when the last bit of the i n rut frame reaches the input port and ending when the first bit of the output frame is seen on the output port (i .e . , u n its of t ime)
In the design of Digi tal's router software and sys
tems, a balance has been targeted with maximizing the packet throughput fo rward i ng rates wh ile m i n i m izing the packet latency. Som e vendors m is
takenly compare loss-free throughput rates with forwarding rates that have high loss rates. Such comparisons must be studied carefu l l y, because they do not compare rou te perfo r m a nce measures of equal i m pact to the total net work. To reiterate, the throughput forward i ng rate occu rs only at the po int when the frame loss rate is zero percent.
Digita l 's routers target throughput designs which, as much as possible, run at ·'wire speed" with zero frame loss rates. Regard less of the throughput value qu oted , router comparison shou ld reference com
mon packet loss rates because network appl ica
t ions need to retransmit any packets that are lost by the rou ters.
In general, the throughput, loss -free forwarding rate is the opti m u m value for d iscussions of router forwarding perfo rmance. The other critical value is the stabi l ity of the router u nder heavy overload.
A " receive l ivelock " con d ition occurs when the offered load, i . e . , input packets received for su bse
quent forward i ng by a given router, reaches the point where the de l ivered throughput, i . e . , packets actual l y fo rwarded, decreases to zero H.� Real-time systems, such as routers, have the potential to l ive
lock u nder traffic loads above their throughput peaks. However, it is extremely important that ro uti ng i m plementations avoid such responses to post-throughput saturation. In the case of Digital 's routers, in all archi tectures and products, the routers do not l ivelock but rem a i n stable eve n when the appl ied input load to a router exceeds tbe peak throughput fo rward i ng packet rate . This key
Viii 5 No. I Wintl!r 1993 Dip, ita/ Tee/mica/ journal
performance measure of router devices remains an underlying design characteristic of a l l D igital DECNIS and DEC WAt'\Jrouter network devices.
Packet Throughput/Forwarding Rate Digital's routing platforms offer a range of through
put measures. For each platform, the throughput is the most often quoted value used to characterize the router's aggregate capabilities. In the case of the DECNIS 600, an aggregate throughput of 80,000 packets per second is offered Hl In s maller routers, the WAN line interface rates (i.e . , 64 kb/s and T l ) optimized the designs of its multiprotocol routers to al low data forwarding to occur as fast as possible.
On the DEC WANrouter products, we han d le all the forwarding on a central CPU with little hardware assistance. In the DECNIS products, forwarding and filtering operations are hand led by li necards. A hardware assist for the performance-critical forwarding function 's address looku p is used on DECNIS routers in support of requirements for very h igh-speed packet switching.·1 On each l inecard , a strea mlined software kernel has been developed along with a l l the required software. The li necard software kernel and modu les were carefu l l y con
structed to have the minimum nu mber of instruc
t ions and the lowest nu mber of execution cycles necessary to perform the h igh-speed forwarding and filtering operations. On the DECNIS MPC, the software kerne.l is also fu l l y capable of the routing forward i ng operations. However, this kernel is mainly requ ired to provide the software processing for the remaining non-performance-intensive oper
ations of the router's software (i.e., the processing of updates to the rou ter topology database and the network management commands/received pack
ets). This partitioning of processing of received packets in the DECNIS rou ter system permits such routers, and the networks that they comprise, to remain h ighly stable when traffic overloads occur.
For the DEC WAN router software, the forwarding operation has no hardware assist. Software lookup assist algorithms have been researched and imp.le
mented to help meet the performance-intensive require ment. As in the m icrocoded DECNIS l inecard
Digital Technical journal Vol. 5 No. I Winter 1993
Digital's Multiprotocol Routing Software Design
adapter software, the software is h ighly tuned for performance. To mi nimize the additional mainte
nance overhead associated with h ighly tuned soft
ware, the amount of such code is kept to a
minimum. The DEC WAN router software design is an example of how Digital ca refu lly balanced prod
uct performance requirements and product devel
opment and maintenance costs to meet the requ ired price/performance goals for its rou ter product fam ily.
Packet Latency (Transit Delay)
The next most frequently specified performance requirement is packet latency or packet transit delay For bridge/router devices, this meas urement clearly depends on software and hardware timings.
However, the definition of l atency utilized con·e
sponds d irectly to the , '1osen system's design . The previously quoted IETF definition for store
and-forward dev ices can be further refined to accom modate d iffering device designs. The I ETF working group clarifies the d ifference between a
"store-and-forward device" and a " bit-forward ing device" internal design model for a router. The latter design model is often referred to as a "cut
through" design and requires a d ifferent definition than previously listed for store-and-forward devices. The definition of latency used for this w hether or not processing starts prior to the packet being completely received. However, another key point is whether or not the packet received can be sent out for transmission prior to complete recep
tion. When reception, forward i ng, and transm is
sion can occur i n para l lel, the design is referred to as cut-through. For D igital's router designs, the DECNIS does process reception and forwarding in parallel prior to a packet being completely received . However, t he DECNIS does not start trans
mission until a packet is completely received. Thus, the DECNlS latency model uses the original store
and-forward definition of the IETF.
In the case of the DEC WANrou ter software, the model and definition used is agai n store-and-for
ward. The factors that control the packet latency i n the DEC WAN router design are as fol lows:
79
DECnet Open Networking
l . Receiving the packet. The packet must be com
pletely received.
2. Performing the forward i ng operation. This fac
tor incl udes packet ver ification, analyzing the packet, performing any required address lookup, performing any required packet mod ifi
cations, and queuing the packet for transmission o n the dest ination interface .
3. Congestion queuing. If the destination interface is not i d le, the packet will have to be queued before transmission. Some transit delay measu re
ments use only uncongested media in terfaces connected to the router. However, l a tency mea
surements must be made to measure the poten
tial latency delays due to congestion at the router output interface. The packet latency due to queue occupation delays is also incl uded here.
Congestion avoidance algorithms have been i mplemented to minimize this congestion delay.
4. Transmitting the p acket. This factor is usua l ly dominated by the t ime taken to clock the bits of the packet out of the interface but also includes media access times, i .e., delays due to another node already using a common connection.
We now examine how the DEC WANrouter and DECNIS routers separately minimize the transit de lay.
The DEC WAN rou ters minim ize the packet recep
t ion and transmission portions by al lowing hard
ware to perform these functions using direct memory access (DMA). Because these systems have only a single processor, the forward ing delay is m in
i mized by the same fast-path optimizations used to i mprove the forwarding rate.
On the other hand , the optimizations for the DECNIS routers are sl ightly different for the various l i necards. The DEC WANco ntroller 622 card has no DMA, and the l i necard on-board processor is i nvolved in receivi ng each byte of the packet. We parse the header as soon as the re is enough infor
mation to do so. For example, the data l i nk packet type field is decoded before the network address bytes have been received, and the network address lookup is initiated as soon as the add ress has been received (i.e . , before the data has been received).
The address lookup is then performed by the add ress recognition engine hardware without fur
ther i nvolvement from the software.
The DEC WANcontro l ler 618 card and the DEC LANcon troller 601 and 602 cards aJ I receive packets
80
one segment at a time. I n ternal ly, these cards use small fixed-size bu ffe rs that are l i nked together as necessary to store a whole packet. Aga in, they per
form the analysis and forwarding lookup as soon as the data is ava ilable (i.e., when the first segment is received).
Thus, for a large packet, the entire forward i ng decision will have been made before the last byte has been received. H owever, note that until the l ast byte has been received, it is not known whether the cyc lic red u nda ncy check (CRC) is correct or the packet has been corrupted . So the packet is not actually passed to the destination l inecard until that check has been completed . As discussed before, this design is still store-and-forward, rather than cut-through. The DECNIS design goals were easily met without using cut-through; however, Digital has used the cut -through design on a n umber of LAN host -based adapters.
\Vhen a packet is to be transmitted, certain changes must be made i n the data. For example, the IP and osr protocols require that time-to-I ive fields and, in some cases, other options be m oclifiecl.
Bridged packets may need address bits modified or conversion between Ethernet and I EEE 802 forms.
As with reception, a!J DEC WANcontrol lers perform these operations as the data is transmitted. Al l cards have hardware assistance for recalcul ating header checksums and CRCs.
These features are designed to reduce the for
ward ing delay as much as poss ible, so that the tran
sit delay is mainly control led by the time it takes to receive and send the packet. The type of archi tec
ture that best describes the DECNIS design is a data
flow, which blends trad itional store-and-forward designs with newer cut-through designs. This data
flow architecture processes packets in a d istrib
u ted manner (i . e . , l inecarcls process packets) without transmitting packets prior to complete reception val idation of these packets. This design l i mits the forwarding of packets that are found to be in error, whereas the similar fu l l cut-through design wou l d propagate invalid packets.
Interaction between Routing and Bridging
Designing a combi ned router and bridge product is com pi ica tecl by the rel ationship between the rou t
i ng and bridging fu nctions. 1 1 A received packet mu st be subjected to either the bridge forwarding or the rou ting forwarding process (or maybe both).
Vol. 5 No. I Winter 1993 Digital Tecb7lical jour1lal
Several designs are possible and are i l lustrated in Figure 5.
(a) Protocol sp lit. In this design, some protocols are bridged, e.g. , Local Area Tra nsport (LAT), and others are routed, e.g., TCP/I P. The bridging and routing functions are completely separate;
t hey merely share I ine interfaces. Every packet received is passed to either routing (if intended for a protocol that is being routed) or bridging.
(b) In tegrated with one interface. In this design, the routing fu nction is modeled as being layered on top of the bridging fu nction.
Theoretical ly, packets are subjected to the bridging process a nd then, if they are addressed to the rou ter, su bjected to the rout
ing process. In this form of the model, the router uses a single logical interface seemi ngly connected to a private LAN contained within the bridge/router.
(c) Integrated with m u l tiple interfaces. This design is similar to the integrated design with one interface, but the router uses a l l the avail
able interfaces and logica l ly connects to the same extended LAl\1 mul tiple times.
Each design model has advant ages and d isadvan
tages, and we conside red a l l three models for the design of the DECNIS routers. The protocol
spl itting model has the advantage of simpl icity. The major disadvantage is that any particular protocol must be either bridged or routed . The in tegrated models have the disadvantage of req uiri ng specific management to prevent a routed protocol from also being bridged . In most cases, a protocol is being routed specifical l y to avoid the problems associated with bridging. The model with one inter
face also has the disadvantage that the network ma nager may get confused at tempting to work out which interface is being used for routing. We chose the protocol-spl itt ing model because of its effec
tiveness and ease of use.