CBM Constraints

The CBM data acquisition system differs from those of previous high-energy physics ex-periments as it is planned as a free-streaming, data-push architecture rather than a system that is based on a hierarchical set of trigger decisions. This induces some constraints on the CBM system architecture.

CBMNet

One of the most challenging tasks with the concept of a free-streaming detector read-out is the common representation of time in all detector components. Unlike in triggered sys-tems, there is no global decision to start the read-out of detector data. Every component decides locally when to send out data. To later enable a correlation of data from different components, all data needs to be tagged with an unambiguous time stamp. Since two clocks derived from two different sources never run at the exact same frequency, they drift apart over time. Therefore, either all the clocks that are used in all time critical com-ponents are derived from the same clock source, or the clock drift of all clock sources is known very precisely. In laboratory setups with only few components it is feasible to

1LHCb: Large Hadron Collider beauty experiment

2PANDA: Proton Antiproton Annihilations at Darmstadt

record and log the drift of all clocks involved. However, this approach does not scale to a big experiment like CBM with some hundreds or even thousands of independently operating entities. Therefore, distribution of a common clock and the synchronization of the front-end electronics is required.

CBMNet is a connection protocol designed at theComputer Architecture Groupin Mann-heim/Germany which provides the features required by CBM [Lem12]. It can be used to

• distribute the clock,

• synchronize the time with special messages that travel through the network with deterministic timing, so called deterministic latency messages (DLMs),

• transport control messages on a reliable channel,

• and transport data messages on a fast channel.

A project with comparable objectives is the GigaBit Transceiver (GBT) project devel-oped at CERN [MMK07]. However, the present work is based on CBMNet.

Clock Distribution Clock distribution is one of the key features of CBMNet. CBMNet is designed to use the recovered clock from one dedicated optical receiver as system clock and also to use this recovered clock as reference clock for other optical connections, The clock can thereby be further distributed to the next components in the read-out chain.

However, to be able to do so, the jitter of the recovered clock needs to be below 40 ps RMS [Lem12, page 68]. Since each step that tempers with the clock also adds jitter to the clock signal [Xil08, page 74], clock distribution over several hierarchy levels requires a dedicated jitter cleaning device. This is the reason why such a the jitter cleaner being present on theSysCore Board Version 3(see section D.2.3).

Finally, with such a clock distribution, a clock from a single time master can be dis-tributed down the tree of the read-out chain and all components will be clocked syn-chronously without clock drift.

Time Synchronization Another key feature of CBMNet is the system-wide time syn-chronization with so called “Deterministic Latency Message” (DLM). DLMs are 4 bit wide messages which are transported through the CBMNet cores and over the optical fiber with a deterministic delay. The delay might be different from connection to connection because of different cable length, but it will not vary for two DLMs over the same connec-tion. Since DLMs work in both directions, the delay can be measured easily via loopback.

Once the delays of all connections are known, DLMs can be used to start the time stamp counters in all components coherently. A dedicated DLM for synchronization can be sent periodically with a defined interval to allow for components to check if they are out of sync.

Time Encoding

The self-triggered and free-streaming approach of the CBM experiment requires a un-ambiguous time encoding for the time span of a whole run. A run typically lasts from several minutes to hours, or even days. Time precision, on the other hand, needs to be in the order of nanoseconds, for CBM-ToF even in the order of tens of picoseconds. A lot of bits are required to encode time with the required precision while still covering the full run time. For example, already 48bitsare required to cover a 3 hours run at a precision of 50ps.

To transport all those bits for every hit in every component would result in an un-necessary high utilization of bandwidth. “Unun-necessary”, because the more significant bits of different hits do not change very frequently and are therefore in most cases just a repetition of the information of the previous hit.

Time Hit with full time information Hit with full time information Hit with full time information Hit with full time information Hit with full time information Hit with full time information Hit with full time information Hit with full time information Hit with full time information Hit with full time information Hit with full time information Hit with full time information Hit with full time information Hit with full time information

(a) Every hit carries the full time stamp.

Time

Epoch Epoch

Hit Hit Hit Hit Hit Hit Hit Hit Hit Hit Hit Hit Hit Hit

(b) Upper bits of the time stamp counter are communicated via Epoch markers, hits messages only carry the lower bits.

Figure 3.1.: Both diagrams show the same hit sequence. Less bandwidth is required when Epoch markers are used as for each hit message the number of bits can be signifi-cantly reduced.

The repetitive sending of the more significant bits is avoided in CBM with the introduc-tion of Epoch markers in the data stream. The bits of the time stamp counter are subdi-vided into two groups, the “higher bits” vector and the “lower bits” vector. The “higher

bits” only change when the “lower bits” counter overflows. Every time the “higher bits”

change, an Epoch marker is inserted in the data stream carrying the updated value of the higher bits. Hit data is only tagged with the lower bits of the time stamp counter. Figure 3.1 illustrates this principle.

However, this concept also entails some challenges. The insertion and handling of Epoch markers has to be implemented very accurately. It would result in a huge time difference for a hit if it is sent out before or after an Epoch marker. In addition, merging of multiple data streams becomes very complicated, especially if the preceding data path contains buffers. In that case, hit data from different data streams is not correlated in time and might belong to different epochs. Furthermore, a lost Epoch marker would corrupt a lot of data.

At the time of writing, the implementations used for the CBM read-out chain are based on Epoch markers. In case of the GET4, a hit message comprises 18bittime information, and an Epoch Message, which is inserted in the data stream every 26.2144µs, comprises 32bit.