The trigger-less TBit/s readout for the Mu3e
experiment
Dirk Wiedner
On behalf of the Mu3e collaboration
24 Sep 2013 1 Dirk Wiedner TWEPP2013
The Mu3e Signal
9/20/2012
Dirk Wiedner, Mu3e collaboration 2
• μ→eee rare in SM
• Enhanced in:
o Super-symmetry
o Grand unified models o Left-right symmetric
models
o Extended Higgs sector o Large extra dimensions
Rare decay (BR<10-12, SINDRUM)
• For BR O(10-16)
>1016 muon decays
High decay rates
O(10
9muon/s)
The Mu3e Experiment
9/20/2012
Dirk Wiedner, Mu3e collaboration 3
• Target double hollow cone
• Silicon pixel tracker
• Scintillating fiber tracker
• Recurl station
• Tile hodoscope
• Muon beam O(109/s)
• Helium atmosphere
• 1 T B-field
The Mu3e Experiment
9/20/2012
Dirk Wiedner, Mu3e collaboration 4
• Target double hollow cone
• Silicon pixel tracker
• Scintillating fiber tracker
• Recurl station
• Tile hodoscope
• Muon beam O(109/s)
• Helium atmosphere
• 1 T B-field
The Mu3e Experiment
9/20/2012
Dirk Wiedner, Mu3e collaboration 5
• Target double hollow cone
• Silicon pixel tracker
• Scintillating fiber tracker
• Recurl station
• Tile hodoscope
• Muon beam O(109/s)
• Helium atmosphere
• 1 T B-field
The Mu3e Experiment
9/20/2012
Dirk Wiedner, Mu3e collaboration 6
• Target double hollow cone
• Silicon pixel tracker
• Scintillating fiber tracker
• Recurl station
• Tile hodoscope
• Muon beam O(109/s)
• Helium atmosphere
• 1 T B-field
The Mu3e Experiment
9/20/2012
Dirk Wiedner, Mu3e collaboration 7
• Target double hollow cone
• Silicon pixel tracker
• Scintillating fiber tracker
• Recurl station
• Tile hodoscope
• Muon beam O(109/s)
• Helium atmosphere
• 1 T B-field
Readout Requirements
24 Sep 2013
Dirk Wiedner TWEPP2013 8
• 2.5 GHz muon decays
• 50 ns readout frames (pixel)
• O(5000) pixel chips
• O(7000) scintillating fibers
• O(7000) timing tiles
• Online filtering
Timing Detectors
24 Sep 2013
Dirk Wiedner TWEPP2013 9
• Scintillating fiber hodoscope
• Timing tiles
• On detector zero- suppression
• Poster Session:
o STiC - A Mixed Mode Silicon- Photomultiplier Readout ASIC for Time-of-Flight Applications (Tobias Harion)
O(7000) fibers O(7000) tiles
Silicon Pixel Detector
24 Sep 2013
Dirk Wiedner TWEPP2013 10
• Inner double layer
• Outer double layer
• Re-curl layers
o Both sides (x2)
• Sensor size
o 1x2 cm2 inner layers
o 2x2 cm2 outer layers 180 inner sensors 4680 outer sensors
HV-MAPS
2/5/2013
Dirk Wiedner, Mu3e collaboration 11
• High Voltage Monolithic Active Pixel Sensors
• HV-CMOS technology
• Reversely biased ~60V
o Charge collection via drift
Fast O(100 ns)
o Thinning to < 50 μm possible
by Ivan Peric
I. Peric, A novel monolithic pixelated particle detector implemented in high- voltage CMOS technology
Nucl.Instrum.Meth., 2007, A582, 876
HV-MAPS
2/5/2013
Dirk Wiedner, Mu3e collaboration 12
• High Voltage Monolithic Active Pixel Sensors
• HV-CMOS technology
• Reversely biased ~60V
o Charge collection via drift
Fast O(100 ns)
o Thinning to < 50 μm possible
by Ivan Peric
I. Peric, A novel monolithic pixelated particle detector implemented in high- voltage CMOS technology
Nucl.Instrum.Meth., 2007, A582, 876
HV-MAPS
2/5/2013
Dirk Wiedner, Mu3e collaboration 13
• High Voltage Monolithic Active Pixel Sensors
• HV-CMOS technology
• Reversely biased ~60V
o Charge collection via drift
Fast O(100 ns)
o Thinning to < 50 μm possible
• Integrated readout electronics
o Zero suppression
o 800Mbit/s serial LVDS outputs
by Ivan Peric
I. Peric, A novel monolithic pixelated particle detector implemented in high- voltage CMOS technology
Nucl.Instrum.Meth., 2007, A582, 876
HV-MAPS
2/5/2013
Dirk Wiedner, Mu3e collaboration 14
• High Voltage Monolithic Active Pixel Sensors
• HV-CMOS technology
• Reversely biased ~60V
o Charge collection via drift
Fast O(100 ns)
o Thinning to < 50 μm possible
• Integrated readout electronics
o Zero suppression
o 800Mbit/s serial LVDS outputs
by Ivan Peric
I. Peric, A novel monolithic pixelated particle detector implemented in high- voltage CMOS technology
Nucl.Instrum.Meth., 2007, A582, 876
Pixel Readout Scheme
24 Sep 2013
Dirk Wiedner TWEPP2013 15
Pixel Readout Scheme
24 Sep 2013
Dirk Wiedner TWEPP2013 16
• Pixel logic
o Pixel address (8 bit) o Frame number (4 bit) o 50 ns frames
• Column logic
o Pixel data
o Column address o Coarse time
• Frame logic
o Super Frame
o Contains 16 x 50 ns readout frames
o + Sensor header
• Readout buffer
• Serializer and fast link(s)
Pixel address
Pixel Logic
Column Logic
Frame logic Readout buffer
Serializer Fine
time
Coarse time
Column address
Pixel Readout Scheme
24 Sep 2013
Dirk Wiedner TWEPP2013 17
• Pixel logic
o Pixel address (8 bit) o Frame number (4 bit) o 50 ns frames
• Column logic
o Pixel data
o Column address o Coarse time
• Frame logic
o Super Frame
o Contains 16 x 50 ns readout frames
o + Sensor header
• Readout buffer
• Serializer and fast link(s)
Pixel address
Pixel Logic
Column Logic
Frame logic Readout buffer
Serializer
8 bit
Fine time
4 bit
12 bit Coarse
time
Column address
Pixel Readout Scheme
24 Sep 2013
Dirk Wiedner TWEPP2013 18
• Pixel logic
o Pixel address (8 bit) o Frame number (4 bit) o 50 ns frames
• Column logic
o Pixel data
o Column address o Coarse time
• Frame logic
o Super Frame
o Contains 16 x 50 ns readout frames
o + Sensor header
• Readout buffer
• Serializer and fast link(s)
Pixel address
Pixel Logic
Column Logic
Frame logic Readout buffer
Serializer
8 bit
Fine time
4 bit
12 bit Coarse
time
Column address
12 bit 8 bit
32 bit
Pixel Readout Scheme
24 Sep 2013
Dirk Wiedner TWEPP2013 19
• Pixel logic
o Pixel address (8 bit) o Frame number (4 bit) o 50 ns frames
• Column logic
o Pixel data
o Column address o Coarse time
• Frame logic
o Contains 16 x 50 ns readout frames
o + Sensor header
Super Frame
• Readout buffer
• Serializer and fast link(s)
Pixel address
Pixel Logic
Column Logic
Frame logic Readout buffer
Serializer
8 bit
Fine time
4 bit
12 bit Coarse
time
Column address
32 bit
4 x serial @ 800 Mb/s
12 bit 8 bit
Data Link Scheme
From detector slices to time slices
24 Sep 2013
Dirk Wiedner TWEPP2013 20
Link Overview
24 Sep 2013
Dirk Wiedner TWEPP2013 21
• Front end links
o Pixel sensor to on-detector FPGA
• 400 – 800 Mbit/s
• LVDS
o Timing detector readout
• Optical links from detector
o Front end FPGAs
o … to readout boards o 5 Gbit/s
• Optical links in counting room
o Off-detector read out boards o …to PC Farm
Link Overview
24 Sep 2013
Dirk Wiedner TWEPP2013 22
• Front end links
o Pixel sensor to on-detector FPGA
• 400 – 800 Mbit/s
• LVDS
o Timing detector readout
• Optical links from detector
o Front end FPGAs
o … to readout boards o 5 Gbit/s
• Optical links in counting room
o Off-detector read out boards o …to PC Farm
Pixel Sensor
Silicon FPGAs
x86 Readout
board x12
PC x48
Link Overview
24 Sep 2013
Dirk Wiedner TWEPP2013 23
• Front end links
o Pixel sensor to on-detector FPGA
• 400 – 800 Mbit/s
• LVDS
o Timing detector readout
• Optical links from detector
o Front end FPGAs
o … to readout boards o 5 Gbit/s
• Optical links in counting room
o Off-detector read out boards o …to PC Farm
Pixel
SensorPixel Fiber Tile SensorPixel Fiber Tile
SensorPixel Fiber Tile
Sensor Fiber Tile
Silicon FPGAs
x86
Fiber FPGAs
x48
Tile FPGAs
x48 Readout
board x16
Readout board
x8
Readout board
x8
x6336 x7000 x7000
PC x48 O(8Tbit/s)
Tile
Link Overview
24 Sep 2013
Dirk Wiedner TWEPP2013 24
• Front end links
o Pixel sensor to on-detector FPGA
• 400 – 800 Mbit/s
• LVDS
o Timing detector readout
• Optical links from detector
o Front end FPGAs
o … to readout boards o 5 Gbit/s
• Optical links in counting room
o Off-detector read out boards o …to PC Farm
Pixel
SensorPixel Fiber
SensorPixel Fiber Tile SensorPixel Fiber Tile
Sensor Fiber Tile
Silicon FPGAs
x86
Fiber FPGAs
x48
Tile FPGAs
x48 Readout
board x16
Readout board
x8
Readout board
x8 PC
x48 x376
Link Overview
24 Sep 2013
Dirk Wiedner TWEPP2013 25
• Front end links
o Pixel sensor to on-detector FPGA
• 400 – 800 Mbit/s
• LVDS
o Timing detector readout
• Optical links from detector
o Front end FPGAs
o … to readout boards o 5 Gbit/s
• Optical links in counting room
o Off-detector read out boards o …to PC Farm
Pixel
SensorPixel Fiber Tile SensorPixel Fiber Tile
SensorPixel Fiber Tile Sensor Fiber Tile
Silicon FPGAs
x86
Fiber FPGAs
x48
Tile FPGAs
x48
Readout board
x16
Readout board
x8
Readout board
x8 PC
x48
x376 x192 x192
O(4Tbit/s)
Link Overview
24 Sep 2013
Dirk Wiedner TWEPP2013 26
• Front end links
o Pixel sensor to on-detector FPGA
• 400 – 800 Mbit/s
• LVDS
o Timing detector readout
• Optical links from detector
o Front end FPGAs
o … to readout boards o 5 Gbit/s
• Optical links in counting room
o Off-detector read out boards o …to PC Farm
Pixel
SensorPixel Fiber Tile SensorPixel Fiber Tile
SensorPixel Fiber Tile Sensor Fiber Tile
Silicon FPGAs
x86
Fiber FPGAs
x48
Tile FPGAs
x48 Readout
board x16
Readout board
x8
Readout board
x8
PC x192 x48
x96 x96
O(4Tbit/s)
Front End FPGAs
24 Sep 2013
Dirk Wiedner TWEPP2013 27
• FPGAs on detector
o 86 (+96) pieces
• Receive sensor data
o 108 LVDS inputs
• 5 Gbit/s outputs
o 8 optical links
o … to counting house
• Switching data between readout boards farms A-D
Front end FPGA 800 Mbit/s
LVDS in x 108
5 Gbit/s optical
Readout board
A
Pixel Sensor
Readout board
B
Readout board
C
Readout board
D
Front End FPGAs
24 Sep 2013
Dirk Wiedner TWEPP2013 28
• FPGAs on detector
o 86 (+96) pieces
• Receive sensor data
o 108 LVDS inputs
• 5 Gbit/s outputs
o 8 optical links
o … to counting house
• Switching data between readout boards farms A-D
Front end FPGA 800 Mbit/s
LVDS in x 108
5 Gbit/s optical
Readout board
A
Pixel Sensor
Readout board
B
Readout board
C
Readout board
D
Front End FPGAs
24 Sep 2013
Dirk Wiedner TWEPP2013 29
• FPGAs on detector
o 86 (+96) pieces
• Receive sensor data
o 108 LVDS inputs
• 5 Gbit/s outputs
o 8 optical links
o … to counting house
• Switching data between readout boards farms A-D
Front end FPGA 800 Mbit/s
LVDS in x 108
5 Gbit/s optical
Readout board
A
Pixel Sensor
Readout board
B
Readout board
C
Readout board
D
Front End FPGAs
24 Sep 2013
Dirk Wiedner TWEPP2013 30
• FPGAs on detector
o 86 (+96) pieces
• Receive sensor data
o 108 LVDS inputs
• 5 Gbit/s outputs
o 8 optical links
o … to counting house
• Switching data between readout boards farms A-D
Front end FPGA 800 Mbit/s
LVDS in x 108
5 Gbit/s optical
Readout board
A
Pixel Sensor
Readout board
B
Readout board
C
Readout board
D
Front end FPGA
Readout Board
24 Sep 2013
Dirk Wiedner TWEPP2013 31
• FPGA readout boards
o 4 per sub-detector
• 5 Gbit/s optical inputs
o 16-28 inputs
• 10 Gbit/s optical output
o 12 outputs to PCs
• Switching network
o A-D sub-farms
o One output per PC
Readout board 5 Gbit/s
Optical x28
PC 10 Gbit/s
Optical
PC
Sub-farm A Front
end FPGA
Front end FPGA
Front end FPGA
x12 PC
Readout Board
24 Sep 2013
Dirk Wiedner TWEPP2013 32
• FPGA readout boards
o 4 per sub-detector
• 5 Gbit/s optical inputs
o 16-28 inputs
• 10 Gbit/s optical output
o 12 outputs to PCs
• Switching network
o A-D sub-farms
o One output per PC
Front end FPGA
Readout board 5 Gbit/s
Optical x28
PC 10 Gbit/s
Optical
PC Front
end FPGA
Front end FPGA
Front end FPGA
PC Sub-farm A
x12
GPU-PC
24 Sep 2013
Dirk Wiedner TWEPP2013 33
• PC with GPU
• 10 Gbit/s Fiber input
o 8 inputs from sub- detectors
• Data filtering
o Timing Filter on FPGA o Track filter on GPU
o Data to tape < 100 MB/s
FPGA PCIe board
GPU computer
Optical mezzanine connectors
GPU-PC
24 Sep 2013
Dirk Wiedner TWEPP2013 34
• PC with GPU
• 10 Gbit/s Fiber input
o 8 inputs from sub- detectors
• Data filtering
o Timing Filter on FPGA o Track filter on GPU
o Data to tape < 100 MB/s
GPU computer
Readout
board Readout
board
GPU-PC
24 Sep 2013
Dirk Wiedner TWEPP2013 35
• PC with GPU
• 10 Gbit/s Fiber input
o 8 inputs from sub- detectors
• Data filtering
o Timing Filter on FPGA o Track filter on GPU
o Data to tape < 100 MB/s
PC 10 Gbit/s
Optical x8
PC 10 Gbit/s
Optical x8 A
Readout board
B
x8 Readout
board Readout
board Readout
board
Timing Filter
24 Sep 2013
Dirk Wiedner TWEPP2013 36
• Entire event on PCIe FPGA
• Tile and Fiber data
o Easy to match
o Look for three tracks
• Reject data without three hits
o … inside time interval
1
3
2
Timing Filter
24 Sep 2013
Dirk Wiedner TWEPP2013 37
• Entire event on PCIe FPGA
• Tile and Fiber data
o Easy to match
o Look for three tracks
• Reject data without three hits
o … inside time interval
1
3
2
Vertex Filter
24 Sep 2013
Dirk Wiedner TWEPP2013 38
• Entire event on GPU
• Large target
o Large spread of muons o Easy vertex separation
• Reject data without three tracks
o … inside area interval on target
1
3
2
Vertex Filter
24 Sep 2013
Dirk Wiedner TWEPP2013 39
• Entire event on GPU
• Large target
o Large spread of muons o Easy vertex separation
• Reject data without three tracks
o … inside area interval on target
1
3
2
Summary
• Mu3e has 280M pixels @ >109 muons/s
• >1 Tbit/s data
• 0-suppressed serial data from active pixel sensors
• Switched optical network
• GPU filter farm with optical inputs
24 Sep 2013
Dirk Wiedner TWEPP2013 40
+ +
Backup Slides
24 Sep 2013
Dirk Wiedner TWEPP2013 41
Physics Motivation
9/20/2012
Dirk Wiedner, Mu3e collaboration 42
Standard model:
• No lepton flavor violation Lepton flavor violation?
Physics Motivation
9/20/2012
Dirk Wiedner, Mu3e collaboration 43
Standard model:
• No lepton flavor violation, but:
o Neutrino mixing
o Branching ratio <10-50 →unobservable
Lepton flavor violation: μ+→e+e-e+
The Mu3e Signal
9/20/2012
Dirk Wiedner, Mu3e collaboration 44
• μ→eee rare in SM
• Enhanced in:
o Super-symmetry
o Grand unified models o Left-right symmetric
models
o Extended Higgs sector o Large extra dimensions
The Mu3e Background
9/20/2012
Dirk Wiedner, Mu3e collaboration 45
• Combinatorial background
o μ+→e+νν & μ+→e+νν & e+e- o many possible combinations
Good time and
Good vertex resolution required
The Mu3e Background
9/20/2012
Dirk Wiedner, Mu3e collaboration 46
• μ+→e+e-e+νν
o Missing energy (ν)
Good momentum resolution
(R. M. Djilkibaev, R. V. Konoplich, Phys.Rev. D79 (2009) 073004)
Pixel Sensor Links
24 Sep 2013
Dirk Wiedner TWEPP2013 47
• Vertex Sensor chips
o 180 chips o 4 LVDS links
o 800 Mbit/s per link
• Central Silicon Tracker
o 936 chips o 2 LVDS links o 800 Mbit/s
• Recurl stations
o 3744 chips o 1 LVDS link o 400 Mbit/s
Pixel Sensor Links
24 Sep 2013
Dirk Wiedner TWEPP2013 48
• Vertex Sensor chips
o 180 chips o 4 LVDS links
o 800 Mbit/s per link
• Central Silicon Tracker
o 936 chips o 2 LVDS links o 800 Mbit/s
• Recurl stations
o 3744 chips o 1 LVDS link o 400 Mbit/s
Pixel Sensor
800 Mbits/s
Front end FPGA
Front End FPGAs
24 Sep 2013
Dirk Wiedner TWEPP2013 49
• FPGAs on detector
o 86 (+96) pieces
• Receive sensor data
o 108 LVDS inputs
• 5 Gbit/s outputs
o 8 optical links
o … to counting house
• Switching between readout boards A-D
Optical transceiver FE board
Average Occupancies
24 Sep 2013
Dirk Wiedner TWEPP2013 50
• All numbers per frame of 50 ns
• Vertex detector
o 2 hits per sensor
• Central silicon tracker
o 0.6 hits per sensor
• Recurl stations
o 0.13 hit per sensor
• Fiber hodoscope
o 0.16 hits per fiber
• Timing tiles
o 0.09 hits per tile
Tile occupancies
Maximum Occupancies
24 Sep 2013
Dirk Wiedner TWEPP2013 51
• All numbers per frame of 50 ns
• Vertex detector
o 5 hits per sensor
• Central silicon tracker
o 2 hits per sensor
• Recurl stations
o 1 hit per sensor
• Fiber hodoscope
o 0.24 hits per fiber
• Timing tiles
o 0.14 hits per tile
φ
z Vertex occupancies
