switch design, commercialized by Seven Solutions, is based on a MicroTCA MCH-type mod-ule [56]. This would allow a conform White Rabbit solution within the crate including all modules supporting Gigabit Ethernet via the backplane and compatible to the White Rabbit Core implementation. However, until now no MicroTCA based full MCH integration seems to exist.
However, there might be scenarios, where a MicroTCA.4 system should be integrated into a White Rabbit based synchronization system. This could be achieved by directly connecting the XFEL Timing Receiver module with a White Rabbit switch. In this case the European XFEL Timing Receiver would act as a White Rabbit node and implements the White Rabbit Core within the FPGA. The 125MHz reference frequency can either be generated directly with the FPGA or via the PLL within the clock buffer chip in the clock distribution on the board.
The timing module then distributes the time stamp information on the bus lines within the crate as described in chapter 11. The module would also allow to distribute the syntonized 125MHz clock within the crate via the MCH on TCLKA or TCLKB. However, the firmware for that interfacing functionality has not yet been implemented.
12.4 Bunch clock distribution systems
If a MicroTCA.4 system with the European XFEL timing system receiver should be integrated into an existing bunch clock distribution system at other facilities, this can be performed in two different ways: (1) the reference RF clock of the facility can be connected to the RF reference input of the module. Internal clock divider and delay components of the dedicated chips as well as FPGA functions can be used to divide and generate clocks and triggers synchronously to the bunches in the machine. The optional orbit clock or trigger input (connected via LVDS converter to one of the inputs of the lowest RJ45 connector) allow synchronization of the clock dividers and therefore an easier calibration. All signals will then be provided to the other modules within the crate. The alternative (2) assumes, that the bunch clock will be directly provided to the LVDS converter, which is connected to the lowest RJ45 input, or to the SMA clock input of the timing board. Also this clock, together with signals generated within the FPGA, can then be distributed within the crate.
Besides the described signals, the timing board can generate time stamps based on the CPU system time and distribute them within the crate.
Chapter 13
Conclusion
In summary, based on clock distribution, synchronization and sequencing requirements for the XFEL and FLASH a technical solution called XFEL Timing System had been developed. In this process a conceptual design has been worked out, critical components identified, candidates selected and their performance evaluated. The development of an evaluation board allowed a detailed evaluation and analysis of the most important conceptual aspects of the XFEL Tim-ing System design. Before a definition of the complete design of the first generation timTim-ing module was possible, an intensive investigation, discussion and standardization process of the destination hardware platform MicroTCA was required in order to provide the necessary distri-bution channels for accurate and low-noise timing related signals within the MicroTCA crates.
In this process, the new MicroTCA.4 standard (optimized for physics applications) has been defined within the PICMG consortium a group of industrial companies and research laborato-ries. Based on that standard and the experience from previous evaluations, the first generation timing module has been designed and produced. As important as the actual hardware was the design and implementation of the required protocols, firmware and software. Intensive tests and measurements showed, that the module is working properly and first production units had been used as stand-alone timing modules in different places since then. However, it also revealed elements to be improved and fields to be optimized in terms of function, performance and cost efficiency. The result was the second generation timing module and first prototypes became available beginning in 2013. The new design follows a concept of modularity, simpli-fication and optimization of re-programmable logic elements to the actual demands identified during the firmware developments of the first generation module. Measurements and tests confirmed a successful implementation and the first series production had been received to be installed in the extension of the FLASH project within 2013/2014 and to be used for European XFEL developments. Besides further firmware, software and add-on hardware developments the final step in the process is the on-going industrialization process of the XFEL Timing Sys-tem hardware to provide a commercially available product for easier and possibly even more cost efficient sourcing for the European XFEL, FLASH and other interested and potential cus-tomers world-wide.
Besides the in detail described technical functions and achievements, there are two aspects worth mentioning. The development of the XFEL Timing System involved many different disciplines like high-frequency electronic design and measurement techniques, high-speed se-rial data communication and synchronization, low-noise and high-speed printed circuit board design, FPGA firmware design, diver and software programming and more. Furthermore, the time is limited until a stable and production ready integrated solution has to be available in order to install the system in the European XFEL and FLASH and to be available as reference module for tests of other interfacing subsystems. Therefore a group of people was directly involved in the project located at DESY and the Stockholm University. Only due to their
combined effort, skills and discipline of communication, documentation and participation in regular meetings the project was successful in that limited time. Also the impressive active collaboration effort and participation within the working group in the PICMG consortium tak-ing care on the development of the MicroTCA.4 standardization is important to be mentioned.
Weekly telephone conferences and yearly workshops brought people together from America, Europe and Asia. The work within that group in terms of developing complementary ideas, simulating, designing and evaluating concepts and producing a new standard was an important and successful process. Since then an on-going close collaboration in the development of new products and industrialization based on that new standard had been started as well as further developments on firmware, software and protocol guide lines. All this is very important, when a transition to a new standard and technology takes place and a significant amount of new hardware and knowledge is required. Based on the observed continuous increase of interest from new users and the increasing number of producing companies and products, we seem to be on a good track.
Additionally to the previously mentioned very positive experience of the collaboration within the project, the standardization of MicroTCA.4 and the resulting influence on the success of this project, there is at least one other lesson learned. This is, that in complex projects with many stake holders not all aspects of requirements could be specified at the beginning, but that some aspects develop over time in an iterative process. As a consequence, conceptual designs, planned functions and limitations should be written down, communicated and feedback col-lected as early as possible to avoid creating artificial critical paths related to possible redesigns at a late stage.
Finally, the next steps related to the XFEL Timing System project should be outlined. The next most important milestone is the ongoing installation and commissioning of the timing sys-tem at FLASH2, the extension of the FLASH machine. This involves the procurement, setup and integration of XFEL Timing System modules and MicroTCA crates as well as bringing the firmware and software into the official release state. Besides that, the industrialization of the hardware and possible firmware is ongoing and expected to be finished within 2014. This pro-cess includes redesign steps related to the design tools and in order to optimize the production.
In 2014/2015 the procurement and setup of European XFEL hardware will be started, followed by first installations as well as further upgrade of FLASH and incremental replacement of the old timing system. In 2015/2016 the complete installation and commissioning for European XFEL is planned in order to prepare the start of user operation.
Chapter 14
Acknowledgments
Many people contributed directly or indirectly to this project and to my personal development.
I would like to take the time to pay credit and thank them.
I would like to start thanking Kay Rehlich, who used to be one of my supervisors during my work at DESY and who assigned me the task of the first conceptual design of the European XFEL Timing System and being officially responsible for the project. Since then he always supported me and we have a close relation with many interesting and inspiring discussions which also influenced my personal development. I also would like to thank Holger Schlarb, who used to be my other supervisor during my time working at DESY. He gave me the chance to take over responsibility for the firmware and software development and related coordination in the group of optical synchronization and special diagnostics as well as being responsible for the participation in an EU FP7 collaboration project. That allowed me to gain a much deeper understanding of the physics and timing related requirements of free-electron lasers, coordi-nation and intercoordi-national collaboration. Thank also goes to Christopher Youngman, Markus Kuster and Andreas Schwarz and their trust and confidence in me to set up and take over the joint DAQ and detector electronics group and who allowed and supported me to continue my work on the XFEL Timing System. Furthermore I would like to thank my doctoral thesis supervisor Klaus Schünemann, who allowed me to start this project and supported me since then. I also thank Friedrich Mayer-Lindenberg for being in the examination committee of this dissertation, for the collaboration between his working group and my group at the European XFEL, and finally his lectures related to FPGAs and DSPs, which became an important part of my current work. I also like to thank the remaining professors Sybille Schupp, Wolfgang Krautschneider and Arne Jacob, for being in the examination committee.
Great thank also goes to the colleagues and collaborators directly involved in the project. Be-sides Kay Rehlich these are in alphabetical order: Arthur Aghababyan, who is implementing and supporting the software, Kai-Erik Ballak, who participates in the testing, evaluation and measurement of the timing modules, Christian Bohm, who is the responsible person within the collaboration (European XFEL in-kind contribution) on the side of the Stockholm Univer-sity, Bruno Fernandes, who implemented the level converter adapter in order to provide timing related signals to consumers outside of the MircoTCA crate and develops FPGA firmware mod-ules to interface with the timing system within the crate, Attila Hidvegi, who is responsible for the schematic and PCB design of the timing and evaluation modules and also participates in the firmware and software development, measurements and evaluation, Holger Kay, who is im-plementing the firmware for the FPGA, Vahan Petrosyan, who is responsible for the firmware of the used micro controllers, Lyudvig Petrosyan, who is writing and supporting the linux driver for the timing system, Gevorg Petrosyan, who is supporting the firmware development of the FPGA and Christoph Stechmann, who is taking part in the measurements, tests and installation.
Besides this team, I would also like to thank all stake holders of the XFEL Timing System, who provided valuable feedback and suggestions for improvements. One of these persons is Matthias Werner, who not only identified critical issues, but also provided input and ideas for possible alternative solutions.
I also like to thank the members of the PICMG xTCA for Physics working group, who de-fined the new MicroTCA.4 standard, which was required in order to provide standard ways to distribute timing related signals within the MicroTCA crate and where industry and research laboratories could build on. Here I would like to single out Ray Larsen as the chair of the hardware committee and a person who provided good ideas and inspired me.
There are certainly more people, who contributed to this project, who I have not mentioned. I would like to thank all of them for their support and provided work, which finally lead to this successful implementation of the XFEL Timing System.
Finally I would like to thank my beloved wife and children for all that support and time they allowed me to work on this project and not being able to participate in the family activities.
LIST OF ABBREVIATIONS
M
MCH MicroTCA Carrier Hub MISO Multiple In Single Out
MTCA Micro Telecommunication Computing Architecture MRF Micro Research Finland
N
NI National Instruments NTP Network Time Protocol P
PAPR Peak-to-Average Power Ratio PCB Printed Circuit Board
PCI Peripheral Component Interconnect
PCIe Peripheral Component Interconnect Express
PICMG PCI Industrial Computer Manufacturer Group (www.picmg.org) PLL Phase Locked Loop
PSOF Phase Stabilized Optical Fiber PTP Precision Time Protocol R
RF Radio Frequency S
SER Symbol Error Rate
SFP Small Form factor Pluggable SNR Signal-To-Noise Ratio S/W Software
SyncE Synchronous Ethernet T
TCLK Telecommunication Clock TCP Transmission Control Protocol TDC Time to Digital Converter TDD Time Division Duplex TTL Transistor Transistor Logic U
UTC Coordinated Universal Time UDP User Datagram Protocol V
VME Versa Module Eurocard X
XFEL X-Ray Free Electron Laser
LIST OF SYMBOLS φ1(t) Time dependent phase of component 1 φ2(t) Time dependent phase of component 2 v¯2n Power spectral density
kB Bolzman constant
R Resistance
T Temperature
t1 Time when request was sent t2 Time when request was received t3 Time when reply was sent t4 Time when reply was received
∆tmaster−slave Link delay between master and slave
∆tserver−client Link delay between server and client tring Orbit period
tRF Period of RF frequency
N Number of bunches
l∆t Adjustable time delay circuit N Phase comparator circuits
÷/×/∆t Clock buffers with dividing, multiplication and adjustable time delay
∆tsynch Time between two successive synchronization pulses fsyncclk Frequency of synchronization clock
fuserclk Frequency of clock to be synchronized
TLoop Loop time from transmitter to receiver and back to master tD1 Time delay of first delay component
tEOT Time delay of electrical-to-optical converter at transmitter
tFT R Time delay of signal propagation through fiber (transmitter to receiver) tOER Time delay of optical-to-electrical converter at receiver
tCDRR Time delay of clock and data recovery component at receiver tEOR Time delay of electrical to optical converter at receiver
tFRT Time delay of signal propagation through fiber (receiver to tranmitter) tOET Time delay of optical-to-electrical converter at transmitter
tD2 Time delay of second delay component
tCDRT Time delay of clock and data recovery component at receiver TReceiver Time delay from transmitter to the receiver
dTReceiver Variation of time delay from transmitter to receiver
dTLoop Variation of loop time delay
n Number of whole data words
m Number of bits within the last word σ Sigma of Gaussian distribution
BIBLIOGRAPHY
Bibliography
[1] M. Altarelli et. al.XFEL Technical Design Report. DESY XFEL Project Group, Hamburg, 2007. ISBN 979-3-935702-17-1.
[2] Patrick Gessler. Entwicklung eines pikosekunden stabilen Timingsystems fuer das eu-ropaeische Roentgenlaserprojekt XFEL. TU Hamburg-Harburg and DESY, 2007.
[3] J. Johnson. Thermal agitation of electricity in conductors. Phys. Rev., 32:97, 1928.
[4] H. Nyquist. Thermal agitation of electric charge in conductors. Phys. Rev., 32:110, 1928.
[5] W. Schottky. Über spontane stromschwankungen in verschiedenen elektrizitätsleitern.
Annalen der Physik, 57(23):541–567, 1918.
[6] J.B. Johnson. Bemerkunge zur bestimmung des elektrischen elementarquantums aus dem schroteffekt. Annalen der Physik, 67(1):154–156, 1922.
[7] J.B. Johnson. The schottky effect in low frequency circuits. Physical Review, 26(1):71–85, 1925.
[8] W. Schottky. Small-shot effect and flicker effect. Physical Review, 28(1):74–103, 1926.
[9] Noise Analysis in Operational Amplifier Circuits. Texas Instruments, 2007.
http://www.ti.com/lit/an/slva043b/slva043b.pdf.
[10] Operational Amplifier Noise Prediction. Intersil, 1996.
http://www.intersil.com/content/dam/Intersil/documents/an51/an519.pdf.
[11] G. P. Agrawal. Nonlinear Fiber Optics - Third Edition. Academic Press, 2001. ISBN 0-12-045143-3.
[12] A. E. Siegman. Lasers. University Science Books, 1986. ISBN 0-935720-11-3.
[13] M. Salehi J. G. Proakis. Communication Systems Engineering - Second Edition. Prentice Hall, 2002. ISBN 0-13-095007-6.
[14] D. Mills et. al. Network Time Protocol Version 4: Protocol and Algorithms Specification (RFC5905). IETF RFC, 2010.
[15] D. Mills. Network Time Protocol Performance Analysis. University of Delaware, 2004.
https://www.eecis.udel.edu/ mills/database/brief/perf/perf.pdf.
[16] Lewis Carroll. Alice’s Adventures in Wonderland. 1865.
[17] G. Daniluk and T. Wlostowski. White rabbit: Sub-nanosecond synchronization for em-bedded systems. Proceedings of the 43rd Annual Precise Time and Time Interval Systems and Applications Meeting, pages 45–60, 2011.
[18] P. Moreira et. al. White rabbit: Sub-nanosecond timing distribution over ethernet. In-ternational Symposium on Precision Clock Synchronization for Measurement, Control and Communication, pages 45–60, 2009.
[19] M. Lipinski et. al. White rabbit: a ptp application for robust sub-nanosecond synchroniza-tion.International IEEE Symposium on Precision Clock Synchronization for Measurement Control and Communication, pages 25–30, 2011.
[20] W. Stallings. Data and Computer Communications (9th Edition). Prentice Hall, 2010.
ISBN 0-13-2172178.
[21] A. X. Widmer and P. A. Franaszek. A dc-balanced, partitioned-block, 8b/10b transmission code. IBM Journal of Research and Development, 27(5):440, 1983.
[22] Jukka Pietarinen. Timing System Evolution - Progress Towards Synchronous Data Distribution. EPICS Meeting Vancouver, 2009.
http://www.mrf.fi/pdf/presentations/MRF.EPICS2009.pdf.
[23] Jukka Pietarinen. Timing System with Two-Way Signaling, cRIO-EVR. EPICS Meeting Padova, 2008. http://www.mrf.fi/pdf/presentations/MRF.EPICS.2008.Padova.pdf.
[24] Jukka Pietarinen. Latest Timing System Developments. EPICS Meeting Shanghai, 2008.
http://www.mrf.fi/pdf/presentations/MRF.EPICS.2008.Shanghai.pdf.
[25] Jukka Pietarinen. MRF Timing System. CERN Timing Workshop, 2008.
http://www.mrf.fi/pdf/presentations/MRF.CERN.Feb2008.pdf.
[26] Y. Chernousko et. al. Diamond timing system developments. Proceedings of International Conference on Accelerator and Large Experimental Physics Control Systems, pages 244–
246, 2003.
[27] Y. Chernousko et. al. Review of the diamond light source timing system. Proceedings of the Russian Partical Accelerator Conference, pages 144–146, 2010.
[28] J. Dusatko et. al. The lcls timing event system. Proceedings of the Beam Instrumentation Workshop, pages 379–383, 2010.
[29] Master oscillator for the european xfel. Proceedings of of the International Particle Accel-erator Conference, pages 2771–2773, 2014.
[30] D. McDowell R.J. Pasquinelli. Fiber Optic Delay Tracking Experiment. Fermi National Accelerator Laboratory, 2000. BD RFI Note No.001.
[31] Specification of Phase Stabilized Optical Fiber Cable. Furukawa Electronic Co Ltd, 2004.
Datasheet.
[32] M. Lorek S. Thyagarajan. Jitter Reduction Techniques for Phase Locked Loops in Deep-Submicron Technologies. EECS UC Berkeley, 2012.
http://www.eecs.berkeley.edu/ sivavth/EE241_Midtermreport_Lorek_Siva.pdf.
[33] T.H. Smilkstein. Jitter Reduction on High-Speed Clock Signals - Tech-nical Report No. UBC/EECS-2007-96. EECS UC Berkeley, 2007.
http://www.eecs.berkeley.edu/Pubs/TechRpts/2007/EECS-2007-96.pdf.
[34] ITU-T G.652 Characteristics of a single-mode optical fibre cable. International Telecom-munication Union, 2000. http://www.iet.unipi.it/m.luise/HTML/AdT/ITU_G652.pdf.
BIBLIOGRAPHY
[35] J.P.H. Sladen E. Peschard. Phase compensated fibre-optic links for the lep rf reference distribution. Proceedings of the IEEE Particle Accelerator Conference, 3:1960–1962, 1989.
[36] D. Marcuse. Principles of Optical Fiber Measurement. Academic Press, 1981. ISBN 0-12-470980-X.
[37] Analog Devices. Continuous Rate 12.3 Mb/s to 2.7 Gb/s Clock and Data Recovery IC with Integrated Limiting Amp - ADN2812. Analog Devices, Inc., 2012.
[38] Analog Devices. LF 2.7 GHz RF/IF Gain and Phase Detector - AD8302. Analog Devices, Inc., 2002.
[39] Micrel. 2.5V/3.3V 1.5MHz Precision LVPECL Programmable Delay with Fine Tune Con-trol - SY89296U. Micrel, Inc., 2006.
[40] SFF Committee. INF-8074i Specification for SFP Transceiver. SFF Committee, 2001.
ftp://ftp.seagate.com/sff/INF-8074.PDF.
[41] P. Gessler et.al. A pico-second stable and drift compensated high-precision and low-jitter
[41] P. Gessler et.al. A pico-second stable and drift compensated high-precision and low-jitter