A GLIB-based uTCA demonstration system for HEP experiments

The Gigabit Link Interface Board (GLIB) project is an FPGA-based platform for users of high-speed optical links in high energy physics (HEP) experiments. The project delivers hardware, firmware/software and documentation as well as provides user support. These resources facilitate the development of evaluation platforms of optical links in the laboratory as well as triggering and/or data acquisition systems in beam or irradiation tests of detector modules. This article focuses on the demonstration of a triggering and data acquisition setup for HEP experiments using hardware and firmware/software resources provided by the GLIB project.


Introduction
The future upgrade of the LHC accelerator, the HL-LHC, will increase the beam luminosity by a factor of ten leading to a corresponding growth of the amounts of data to be treated by the data transmission and acquisition systems [1].
In order to address this issue, a new GBT-based high-speed radiation-hard optical link is currently under development that in this article will be referred-to as "the new link" [2].The achieved line rate of 4.8Gb/s increases the data throughput by up to a factor of 6 with respect to the current GOL-based link [3] (operating at either 800Mb/s or 1.6Gb/s).A key feature of the new link is its bi-directionality that, in addition to the readout of detector data, allows the transmission of Trigger/Timing and fast Control (TTC) [4] information as well as the slow control of the frontend (FEC) through a single point-to-point interconnection, contrary to the current implementation where dedicated hardware is used for TTC, FEC and data readout using totally independent links.The channel unification of the new link simplifies the topology and reduces the number of optical links.On the other hand, this topology introduces new technical challenges related with the reference clock that will have to be recovered from the incoming data stream [5], while in the current topology the clock is forwarded through a dedicated path.
A simplified diagram illustrating the two link generations is shown in figure 1.
A diagram of a typical system featuring the new link is shown in figure 2, highlighting its major components.On the off-detector side, a Back-End (BE) card acts as the single-connection point with the detector transmitting TTC information, controlling & configuring the Front-End (FE) as well as receiving and forwarding detector data to the central data acquisition (DAQ).On the ondetector side, the GBTx [2] serializer/deserializer (SERDES) ASIC forwards the TTC information to FE ASICs and reads them out through low-speed (80, 160 or 320 Mb/s) electrical links named Elinks [6].The physical link between the BE and the GBTx ASIC is known as the "Versatile Link" (VL) [7] and its major component is a custom plug-in module performing optical-to-electrical  conversion (and vice versa) named the Versatile Link transceiver (VTRx) [8].It is important to mention that only custom parts are used on-detector since they have to cope with extremely high radiation levels.
In addition to the principal new link developments i.e. the GBTx and the VL, a side-project named the Gigabit Link Interface Board (GLIB) [9,10] has been launched in order to facilitate the integration of the new link into a system.More specifically, the GLIB is a development platform that integrates hardware, firmware/software, documentation and user support.The major hardware component is an FPGA-based double-width Advanced Mezzanine Card (AMC) (figure 3, left) [11] featuring Multi-Gigabit Transceivers (MGTs) conceived to operate either inside a µTCA [12] shelf or stand-alone on a bench.In order to ensure the GLIB AMC compatibility with legacy and future interfaces as well as enhance its I/O bandwidth, we have also developed three additional FPGA Mezzanine Cards (FMCs) [13], namely the TTC FMC, VL FMC and E-Link FMC [14] shown in figure 3, right.The TTC FMC receives optically transmitted TTC information recovering the encoded clock and serial data, the VL FMC hosts VTRx modules that require a custom interface while the E-Link FMC translates the I/O levels of the E-Link (currently not compatible with most FPGA families) to LVDS.
One of the major milestones of the GLIB project was the development of a hardware setup demonstrating the in-system implementation of the new link that could serve both for system feasibility studies as well as a start-up kit for developers.However, many of the components of this new link are currently still in an early phase of development.Among the off-detector components, only the legacy TTC system (currently used by the LHC experiments) is available whereas the other off-detector components (the BE card and the DAQ) are not compatible with the new link yet.Concerning the on-detector components, the VTRx is in preproduction stage (several prototypes have been delivered to selected users), the GBTx is already in a prototype stage (but still not available to -2 -  users) and the FE ASICs are not widely available either.For that reason, the first prototype of the above mentioned setup emulates the functionality of the currently unavailable components.This article focuses on the first prototype implementation of that setup which will be referred-to as the "demonstration system".

Implementation
The implementation of the demonstration system depicting its main components mapped to a simplified diagram of the new link is shown in figure 4. Those components as well as some results of phase and latency determinism studies are presented in the subsections that follow.For simplicity reasons, the flow of information towards the FE (shown with orange arrows) will be referred-to as "downstream" whilst the flow in the opposite direction (shown with blue arrows) will be referred-to as "upstream".

TTC source
The first hardware component of the demonstration system to be described is the TTC source (see figure 5).As aforementioned, the new link uses the legacy TTC system.For this demonstration, the TTC source is a standard VME-based TTC system.However, to improve the portability of our setup and make it easy to replicate, we decided also to implement an FPGA-based TTC source using a commercial FPGA development kit and the TTC cores provided by the TTC team [15].This emulated TTC source has also been used in this demonstration system.

Back-End card & DAQ
The BE card is composed of a GLIB AMC carrying a TTC FMC (see figure 6).Like in a real experiment, the BE card is inserted in a shelf, which in this case is a µTCA standard shelf (also used in several upgrades of LHC experiments).
The role of the DAQ is played by a processor AMC that for simplicity reasons (as well as for exploring the PCIe capability of the GLIB AMC) is inserted in the same shelf as the BE card (see figure 6).
-4 - The functionality of the BE card is shown in figure 7.In the downstream direction, the TTC FMC (which is optically connected to the TTC source) separates the clock and the data from the incoming TTC frame.The recovered clock is then used by the GLIB AMC as reference clock for the FPGA logic and MGTs, whereas the data is also forwarded to the FPGA where it is decoded by the TTC core, encoded into GBT-fame format by the GBT-FPGA core [2,16] and sent to the GBTx through the VL using an SFP+ transceiver [17].In the upstream direction, the GLIB AMC receives the GBT-encoded readout data from the GBTx through the VL.Then, within the FPGA, the GBT-FPGA core decodes the incoming GBT-frame, adapts it to a format compatible with the PCIe core and finally, sends it to the DAQ where is stored for further processing.In this system prototype, the FEC data path is not implemented.

GBTx
As previously mentioned, the GBTx ASIC is not available yet.For this reason, a GLIB-based GBTx emulator consisting of a GLIB AMC carrying an E-link FMC and a VL FMC equipped with a VTRx (see figure 8) is used instead.
The role of the GBTx ASIC is played by the GBT-FPGA core (see figure 9) in the GLIB.In the downstream direction, the GBT-FPGA core deserializes the incoming TTC from the VL and serializes it again towards the FE ASICs through the E-link.In the upstream direction, the GBT-FPGA core deserializes the incoming readout data from the E-link and is serializes it again to be sent towards the BE system through the VL.
During the course of the development of this demonstration system, members of the GBTx test team were testing the first prototype of the GBTx using a custom board (Stand Alone Test (SAT) board [2]) featuring optical and E-link ports, an FPGA (for testing and I/O compatibility purposes) as well as the GBTx ASIC prototype inserted in a socket (see figure 10).The SAT board was successfully used to replace the GLIB-based emulator version of the GBTx in the demonstration system, so then implementing the first full system including a GBTx ASIC.

Front-End ASIC
The last hardware component of the demonstration system is the E-link interconnected FE ASIC.Due to the limited availability of these ASICs, a realistic emulation of one FE ASIC using an FPGA-based development kit carrying an E-link FMC was implemented instead of the real FE ASIC.The implementation and characteristics of the FPGA-based FE ASIC emulator is shown in figure 11.
-6 -  Concerning functionality, the emulated FE ASIC receives the TTC through the E-link.After each trigger, the emulated FE ASIC generates a readout data frame that mimics data from the detector and sends it to the GBTx through the E-link.
The diagram and readout data frame structure of the emulated FE ASIC are shown in figure 12.

TTC phase & latency determinism
The reference clock of the FE ASIC, forwarded through the downstream path, is used for sampling the analogue signals from the various detectors.In order to sample properly, the TTC must be deterministic in phase and latency.
As mentioned in the introduction, the reference clock for the FE ASIC in the new link is recovered from the incoming data stream.The clock recovery procedure may lead to uncertainty in the phase and latency of the TTC.
In order to avoid this uncertainty, the demonstration system was designed in such a way to ensure phase and latency determinism of the TTC by optimising MGTs and Clock Domain Crossings (CDC).This determinism was verified through a preliminary manual test by adding some flags in critical points along the downstream path and measuring them with an oscilloscope (see figure 13).Nevertheless, further studies of the TTC phase & latency determinism based on automated repetitive measurements are foreseen and will be reported elsewhere.

Software
In addition to the hardware and firmware described above, dedicated software had to be developed for control and monitoring purposes.The control of the different components of the system is done through custom scripts for Windows, based-on a new generation software called IPbus [18] running over Ethernet.This control software is the equivalent of the legacy HAL over VME in the current CMS experiments.The monitoring is done through a custom software for Linux (executed on the processor AMC) running over PCIe.One screenshot of the monitoring software displaying part of a readout data frame is showed in figure 14.

Summary and outlook
One of the major milestones of the GLIB project for 2013 was the implementation of a demonstrator system featuring the new link for LHC upgrades, in order to be used as example and/or starting -8 -point by developers of new generation HEP systems.In summer 2013 we have managed to deliver such a system even though several of the components of this new link were either in an early phase of development or not available at all.In order to implement the first prototype of the system, we had to emulate the functionality of the components unavailable at that time using resources from the GLIB and other projects.With that prototype system, we have successfully demonstrated the feasibility of the new link, including some of its important features such as its clock phase and latency determinism in the downstream path, something extremely important for the correct sampling the analogue signals of HEP detectors.In early September 2013, we have also managed to successfully integrate to our system a board carrying a GBTx ASIC (arrived few weeks before).For the future, we are planning to develop an automated procedure for testing intensively the TTC phase and latency determinism.In addition, we also envisage improving and updating this new system with real components when they become available.We strongly believe that this new system could well be the starting point for test/commissioning systems in beam and/or laboratory setups.

Figure 2 .
Figure 2. Detailed diagram of typical system featuring the new link.

Figure 4 .
Figure 4. Implementation (bottom) and diagram (top) of the demonstration system.

Figure 7 .
Figure 7. Diagram of the Back-End card and DAQ.

Figure 10 .
Figure 10.Stand Alone Test (SAT) board featuring a GBTx ASIC inserted in a socket.

Figure 11 .
Figure 11.Implementation (left) and characteristics (right) of the FPGA-based FE ASIC emulator.

Figure 12 .
Figure 12.Diagram (top) and readout data frame structure (bottom) of the emulated FE ASIC.

Figure 14 .
Figure 14.Screenshot of the monitoring software displaying part of a readout data frame.