40 MHz triggerless readout of the CMS Drift Tube muon detector

The Level-1 trigger scouting system of the CMS experiment aims at intercepting the intermediate data produced by the L1 trigger processors before the final trigger decision. This system can be complemented by adding the raw stream of data collected from the detector front-end, whenever the throughput is manageable. In this work, the triggerless readout of the CMS Drift Tubes (DT) detector is presented. This is realized by reading a sector of the DT which has been equipped with the preproduction of Phase-2 upgrade front-end boards. A Xilinx VCU118 acts as a concentrator of the Phase-2 demonstrator lpGBT links and transmits data to a server via 100G TCP/IP. First results coming from a test-stand mimicking the sector demonstrator are shown.


Introduction
The 40 MHz L1-trigger scouting project [1] of the CMS experiment [2] aims at capturing intermediate data produced by the first stage of trigger processors (L1) at the full bunch crossing rate, to perform online analysis independently of the final trigger decision.The system works as a parallel readout chain that processes copies of the streams between the L1 trigger boards, which are obtained via spare optical links.The data is then concentrated using commercial development boards equipped with FPGAs and transmitted to dedicated computing resources.
The stream of L1 trigger data can be complemented with the raw stream of data collected from detector front-end boards, when the throughput is manageable in terms of links bandwidth and computing resources.This would enable the reconstruction of physics processes without the bias introduced by the L1 trigger, on top of real-time diagnosis of the detector status.The CMS Drift Tubes (DT) muon detector offers a perfect candidate for the implementation of a triggerless-readout system thanks to its low occupancy.One of the DT sectors has been equipped with front-end boards designed for the Phase-2 upgrade of the CMS DT system, the OBDT (On detector Board for the Drift Tube chambers) [4,5].These boards digitize the chamber's time signals, referred as TDC hits (Time to Digital Converter), and transmit them to the L1 trigger boards via high-speed optical links using the lpGBT protocol.Using secondary spare links in the OBDTs, a copy of such data is sent to the 40 MHz DT scouting system.In this context, this demonstrator system, called slice test, is used to validate the OBDT boards in situ.Therefore, the 40 MHz DT scouting is a tool for evaluating and debugging the new on-detector electronics.
A commercial Xilinx Virtex Ultrascale+ FPGA evaluation board is used to collect the front-end links.The input streams from the lpGBT links are merged into frames and transmitted to a dedicated server via a 100G TCP/IP link.The target protocol for this DAQ link is an RDMA protocol.The current TCP/IP implementation is used as a starting point of the development and for performance comparisons with the one based on RDMA.

Hardware implementation
The demonstrator of the readout of the OBDTs at 40 MHz is based on COTS electronics.A schematic view of the system is shown in figure 1.The Xilinx VCU118 evaluation board is used to collect hits sent by the secondary links of the OBDTs, while primary links are routed to a prototype of the -1 -

JINST 19 C02050
Phase-2 backend that implements a local trigger algorithm and sends data to the central CMS DAQ every L1 accept signal produced by the CMS trigger [6].The VCU118 is equipped with an FMC+ module that can host up to 6 QSPF transceivers able to receive up to 24 lpGBT links that correspond to the 24 OBDTs, as there is one link per OBDT.The connection to the CMS Timing and Control Distribution System (TCDS) [8] has been realized via a custom board that receives the clock and fast control from the current Phase-1 electronics, and delivers them to the VCU118.The firmware developed for the 40 MHz DT readout demonstrator is based on the CMS EMP Framework [9], which is currently under development for the Phase-2 upgrade, in order to be fully compliant with the protocols and standards that will be adopted in CMS.This framework contains firmware modules common to multiple subsystems, such as the implementation of the lpGBT-FPGA firmware [7] for decoding the input links, and infrastructural firmware with slow control and debugging capability.Moreover, it provides the needed software for controlling and monitoring the firmware.The existing lpGBT module has been instantiated along with the payload formatter and the spy buffers to capture the data stream for debugging.Besides the TCDS decoder present in EMP, a firmware module allowing the integration with the current TCDS [8] has been added.
From the EMP input buffers, data are zero suppressed and only the hits are written into Clock Domain Crossing (CDC) FIFOs to decouple the LHC clock synchronous readout of the lpGBT links to the asynchronous data transmission towards the server.A payload builder process is in charge of reading hits from all the FIFOs, one per channel, and it merges them in a single data stream that feeds a 100G TCP/IP module implemented using a scalable network stack for FPGA developed by ETH [10].This framework allows for a flexible configuration of the number of concurrent connections and the width of the data path.The choice of the network protocol is also parameterized at compiling time.For our implementation, we opted for the full TCP/IP stack bufferized on an external DDR4 memory for retransmission in case of lost or malformed packets at the receiver.point-to-point connection is established between the ConnectX NIC and one of the VCU on-board QSFP.The data stream is received by the Readout Server with a standard TCP socket and temporarily stored on a RamDisk.The data is then transferred to storage for long-term preservation and offline analysis.An online data processing pipeline, capable of data monitoring and automatized detection of anomalies in the data stream, has also been developed, and it is discussed in another work [3].The VCU board is mounted through an extender on the PCIe bus of a Control Server, in charge of the monitoring and the slow-control of the board.These operations consist of simple actions such as resetting the links and checking their status, and are implemented using IPBus registers.The user interface is provided via the EMP software.

First results
The system has been developed, tested, and first deployed at the INFN Legnaro National Laboratories (LNL), where a replica of the components installed at CMS for the slice test is available.The setup available in LNL consists of 4 OBDTs-phi boards, and their control and timing distribution systems.A campaign of data-taking has been carried out in two scenarios by collecting pure electronics noise, and by injecting periodic signals to the front-end.
In the first data-taking condition, the threshold of the chambers' front-end discriminator is set to zero.This configuration allows all signals generated at the front-end level to be passed to the OBDTs and digitized.Since the triggerless readout collects the whole set of TDC hits, the data acquired in this way is dominated by the noise induced by the electronics, and not by genuine hits produced by muons crossing the detectors.In the case of pure white noise, a flat distribution in the TDC bins is expected.This allows for checking the differential nonlinearity (DNL) of the TDCs generated by the OBDTs by measuring the distance of each bin from its ideal width of 25/32 ns.The latter is due to the TDC implementation, which samples the signal 32 times per each LHC bunch cycle, of 25 ns.The results of this study, presented for a randomly selected OBDT channel in figure 2, show that the DNL is within the desired range of ±10% for each channel.The OBDT boards are designed to be capable of generating periodic signals that can stimulate the chamber's front-end output, called Test Pulses (TPs).Enabling TPs on the OBDTs generates a train of signals with a user-defined time period, but always produced in correspondence of the same TDC bin for every TP.Ideally, collecting a train of TPs will thus result in a histogram spanning -3 -exactly one TDC bin.However the TDC histogram of a TP could happen to be larger due to the convolution of the TP generation and TDC uncertainties. Figure 2 shows the resulting TDC histogram produced in a TP run for a randomly chosen OBDT channel.

Summary and future work
This work has described the triggerless readout system used for the validation of the new front-end electronics boards (OBDTs) of the Drift Tubes (DT) component of the CMS experiment.The system has been developed and tested in the Legnaro INFN National Laboratories, using a replica of the DT detector and the Phase-2 Upgrade electronic components, and is currently under evaluation in CMS as part of the DT slice test.The readout system here described was employed to collect the data used to validate the OBDT's TDC implementation.The current triggerless readout prototype is based on the CMS EMP Framework, and it implements data transmission over a 100G TCP/IP link.Efforts are currently directed towards the transition from TCP/IP to ROCE (RDMA Over Converged Ethernet), as part of the INFN FEROCE project (Front End ROCE).An implementation of ROCEv2 protocol is already available in the network stack used for the 100G TCP/IP, and it is currently being employed for the development of a real-time firmware simulation of the system, as discussed in [11].

Figure 1 .
Figure 1.Schematic representation of the DT 40 MHz readout system.From left to right: the connection to the CMS TCDS and secondary lpGBT links from the OBDTs are received by a VCU118.A simplified illustration of its main components is shown in the middle box.The firmware is based on the CMS EMP framework, connected via a set of FIFOs and a payload builder process to a 100G TCP/IP module.Board and firmware are monitored and controlled via PCIe.A second server is used to receive and process the TCP/IP stream.

Figure 2 .
Figure 2. First results from the LNL test stand.On the left, distribution and differential nonlinearity of the TDC hits produced by the OBDTs.On the right, the TDC distribution for the periodic Test Pulse signal.