The CMS HGCAL trigger data receiver

As part of the CMS Phase-2 upgrade, a prototype receiver for the HGCAL endcap front end has been implemented using the Serenity ATCA platform. The receiver firmware was developed to test the unpacking of data from the front-end endcap trigger concentrator ASIC (ECON-T) and measure its performance and stability. The firmware was integrated with prototype DAQ firmware as well as ancillary blocks to generate a trigger using ECON-T data, process scintillator trigger and timing distribution system, to evaluate the complete HGCAL vertical slice. The data was read out using custom 10G UDP links and upgraded to CMS DTH system at 25 Gbps. The system successfully achieved prototype TPG Stage-1 and DAQ path readout using generated beam triggers and delivered ~20 TB of data containing physics events at an average trigger rate of about 100 kHz.


Introduction
The High Granularity Calorimeter (HGCAL) [1] is designed to replace the current end-cap detector of the CMS experiment [2].The HGCAL consists of a mix of hexagonal silicon pad sensors and scintillator tiles as the active sensing element of the detector.The HGCAL detector is proposed to be segmented into 47 sensor layers which will be separated with absorber material and cooling mechanics.The first 26 layers will consist of hexagonal silicon sensors consisting of 100-400 hexagonal pad cells [3], with a smaller pad size (0.52 cm 2 ) closer to the beam and larger pad size (1.18 cm 2 ) away from the beam.The next 21 layers will be a mixture of silicon sensors near the beam and scintillator tiles towards the periphery of the detector.
The HGCAL front-end (FE) system includes several custom radiation-hard ASICs to accomplish the readout [4].The HGCROC is a digitiser ASIC directly interfaced to individual cells (i.e.pads) of the silicon sensor [5,6].It consists of a multichannel ADC and TDC and creates independent data streams for the DAQ and the trigger path.The HGCROC is followed by concentrator ASICs (ECON-T and ECON-D) which aggregate data from multiple HGCROCs [7,8].Aggregated data is serialised using LpGBT ASICs [9,10] and transmitted to the back-end (BE) system using optical transceivers (VTRx+) [11].The HGCAL Trigger Primitive Generator (TPG) will construct calorimeter clusters for the CMS Level-1 trigger decision using trigger cells (TC) from the FE chain.Stage-1 of the TPG is responsible for decoding the compressed TCs from the trigger concentrator ASIC (ECON-T), filtering the TCs, and sending them to the system's time-multiplexed (x18) Stage-2 boards.The HGCAL BE systems (TPG and DAQ) will be implemented using the Serenity ATCA platform [12].A vertical slice of the HGCAL readout electronics system for the silicon section is shown in figure 1.

HGCAL trigger receiver
The baseline architecture of the HGCAL back-end (BE) system is asymmetric.The DAQ BE is responsible for slow control, fast control and DAQ stream readout (on reception of a Level-1 accept), while the trigger BE is a read-only streaming system.The trigger data is sent for every LHC clock cycle (40 MHz) and has to forward trigger primitives, i.e. 3D clusters and towers, to the Level-1 trigger -1 - system with a fixed low latency.Out of the 47 layers, only 34 layers take part in the trigger.Each of 84 TPG Stage-1 cards (trigger receivers) is expected to handle about 300 ECON-Ts over 120 LpGBT links.This would amount to about 5k links per end-cap of the detector resulting in an FE bandwidth of ~50 Tbps.These requirements make the trigger system very challenging.

Trigger concentrator ASIC and the unpacker firmware
The front-end endcap trigger concentrator (ECON-T) provides on-detector data compression using different compression algorithms which can be selected at the run-time.The initial baseline design envisaged the use of the Threshold Sum (TS) algorithm which selects TCs above a programmable threshold.Such dynamic selection offers high compression but results in variable length and latency of the data.Considering the baseline proposal and ability to read more information with TS, it was used extensively to test and validate ECON-T functionality.The BE firmware and the FE ASICs were tested in steps, integrating one ASIC of the chain at a time.At each step, TCs received and decoded at the BE were compared against expected values from the ECON-T (constant and PRBS patterns).Ultimately, the challenging task of integrating the complete trigger chain was achieved by successfully validating TCs generated at HGCROC at the BE.The unpacker firmware mainly consisted of unpacker blocks to decode ECON-T packets, error detection mechanisms and online data analysis by histogramming different parameters.Status and error counters were periodically read out using an onboard control bus and metrics were displayed using a Grafana web interface.Tests were repeated with different compression algorithms and packet types to validate the performance of the ECON-T and unpacker firmware.Additional tests were also conducted to establish the reliability of the communication interfaces.The link quality test carried out with constant data patterns showed a bit error rate to be better than 10 −12 , while the ECON-T ASIC reliability test showed zero errors in frame sequence numbers and zero malformed frames over several days.
Variable latency and dynamic packet formats of TS proved complex and resource-intensive to decode and hence full-scale TPG Stage-1 firmware builds with TS were not viable.The fixed latency algorithms are much easier to handle and have been shown to offer comparable performance in detector simulation studies.Thus, fixed latency algorithms -Best Choice for the electromagnetic section and Super Trigger Cells (STC) for the hadronic section -have been selected as the new baseline.The prototype firmware development for TPG Stage-1 leveraged the EMP firmware framework which provides essential interfaces for the Serenity platform.A few more ancillary firmware blocks had to be developed to establish proper communication with the FE, such as a TCDS2 emulator for fast command generation [13] and a priority encoder for encoding and distributing fast commands to the front end.

HGCAL trigger receiver testing at beam test
Once firmware and FE hardware were qualified using bench-top testing, the next step was to build a complete vertical slice of the HGCAL FE and BE system to be tested with a test beam at CERN.Both branches of the BE system (DAQ and TPG) were to be validated during the beam tests since standalone TPG (trigger) system validation would be incomplete without a high-resolution DAQ path readout.To simplify implementation in the absence of an actual TCDS2 (timing distribution system) decision was made to develop an integrated prototype firmware with down-scaled TPG and DAQ systems on the same FPGA.Over the summer of 2023, two beam tests were conducted at CERN.That was the first complete integration test for the HGCAL collaboration.The setup consisted of two low-density silicon modules with their associated HGCROC and ECON ASICs (ECON-D and ECON-T) as well as an LpGBT readout board (engine).The two modules were placed one after another on cooling plates and were preceded by 10 Pb absorber plates.The whole FE setup was housed in a cool box with continuous temperature and humidity monitoring.A dedicated scintillator trigger acquisition system was developed using another LpGBT to sample the scintillator coincidence signal at 1.28 Gbps and send to the BE.This allowed measurement of the trigger time with respect to the 40 MHz internal clock to a precision of 0.78 ns.The first beam test used ECON emulators housed in a dedicated FPGA board at FE in absence of actual ECON ASICs.

Integrated TPG and DAQ firmware
The block diagram of the integrated TPG and DAQ firmware is shown in figure 2. The firmware was targeted for the KU15P FPGA.Apart from DAQ and TPG blocks, the system also included a scintillator trigger processor as well as ECON-T self-trigger firmware to trigger on the beam.The fast command firmware played a crucial role in the distribution of timing and fast command signals to FE as well as BE data processing blocks and hence coordinated the data flow of the whole system.As part of the fast command system, a TCDS2 emulator was developed to mimic the CMS standard fast command and timing signal distribution.The emulator included a fast command sequencer to program and execute arbitrary fast command sequences such as calibration sequences, internal random and periodic trigger generators to do pedestal or test runs and throttling logic to protect the system overflow.
-3 - The DAQ stream only included ECON-D data received from the front end while the TPG stream included ECON-T raw data, ECON-T unpacked data, scintillator raw data as well as TC processor output data, for a range of bunch crossings around true L1A.The extended readout for the trigger blocks allowed for offline validation of firmware blocks.The typical event size on each stream was about 2 kBytes.The DAQ and TPG streams were read out using a custom 10 Gbps UDP readout directly to a PC.Due to packet size limitation, DAQ streams corresponding to two modules were read out using separate UDP links.Alternatively, an interface with a CMS standard DTH board (DAQ and Timing Hub) using the Slink protocol (at 25 Gbps) was also implemented to read out DAQ and TPG data streams to the PC.The latter completed the whole vertical slice of the HGCAL FE and BE system.The data-taking runs were managed by central run control software which read out the configuration, started DAQ software and also sent the run number to the online DQM system.The complete system schematic is shown in figure 3.

Trigger, TPG data and DAQ data alignment
Aligning the multiple data paths in the system so that the beam trigger (external or internal) correctly picked up DAQ data from the HGCROC buffer and TPG data from the BE latency buffer was one of the trickiest parts of the system commissioning.HGCROC's internal calibration (charge injection) feature was used to align the DAQ event readout from the HGCROC buffer and the ECON-T stream.the HGCROC buffer depth could be tuned by adjusting the delay between the injection fast command and an L1A.Using the self-trigger mechanism of the ECON-T unpacker, the TPG buffer latency in the firmware was adjusted so the DAQ and trigger streams were aligned.The same exercise was also carried out for the scintillator trigger to eventually align both triggers.Figure 4

Results and conclusions
Once the data paths of the system were synchronised, it was used to record data for different beam conditions.Namely, data with high-energy (200 GeV) electron and pion beams were recorded, and electron energy scans were performed as well as position scans of the module.The recorded data was used to validate the accuracy and consistency of the trigger receiver firmware (unpacker) by comparing ECON-T raw data and unpacker output.Figure 5(a) shows STC energy distributions for one STC decoded from ECON-T raw data as well as unpacker output in perfect agreement.Along with the physics runs, different engineering runs were also recorded to assess the performance of the system under different operating conditions such as unexpectedly fast command sequences and also to stress test the system with an artificially boosted trigger rate.The system was able to run with an average trigger rate of about 100 kHz during physics runs and more than 500 kHz of instantaneous rate during engineering stress test runs.
The prototype integrated BE system allowed successful testing and validation of some major BE firmware components and FE systems including recently available ECON ASICs.A complete vertical slice of the HGCAL FE (silicon region) and BE system including data readout using CMS standard -5 -

Figure 1 .
Figure 1.Slice of HGCAL readout electronics for the silicon section.

Figure 2 .
Figure 2. Block diagram of the integrated TPG Stage-1 and DAQ firmware for beam test.

Figure 3 .
Figure 3. Beam test setup with FE and BE chain along with software interface.
(a) shows time distributions of scintillator and ECON-T self-trigger after synchronisation and figure 4(b) shows a typical STC energy distribution (used for internal/self-trigger) as a function of the scintillator trigger arrival time.-4 -

Figure 4 .
Figure 4. (a) Distribution of scintillator trigger arrival time and ECON-T derived self-trigger arrival time (in steps of 0.78 ns).(b) Typical STC energy (encoded) distribution with scintillator trigger arrival time (in steps of 0.78 ns).

Figure 5 .
Figure 5. (a) Histogram of raw (blue) and unpacked (red) STC energy during calibration runs.(b) 2D histogram of beam signals for each HGCROC channel, i.e. hexagonal pad.(c) ADC values for a single channel in the beam region as a function of scintillator trigger arrival time (in steps of 0.78 ns).This is a direct measurement of the HGCROC pre-amp shaper response.

Figure 5 (
b) shows a 2D histogram of beam signals in one module and figure 5(c) shows ADC data for one channel as a function of trigger phase, clearly showing the effect of HGCROC sampling and shaping mechanism.