CMS HGCAL electronics vertical integration system tests

: In preparation for the High-Luminosity phase of the CERN Large Hadron Collider, the Compact Muon Solenoid collaboration will replace the existing endcap calorimeters with the High Granularity Calorimeter. Considering both endcaps, this novel sub-detector will have around six million readout channels, pushing approximately 108 Tbit s − 1 of usable data between the detector front-end (on-detector) electronics and the back-end (off-detector) electronics. Given the scale and complexity of the endcap calorimeter upgrade project, the electronics testing must be carefully planned. The strategy has been to split the efforts between vertical (start-to-end) and horizontal (parallelisation) test systems wherever possible. An important milestone was the development and operation of a test system that prototypes a complete vertical slice of the future endcap electronics system. For the first time, a version of a test system consisting of the full vertical electronics readout chain was successfully operated in a beam test, where it was used to acquire real physics data.


Introduction
The high-luminosity (HL) upgrade of the Compact Muon Solenoid (CMS) detector at CERN's Large Hadron Collider (LHC) will include the replacement of the existing endcap calorimeters with a novel device -the High Granularity Calorimeter (HGCAL) [1].It will measure collision products within a calorimetric context, providing precise information regarding their spatial location, temporal occurrence, and energy deposition, with resolutions as good as 1 cm, 10 ps, and 0.3 MIP (Minimum Ionising Particle) respectively, over large dynamic ranges.The design employs silicon sensors in areas with the most intense radiation and plastic scintillator tiles with on-tile silicon photo-multipliers (SiPMs) in the less challenging regions.
The final HGCAL electronics system is highly specialised and complex, involving multiple layers of data transfer and control as illustrated in figure 1 [2].It has two distinct parts, the front-end (FE) electronics and the back-end (BE) electronics.In both these parts there are two sub-systems that co-exist, the Trigger Primitive Generator (TPG) and the Data Acquisition (DAQ) system.The TPG BE acquires coarse resolution data from the FE for every proton bunch crossing (BX) that happens at 40 MHz, processes those data, and sends the results to the Level-1 Trigger System (L1T).There, a fast Level-1 Accept (L1A) decision on which crossings to save is made.From the Phase-2 Timing and Control Distribution System (TCDS2), through the DAQ BE, a precise 40 MHz -1 -LHC clock reference and fast control information are distributed to the FE Application Specific Integrated Circuits (ASIC).From the Run Control and Configuration system, the DAQ BE also provides the entry point for slow control (viz., configuration, control, and monitoring) of all the FE ASICs.For every L1A, the DAQ BE acquires the full-granularity data from the FE, and sends it to the Central DAQ system for further selection and analysis at the High-Level Trigger (HLT).This occurs at average rates up to 750 kHz, hence the DAQ data from 98% of BXs on average is discarded by the L1T [3].For both the DAQ and the TPG system, there are Low Power Gigabit Transceiver (lpGBT) and Versatile Transceiver Plus (VTRX+) [4,5] fibre optical (F/O) links running at 10 Gbit s −1 between the FE and the BE.The DAQ and TPG systems will each have around 5000 F/O links per endcap.The projected usable rates of DAQ and TPG data coming from the HGCAL FE are 9 Tbit s −1 and 45 Tbit s −1 per endcap, respectively [2].

Conceptual design of a vertical HGCAL electronics test system
A vertical slice of the HGCAL Electronics System should be a system on a much smaller scale than the final one, i.e. with data volumes being approximately 10 000 times smaller.However, starting with the FE sensors and going all the way to permanent storage, such a system would ideally have all the same final devices and their interfaces on its trigger, readout, and control paths as those in the final detector.The main goals are to test various kinds of data exchanges between the FE and the BE, specifically the timing distribution, slow control and fast control, and acquisition of trigger and event data, across all the relevant interfaces.Moreover, it is important to achieve these points in a realistic environment such as a beam test, where such a system would be used to acquire data from particle interactions.Success in these activities would allow the project to move towards horizontal system scaling, where vertical systems would be put together in parallel, building up to the final detector.
Conceptually, the test system brought together at CERN is illustrated in figure 2 and consists of: FE hardware with real ASICs, a BE board running custom firmware and software, a DAQ and TCDS2 Hub (DTH) board, and a DAQ PC to orchestrate run control and configuration as well as writing the acquired data to disk.The FE consists of silicon modules that include HGCAL Readout Chips (HGCROC), and TPG and DAQ data concentrator ASICs (ECON-T and ECON-D).The modules are connected to an "engine" PCB that hosts the lpGBT and the VTRX+ ASICs.The FE and the BE are separated by an 80 m -2 -long F/O cable, to be close to the realistic latency of the final system.The BE uses scaled-down HGCAL-specific systems (TPG and DAQ), but also several parts which are not HGCAL-specific, such as the timing source, a local L1A generator and a TCDS2 emulator.The TPG BE unpacks the data received from the ECON-T ASIC and sends it towards two destinations.Firstly, they are sent to the local L1A generator which is an HGCAL self-trigger that generates a L1A signal if a FE trigger cell is above an energy threshold.Secondly, the same data are sent through the DTH readout board [6] and subsequently to the DAQ PC for permanent storage so it can be studied offline and compared to the DAQ data.To measure the self-trigger efficiency and purity, it is also possible to trigger on signals coming from the external scintillator trigger setup.The external trigger signals are sampled at 1.28 Gbit s −1 by an lpGBT, giving a measurement of the particle time with respect to the 40 MHz clock with sub-ns precision, an important feature for data analysis.The generated L1A signal is sent from the TCDS2 emulator to the DAQ BE in the form of the fast control commands.The TCDS2 emulator also generates other fast commands, responsible for various levels of FE to BE synchronisation, data buffer clearing, etc.The DAQ BE distributes both fast control and slow control information to the FE, interfacing with the TCDS2 emulator and the Run Control and Configuration system, respectively.Through the fast control stream, the DAQ BE also distributes the 40 MHz timing reference from an external clock generator board source.The interface of the BE to the DTH board are two 25 Gbit s −1 Slinks (one for TPG, one for DAQ), while the interface between the DTH and the DAQ PC is a single 100 Gbit s −1 TCP/IP link.The interface between the Run Control and Configuration system and the BE is a standard 1 Gbit s −1 Ethernet link.While the FE TPG data is sent to the BE at a frequency of 40 MHz (as in the final system), the DAQ rate is determined by the characteristics of particle spills in the beam test area at CERN where the system was planned to be tested, namely the beam intensity and spill length.

Operating an implemented vertical system in a beam test
The HGCAL collaboration successfully implemented and operated a vertical HGCAL electronics test system (figure 3) in a Super Proton Synchrotron (SPS) beam test.Comparing the implemented system to the conceptual design (figure 2) there are a few details worth considering.The FE housed two back-to-back silicon hexagonal modules with 192 sensors cells each (where each cell has an area of 1.18 cm 2 ) and their corresponding readout PCBs and ASICs (Silicon "Train" v3).The two modules were used to cross-check the results from each other, as well as being a good exercise in synchronising the readout from two FEs, something that will also be needed in the final system.The BE algorithms were running on a Serenity v1.1 [7] board and the high-speed readout was done using the DTH p1-v2 board.The data from the DAQ PC was brought to the CMS Software (CMSSW) offline Data Quality Monitoring (DQM) system where various plots were automatically generated for every data acquisition run.For example, the CMSSW DQM generated the hexagonal plot shown on the bottom right of figure 3 that represents the distribution of the standard deviation of ADC counts for different DAQ channels.Similarly (however, using a different tool), the hexagonal plot shown on the upper right of figure 3 represents the distribution of the mean ADC counts for different trigger cells.Both hexagonal plots are from a run where an electron beam was used.
Regarding timing distribution, the BE could reliably distribute the clock to the FE so that both the BE algorithms and the FE ASICs would synchronise to the same 40 MHz reference.A measurement of the standard deviation of the clock phase noise jitter between the clock source in the BE and the -3 - recovered clock at the input of an HGCROC in the FE was made, resulting in 8.5 ps.The impact of the HGCROC clock-recovery circuits is less than 2 ps additive jitter.In summary, the clock distribution system is well within the 15 ps specification.The DAQ BE was able to reliably read from and write to any ASIC register in the FE.The former was tested by performing O(1 M) write and read slow control transactions with the FE ASICs.The system was able to reliably send a variety of fast commands to the FE and confirm their proper reception by checking the ASIC responses were as expected.The system was able to bring both TPG and DAQ data through the BE readout and processing chain where they were captured, unpacked and combined, before being written to disk.Depending on the particle beam type and energy, and depending on the system configuration (including the algorithm choices in the ECON ASICs) the rates of events during particle spills was as high as O(100 kHz), generating up to O(10 Gbit s −1 ) worth of data rates.The system was able to reliably acquire up to O(100 M) events in overnight runs without human intervention.

Most notable beam test results
Preliminary studies of the acquired beam test data show two important results.Firstly, the readout electronics are synchronous to the 40 MHz BE clock reference.In the absence of a possibility in this beam test area to synchronise to the SPS clock, the 40 MHz BE clock reference was derived from a commercial "off-the-shelf" clock generator (i.e., independent of the SPS clock).Hence, the SPS beam particles could arrive at any random time with respect to the used 40 MHz clock reference.Consequently, the particle signal is maximum when the particles are in coincidence with the sampling clock.The experiment was therefore able to reconstruct the preamplifier signal shape by measuring the particle time using the external trigger signals.The analogue pulse size from the deposited charge in the silicon cells depends on whether there were beam particles passing through the cell or not (figure 4, Left).For cells with no particles, a base ADC count level is read out (horizontal yellow distribution in the plot).For cells hit by particles, the pulse shape of the deposited charge emerges (the non-horizontal yellow distribution in the plot).The peak of the pulse approximates the so-called "MIP peak", as high -4 -as approximately 20 ADC counts above the no-signal level, as expected for the pion beam used for this data.The width of the time window is 25 ns, which corresponds to one 40 MHz clock period.
The second result concerns the use of the ECON-D ASIC in two distinct modes of operation: passthrough (figure 4, Middle) and zero-suppression (figure 4, Right).Both plots show how many times the signal recorded in hexaboard channels were passing the threshold.When the ECON-D is in pass-through mode, signals coming from all the channels are observed as expected.When the ECON-D is in zero-suppression mode (i.e.filtering out the channels with ADC counts below a programmable threshold) only signals from a subset of channels, which are exposed to the beam, are retained.

Conclusions and outlook
The components (i.e.PCBs and ASICs) of this system are products that came from different parts of the world and had only been tested in their home institutes standalone.Prior experience of assembling this vertically-integrated setup and getting all these devices to work together was very limited or nonexistent.Nevertheless, most of the interfaces have now been tested and the full chain of timing, slow control, fast control and data acquisition was shown to work in a realistic environment of a beam test.The ability to acquire real physics data using this system is practical evidence that such an architecture of the vertical detector readout and control chain is feasible.However, the HGCAL Collaboration is yet to carefully examine the acquired data in order to evaluate the system performance on several levels.With that in mind, studies must be done using the data taken to quantify, characterise, understand and (if possible) remove the possibility of rare scenarios in which the different parts of the system lose synchronisation, there is packet loss or packet corruption in the readout chain, etc.In other words, the vertical test system needs to be understood in detail as the project moves towards horizontal scaling.

Figure 1 .
Figure 1.A simplified block diagram of the HGCAL Electronics system and an illustration of the multiple interfaces it has with global CMS systems, i.e. not specific to HGCAL [2].The given link counts are per endcap.The acronyms used are explained in the text.

Figure 2 .
Figure 2. The conceptual block diagram of the Vertical Electronics Test system architecture.From left to right, the HGCAL FE and BE electronics are shown and on the far right are the storage and central services provided by the DTH board.The acronyms used are explained in the text.

Figure 3 .
Figure 3. Block diagram of the implemented system used in the beam test with its most notable interfaces and example hexagonal plots produced after analysing the acquired data of an electron beam run.

Figure 4 .
Figure 4.The ADC count as a function of external trigger time showing the analogue pulse shape from the deposited charge in the silicon sensor channels (Left).The units on the x-axis are not ns, but 25 32 ns = 0.78 ns.That is why there are 157 − 125 = 32 bins for the time window.Distribution of the number of times signals recorded in HGCROC channels were retained by the ECON-D ASIC when programmed in pass-through (Middle) and zero-suppression (Right) mode.