The data acquisition system for the PANDA Micro-Vertex Detector

The PANDA (antiProton ANnihilation at DArmstadt) experiment will study the strong interaction in annihilation reactions between an antiproton beam and a stationary cluster jet target. The PANDA detector will be composed of several sub-detectors designed for tracking, particle identification and calorimetry. The Micro-Vertex Detector (MVD) is the innermost part of the tracking system surrounding the interaction region, which is designed for precise vertex and tracking detection. It consists of silicon pixel and double-sided microstrip detectors. For the readout of the microstrip sensors an ASIC called ToASt (Torino Asic for Strip readout) is being developed in 0.11 μm CMOS technology at INFN Turin. The ASIC takes advantage of both Time-over-Threshold and Time-of-Arrival methods to accurately measure the event's energy and timestamp. To sustain the acquisition of the microstrip sensors a MDC (Module Data Concentrator) ASIC is under development at KIT. Up to eight ToASt front-ends' data streams are multiplexed, read out and processed by one MDC. The data of several MDCs are collected and processed by the off-detector readout card MMB (MVD Multiplexer Board), also under development at KIT. The processed data is then transferred via 100 GbE optical links to the computing nodes. The ToASt chips have been integrated with the FPGA implementation of the MDC to form the first fully functional detector module. Beam tests have been performed at the COSY facility in Jülich. This paper focuses on the design of MDC ASIC and MMB board, the integration with the ToASt and presents preliminary beam test results.

Abstract: The PANDA (antiProton ANnihilation at DArmstadt) experiment will study the strong interaction in annihilation reactions between an antiproton beam and a stationary cluster jet target.The PANDA detector will be composed of several sub-detectors designed for tracking, particle identification and calorimetry.The Micro-Vertex Detector (MVD) is the innermost part of the tracking system surrounding the interaction region, which is designed for precise vertex and tracking detection.It consists of silicon pixel and double-sided microstrip detectors.For the readout of the microstrip sensors an ASIC called ToASt (Torino Asic for Strip readout) is being developed in 0.11 μm CMOS technology

Introduction
The PANDA experiment, which stands for antiProton ANnihilation at DArmstadt, will be installed at the Facility for Antiproton and Ion Research (FAIR) at GSI in Darmstadt.This experiment aims to explore fundamental questions of hadron physics through proton-antiproton annihilation.The Micro-Vertex Detector (MVD), shown in figure 1, is the innermost detector of PANDA.The main tasks are to provide high vertex resolution for primary interaction and secondary decay vertex, improve the momentum resolution and provide additional input for particle identification.The MVD consists of four detection layers in the barrel region and eight disks in the forward spectrometer region.To provide high-granularity and vertex information combined with a high multiplicity environment, the two innermost layers (barrel) and 6 disks (forward) comprise of more than 10 million silicon pixels, while the outer tracker consists of 2 layers (barrel) and 2 disks (forward) of more than 200 k double-side microstrip channels.These modules, shown in figure 1, are complex hybrid structures that integrate sensor, front-end readout chips and a Module Data Concentrator (MDC) with a cooling pipe on a lightweight carbon fiber support.The DAQ system of PANDA is based on trigger-less acquisition, where the sub-detectors are free-running and synchronized by the SODANET time distribution system.Using zero-suppression, only physically relevant signals are transmitted to the computing node for high-level processing.

Data acquisition architecture of the PANDA-MVD
The DAQ concept of PANDA requires that every subdetector is able to detect hits without an external trigger.Thus, the front-end electronics have to be able to distinguish physical events from noise and send digital hit information along with a precise timestamp to the event builder.Due to the high luminosity, the detector must be able to cope with high interaction rates of up to 20 million antiproton-proton annihilations per second.
To meet these requirements, a new acquisition scheme has been developed.The architecture employs state-of-the-art technologies to optimize performance while reducing the required hardware resources.The data received by strip detector modules, processed by ToASt [1] front-ends, are grouped and formatted by the MDC ASIC.Each microstrip detector module will be equipped with two MDCs, one for each sensor polarity.As shown in figure 2, the MDC is connected to the off-detector electronics, the MVD Multiplexer Board (MMB), by optical low power GigaBit Transceiver (lpGBT) [2] link.The MDC receives commands, configurations, slow-control, and clock signals via the lpGBT in downstream direction and sends the detector data via lpGBT in upstream direction through the e-link channel [3].The lpGBT links employ the radiation tolerant electro-optical transceivers Versatile Link PLUS (VTRX+) developed at CERN [4].The MMB is connected to the Synchronization Of Data Acquisition (SODANET) which provides information on the beam structure and global timing, necessary for accurate timestamping of the detector data.

Development of the Module Data Concentrator
The module data concentrator serves as a link between front-end chips and DAQ system.An overview of the internal architecture of the MDC is shown in figure 3. The MDC acts as a local digital controller on the detector module.It decodes and buffers the data frames from the front-end chips and performs a time-ordering.This is necessary as the data frames are not aligned according to the event time.The MDC provides interfaces for slow-control and configuration, it also broadcasts clock and commands to all front-end chips on the detector module.Additionally, depending on the detector occupancy, the ToASt chips can be configured to operate with one or two high-speed serial data links.
-2 - To accommodate the requirement of different numbers of ToASts combined with a programmable number of serial links, the readout channels of the MDC, shown in figure 3, have been developed with a dynamic negotiation of the number of serial links.If the serial data link of the ToASt is active, the readout channel will de-serialize the data stream by a Serializer/Deserializer (SerDes) logic shown in figure 3, align the MSB/LSB of the parallel data according to a defined training pattern and LOCK the channel.If the channel is locked, it will automatically be included in the readout, otherwise, it will remain in low-power mode to reduce the MDC's power consumption.All locked readout channels are managed by the main readout Finite State Machine (FSM).The FSM acts as an intelligent multiplexing logic that balances the data occupancy in the FIFO channels.The readout data of the detector module are temporarily stored in a local FIFO before being transmitted to the DAQ system via the e-link interface.
The proper control and configuration of the front-end chips are crucial for the optimal performance of the detector.Single-event upsets (SEUs) on the front-end logic and memories, as well as bit errors that may occur during the transmission of the configuration sequences, pose a risk of malfunction.To mitigate the errors, the MDC contains a dedicated configuration logic unit that is able to configure one or more ToASts in parallel, read back the configuration and validate it.If necessary, the logic automatically re-programs the ToASt chips.

Development of the off-detector readout electronics
The MVD off-detector electronics consist of the MMB card fully compliant with the Advanced Mezzanine Card (AMC) standard hosted on a full-size Advanced Telecommunications Computing Architecture (ATCA) carrier [5].This form factor makes the MMB also compatible with the DAQ of other sub-detectors, which are planning to use the Micro Telecommunications Computing Architecture (μTCA) standard.On the MMB, the incoming data rate will be reduced and only physically relevant signals will be transmitted to the burst-building network.The Zynq Ultrascale+ Multiprocessor System on Chips (MPSoCs) located on the MMB cards host multiple ARM processors and a powerful programmable logic layer.The latter allows the implementation of algorithms based on AI for data processing, such as feature extraction and track finding, as well as logic for the dynamic reconfiguration of detectors.The processor part gives the possibility of implementing an online data quality monitor.The MMB architecture is shown in figure 4 on the left, on the right the 3D render of the ATCA carrier [5] is shown.The MMB will be equipped with six 12-lane FireFly modules for the interface -3 -  [5] (right).The architecture of the carrier can be found in [6].Reproduced with permission from [6].
from/to the lpGBT, connected by GTH/GTY transceivers.Each link will be able to operate up to 16.3 Gb/s with a total of up to 36 full-duplex optical links, resulting in a total data rate exceeding 580 Gb/s.Up to two 100 GbE optical links will be utilized for the uplink connection to the burst building network.It is expected that the data rate after the data processing will be reduced to less than 100 GbE, thus presenting the opportunity to utilize the second uplink as a spare connection to the building network.Moreover, the MMB provides several standard interfaces, such as PCIe (Gen2), SATA, and GbE (TCIP/IP), which makes it a versatile readout card for a wide range of scientific applications.
The ATCA carrier [5] hosts four MMB cards.The carrier board is designed using the OpenIPMC DIMM [7], which uses the JEDEC MO-244 low profile form factor employed in several ATCA cards at CERN [8][9][10].The OpenIPMC acts as board controller [11] according to the PICMG specifications [12].The carrier has several point-to-point connections between the Zone 2 connector and the AMCs and a single multi-gigabit transceiver link to each AMC from the onboard AMD Xilinx KRIA System on Module.The carrier also distributes the clock network to all AMC cards by means of a jitter cleaner (SI5395) in the zero-delay configuration for precise timestamping of events.Finally, it provides Ethernet connectivity to all end-points using a 10-port Ethernet switch (VSC7420).

Prototype of the PANDA-MVD DAQ system and test beam results
To test and validate the on-detector functionality of MDC and multiple ToASt/sensor modules, an FPGA version of the MDC has been developed.The prototype of the readout system, based on a HighFlex card [13], utilizes the powerful data transfer with direct FPGA-GPU communication, developed at KIT.The setup enables full control, testing and configuration of the microstrip detectors.
For a first in-beam test of the complete detector chain a setup with two single-sided silicon strip sensors, each connected with 64 strip channels to a ToASt front-end, was operated at the COSY proton accelerator facility.The beam test setup is shown in figure 5 (left).The sensors were fully depleted, one with the n-side strips and one with the p-side strips.The ToASt boards were attached to a frame in such manner that the sensors are about 2 cm apart, but the strips are orthogonal to one another and the active areas overlap.The test sensors originate from the same wafers as the final MVD sensors and therefore have comparable properties, but are smaller in size.The entire system -4 - was mounted in such a way that the sensors could be moved in or out of the beam center and be rotated to realise different angles relative to the beam plane.
The experiments at COSY were carried out with a 2740 MeV proton beam at an intensity of 10 10 particles per spill.Since the experiment was a secondary user of the beam, it can be assumed that impact products also reached the sensor in addition to the protons.The particle rate should therefore be significantly lower.Figure 5 (right) shows the energy loss of particles in one of the two ToASts, measured by ToT technique, for angles 0, 20, 60 and 240 degrees.The angle 0 degrees corresponds to the perpendicular impact of the beam on the active sensor surface.

Conclusion and outlook
The new architecture of the MDC for the microstrip detector of PANDA has been developed and integrated as fully functional prototype and tested at COSY.For the first time the sensor modules and the complete readout chain has been tested.The results demonstrate that the concept matches the specifications.The prototype system will be used for in-depth characterization of the detector performance.Meanwhile, the implementation of the final electronics will be completed.

Figure 2 .
Figure 2. Data acquisition architecture of the PANDA-MVD detector.

Figure 3 .
Figure 3. MDC internal architecture and its logic blocks.

Figure 4 .
Figure 4. Architecture of the MMB card (left) and 3D render of the ATCA carrier[5] (right).The architecture of the carrier can be found in[6].Reproduced with permission from[6].

Figure 5 .
Figure 5. Beam test setup (left) and energy loss in ToASt02 at different angles (right).