Test bench of a 100 Gbps radiation hardened link for future particle accelerators

Pioneering physics experiments require increasingly faster data transfers and high-throughput electronics, which drives the research towards a new class of serialisers and optical links. In this framework, the DART28, a 100 Gbps radiation tolerant serialiser and driver, has been designed in 28 nm CMOS technology, submitted in April and delivered in August 2023. The development has been coupled with an FPGA based emulation, which provided an early assessment of its behaviour, a scalable system-level demonstrator and an effective evaluation tool for compatible commercial solutions. The challenges faced in this research and the architecture of both the hardware setup and the firmware will be described.


Introduction
The data communication between the detectors (front-end) and the readout infrastructure (back-end) represents one of the most significant challenges in the engineering of modern high-energy physics accelerators and related experimental apparata [1][2][3].As instruments with higher resolution and sampling rates are demanded to boost the statistics of measurements, the density of data generation is dramatically increased near the beam interaction point.Thus, the transmission of results requires the deployment of solutions featuring: data throughput above 10 Gbps, a low power consumption and material budget, a reliable operation in a highly radiation environment.
Since the launch of the LHC (Large Hadron Collider), the Electronic Systems for Experiments Group at CERN has released different options for radiation hardened data communication, the state of the art of which can operate up to 10.24 Gbps [4,5].Currently, in the framework of the CERN EP R&D programme [6,7], a new concept has been proposed aggregating four 25. 6 Gbps lanes over a single-mode fibre in wavelength division multiplexing, which paves the way for a new class of 100 Gbps links.This requires application specific integrated circuits (ASICs) resistant to total-ionising doses (TID) up to 1 Grad and mixed-field fluxes of ∼ 10 16 MeV n/cm 2 [8][9][10].Particularly, the new platform embeds a 28 nm CMOS based ASIC and a paired silicon-photonic modulator [11,12].The development of such components involves: a complex design flow, a detailed verification of the features and the post-production testing of prototypes.A demonstrator ASIC for radiation tolerant transmitter in 28 nm technology, the DART28, has been engineered as the proof of concept, including the fast serialiser [13] and the output driver [14].
To effectively verify the functionalities of the DART28, a detailed emulation activity, based on modern AMD Ultrascale Plus series FPGAs [15], was undertaken prior to submission to the foundry.It has brought many benefits for the analysis of the system-level characteristics, providing a realistic reproduction of the device's behaviour.The architecture of both the developed firmware and the hardware setup will be described in section 2.Then, the test-bench objectives and the back-end logic are presented in section 3. Finally, a dedicated printed circuit board has been designed for the DART28 testing stage.Its features have been carefully tailored to cope with the targeted tests and they are reported in section 4, highlighting one of the key challenges faced during the production.

JINST 19 C02072 2 Emulation of the demonstrator ASIC
Here, a device emulation is intended as a replica of its electrical behaviour from the perspective of the input/output (I/O) interfaces.Ideally, the device and its emulator, considered as black boxes, should be indistinguishable from the point of view of the external system.This is generally obtained by exploiting an already available technology that resembles the device one up to a desired depth.The target of the emulating framework is a quicker and a more effective adaptation to design changes, rather than an optimisation of the specific application, which is reserved for actual prototypes.
Field-programmable gate arrays (FPGAs) are well suited for emulating digital oriented ASICs.In fact, they share many stages of the design flow, the register-transfer level (RTL) code and part of the verification methods.Furthermore, the FPGA reconfigurability allows iterating many different code revisions and exploiting additional tools for a real-time diagnosis of the logic.Unavoidably, such a solution provides an approximated model of the original I/O behaviour, since the micro-electronic implementations can be quite different.However, a realistic reproduction can be achieved at the functional level.Finally, some aspects like the transmitter driver performance or the power integrity cannot be emulated.Hence, they must be analysed with different methodologies or characterised on the actual device.To set up an FPGA based emulator, the following steps are performed: • Technology agnostic sources (typically written in Verilog language) are ported to the FPGA project and the I/O ports constrained in agreement to the targeted hardware platform; • Specific foundry or custom primitives are replaced with FPGA ones, for example: the high-speed transmitters, PLLs, I/O buffers; • Additional diagnostic cores can be added to monitor the logic behaviour, to control specific FPGA settings, or to mitigate features no longer available after the replacement of primitives.In figure 1, the data stream flows from left to right in the modules on top without involving external inputs, since the DART28 is a demonstration device.Instead, an embedded generator, operating at 40 MHz, can be configured to produce different types of data, including: constant patterns, emulated clocks and pseudo-random binary sequences (PRBS).The stream is subsequently processed by a scrambler, a forward error correction logic (FEC) and an interleaver.These are essential to establish a reliable transmission in the radiation environment and to facilitate the clock and data recovery on the back-end counterpart.The encoded data is processed by two stages of serialisers, before reaching -2 -the output driver.The circuit is driven by an on-chip PLL and clock distribution network, which is shown at the bottom.Clocks span from 40 MHz (input reference and core logic domain) up to 12.8 GHz (half-rate serialiser clock).Finally, the device is configured by an extensive register space that can be accessed by means of an I 2 C bus.

Data Generator
Components such as the data path and the configuration space are implemented using targetagnostic code, so they have been ported to the FPGA project unmodified.Instead, the serialiser is replaced by a specific core provided by the vendor.For high transmission rates, the parallel bus adopted in the fabric logic has a width of 128 bits.So, the slow serialiser is converted to a 640 → 128 implementation, operating between 40 MHz and 200 MHz.Consequently, the clock infrastructure has been adapted to cope with the new frequencies by means of integrated PLLs.Code parametrisation enables switching between the ASIC and the FPGA implementation.Thus, the same repository can be shared to keep the sources of both targets up-to-date.
The emulator has been flashed into an AMD VCU129 evaluation board [16], featuring a modern Virtex UltraScale Plus FPGA, which has been linked to a twin card used for the back-end counterpart (or to a VCU118 [17] as alternative).The high-speed connection between the emulator and the back-end is achieved by using either a QSFP compatible direct attach copper cable or optical CWDM4 modules.Eventually, a similar setup will be exploited for testing the real prototype.
A characterisation of the DART28 is mandatory to assess its performance and radiation hardness, including detailed testing of the fast serialisers, the output drivers and the PLLs.However, this does not detract from the valuable benefits that the emulation activity has brought to the project.In fact, the installation of the presented system enabled an effective design of the test-bench, including the selection of suitable equipment, cabling and the planning of the overall setup assembly.Secondly, it allowed testing the RTL functionalities, assisting in the correction of errors present in the ASIC Verilog code and the optimisation.In particular, running the design in real-time can be a useful correctness test for the link implementation, e.g. for evaluating the realistic FEC capability with respect to the theoretical one.Furthermore, this approach is efficient to check the start-up behaviour of the logic, the time-out conditions (for instance in the I 2 C slave module) and watchdogs, which have to be tuned accordingly to the integrating environment.In fact, these aspects are difficult to exhaustively study by simulations, which are typically time-consuming.Moreover, the use of the emulator with laboratory equipment can reveal non-compliance to standards quicker than developing a comprehensive feature coverage during verification.Finally, the emulator can be used to validate testing procedures and exploited in all the subsequent studies not requiring the true ASIC, such as the back-end firmware development or the computer-based characterisation routine.

Test-bench objectives and back-end firmware
A high level of reliability is required for operation in the harsh environmental conditions surrounding the particle beam paths.For this reason, an in-depth testing session of the ASIC is planned to validate functionalities over a variety of use cases and parameters, targeting the following subjects: This encouraged the development of modular back-end firmware aimed at automating procedures and scalability to multiple lanes.In particular, the high-throughput tasks, such as the BER, are performed in FPGA logic.Conversely, the slow data accesses are managed by Python scripts running on a computer, to which the board is connected by network.This has permitted a transparent reuse of many procedures from the original ASIC verification framework.The heart of the back-end firmware consists of the reception and decoding chain (see figure 2), the architecture of which is the mirrored counterpart of that deployed into the device.Each module can be bypassed for testing the specific features and additional logic has been instantiated to perform frame alignment.The output data is supplied to a pattern checker, which determines whether the received frames match the generated ones and reports any errors.The core configuration is obtained by means of registers, which can be accessed by standard memory-mapped interfaces.
In the overall back-end firmware, four independent data receivers are instantiated to handle all the DART28's transmitters.Results of the data matching are timestamped and they can be queried by the computer scripts.In parallel, corrupted frames and their adjacent ones are stored in a buffer for subsequent analysis.This allows recovering the status of the data path, hence identifying where the error (e.g.SEU) occurred.Finally, an I 2 C master is also included to access the device configuration and control the characterisation board, which will be presented in the following section.

Characterisation board
Tests on the DART28 ASIC need an appropriate carrier board (see figure 3), which has been developed in the same context.It provides full accessibility to the device-under-test (DUT) features and adopts a low loss Panasonic Megtron 6 substrate [18], in order to minimise the influence of the routing during the characterisation.The fan-out of the lanes is carried out using coplanar micro-strips of 50 Ω, reaching 40 GHz-rated K connectors for the end launch.In addition, the DUT is equipped with a set of test probes that can be configured to output a variety of internal signals.These propagate on the board over 100 Ω differential micro-stripes, up to high-quality SMA connectors.
-4 - The board embeds a low noise power supply unit, which allows for operating the DUT at different voltages and monitoring supply current and power dissipation with high accuracy.Furthermore, temperature sensors are also installed.All the embedded features share the same I 2 C bus, which is operated by the back-end board.
Various efforts have been undertaken toward the minimisation of the DUT wire-bond length, in order to optimise the signal integrity and the power delivery.In particular, a small cavity has been drilled into the substrate for accommodating the DART28 and levelling its pads with the board tracks.A metallic plate has been designed to support the ASIC from the back of the board and to provide the required heat dissipation.Furthermore, a high density routing, with minimum tracks and gaps of 3 mils (76.2 μm), allowed the fan-out to be deployed as close as possible to the device.Inspections of the first prototype showed that the technological process is capable of such a demanding resolution.However, it is prone to over-etching effects, which cause undesirable width shrink of the copper traces.This threatens the success of the wire-bonding procedure, to the point of making it impossible in the worst cases.Solutions to tackle this problem are under discussion, but manufacturers suggest that it can be mitigated by reducing the thickness of the board metallisation from 35 μm to 17 μm.A new production has confirmed such an improvement.
Finally, during the layout design, 3D models of the testing equipment have been compared to optimise the component placement and the board geometry.Hence, planning of the assembly steps and the test-bench connections has been possible since the early development stages.

Conclusions
The development of radiation hardened systems for data transmission at 100 Gbps would not be possible without effective tools for design verification and testing of prototypes.The emulation of the DART28 ASIC on a FPGA based platform proved to benefit the identification and resolution of errors affecting the source code, its optimisation as well as the engineering of the test-bench.Furthermore, it has accelerated the development and validation of the back-end readout firmware, which represents the -5 -core of the testing framework and one of the key deployable products for future project commissioning.Finally, exhaustive planning of the test objectives enabled the design and production of a carrier board, which is already supporting the DART28 in all the characterisation phases.It is expected that the proposed methodologies can be effectively generalised to aid the design of a variety of digital ASICs.

Figure 1 .
Figure 1.Simplified architectural diagram of one of the DART28 data lanes showing the original modules ported to the FPGA emulator and highlighting with symbols "⟳" the primitives subsequently replaced.

3 -
DC electrical tests: current and power consumption; Signal integrity of links: eye diagram and phase noise analysis; System-level validation: device configuration, bit error ratio (BER) test and data encoding; Technology validation and circuit testing: flip-flop primitives, PLL, high-speed transmitter; -Operational corners: different voltages and temperatures; Radiation qualification: TID and single event upset (SEU).

Figure 2 .
Figure 2. Architecture of the back-end receiver and decoder core.