Development of a high data-throughput ADC board for the PROMETEO portable test-bench for the upgraded front-end electronics of the ATLAS TileCal

The Large Hadron Collider (LHC) is preparing for a major Phase-II upgrade scheduled for 2022 [1]. The upgrade will require a complete redesign of both on- and off-detector electronics systems in the ATLAS Tile hadron Calorimeter (TileCal) [2]. The PROMETEO (A Portable ReadOut ModulE for Tilecal ElectrOnics) stand-alone test-bench system is currently in development and will be used for the certification and quality checks of the new front- end electronics. The Prometeo is designed to read in digitized samples from 12 channels simultaneously at the bunch crossing frequency while accessing quality of information in realtime. The main board used for the design is a Xilinx VC707 evaluation board with a dual QSFP+ FMC (FPGA Mezzanine Card) module for read-out and control of the front-end electronics. All other functions are provided by a HV board, LED board and a 16 channel ADC daughter board. The paper relates to the development and testing of the ADC board that will be used in the new Prometeo system.


The ATLAS Tile Calorimeter
The ATLAS (A Toroidal LHC ApparatuS) experiment is a general purpose particle detector at the LHC at CERN. The experiment is designed to record proton-proton collisions occurring at a bunch crossing frequency of 40 MHz. At such high frequencies large amounts of raw data are generated leading to data flows in the Tbps range. The ATLAS experiment comprises of several sub-detector systems each aimed at recording different aspects of a collision.
The TileCal is the central hadronic calorimeter sub-detector in ATLAS (Figure 1a.) [3]. TileCal comprises of four barrel sections, two central barrels (LBA and LBC) and two extended barrels (EBA and EBC). Each section is divided into 64 azimuthal slices with each slice being further segmented by a lattice of alternating steel plates and plastic scintillating tiles ( Figure  1b). When particles travel trough TileCal they collide with steel plates causing them to break up into showers of charged and neutral particles. As these particles travel through the scintillating tiles small amounts of energy is deposited which generates photons. The photons are picked up by wavelength shifting optical fibres which carry the signal to the front-end electronics found within the outermost part of the sub-detector. Photomultiplier tubes (PMTs) are used covert these faint light pulses into analog electrical signals. These signals are then amplified, digitised and stored in temporary pipeline buffers while they wait for trigger selection. Only samples of these events that are selected by the L1 trigger are sent to the back-end electronics for further processing. After three levels of trigger selection the data flow is reduced to a few 100 Mbps which is stored for off-line analysis.

Phase II upgrade
The Phase II upgrade will increase the design luminosity of the LHC by a factor of 5 to 7.
[1] The ATLAS experiment will need to undergo some substantial changes if it is to meet these demands. A complete redesign the detector electronics will be needed in order to meet increased radiation tolerances and higher data processing requirements [4]. The current read-out electronics uses pipeline memories in the font-end to store digitised samples while they wait for trigger selection. After the upgrade the read-out system will transmit all data directly to the back-end using gigabyte optical links during every bunch crossing. This will provide digitally calibrated information with enhanced precision and granularity to the first level trigger in order to improve both trigger latency and resolution.
The current read-out hardware is located in 2 m long "superdrawers" within each module of the TileCal. [5] The read-out electronics systems in each superdrawer are daisy-chained together forcing them to share the same data connection to the back-end and same power supply. A new front-end architecture has been proposed that breaks each superdrawer down into 4 "minidrawers". This architecture will merged several of the old boards onto a more compact 3 board system that will include massive levels of redundancy to increase the reliability of the system. Each minidrawer will be completely independent of the others, having its own powers supply and optical links to the back-end.

Prometeo
The new front-end electronics being developed for the Phase II upgrade will need a compatible test-bench system. The PROMETEO (Portable ReadOut ModulE for Tilecal ElectrOnics) is being developed to replace the existing MobiDICK [6] test-bench which has been in operation service since 2003. Prometeo (Figure 2a) is a stand-alone test-bench system that will be used for the full certification of the new front-end electronics of the ATLAS TileCal. The TileCal has 256 modules each one will need to be independently serviced and evaluated using the Prometeo portable test-bench. All tests must be completed during LHC shut-downs to make sure the sub-detector is ready for data-taking periods. The Prometeo is high data-throughput system required to process all the information produced by one minidrawer in real-time. A test setup needs to be able to analyse data from 12-PMTs at LHC bunch crossing frequency assessing the quality of the data in real time and diagnose malfunctions [7].

Hardware Design
When designing the system it was important that it was easy to upgrade when components become obsolete but still remain relatively cheap to build [7]. The motherboard used in Prometeo is the Xilinx VC707 evaluation board which contains a Virtex 7 FPGA. The board was chosen for it high IO capabilities and ability to process large amounts data in real-time. Several other boards and modules are able to connect to the board through multiple connections. A high speed QSFP+ module attaches to one of the two FMC connectors. It is used for the communication between the minidrawers and test-bench. The other FMC connector supports a custom ADC board which was developed to digitise the trigger signals coming from the front-end adderboards.
Additionally there is a HV board to provide power to the PMTs, a LED driver board to inject light pulses into the PMTs during tests, a router used to communicate to the users and a power supply unit to power the test-bench. The VC707 is programmed with the IPBUS protocol which is capable of storing trigger data samples which can be retrieved upon a trigger request

ADC daughter Board
The ADC daughter boards (Figure 2b) function is to digitise analog trigger signals coming from the front-end electronics adder-cards [7]. The board communicates with a FPGA located on the VC707 through a high speed FMC connection. The board has two high-performance ADC571 chips each with 8 differential input channels used data sampling. The ADCs have a 12-bit resolution and are capable of sampling data at a rate of 50 Mega Samples a Second (MSPS). In the Proemteo system the FPGA sends a 40 MHz clock to a clock buffer chip on the ADC board which cleans the signal and sends it to both the ADCs. This clock is used by the ADCs as a reference for data sampling. Several Different voltage regulators are used on the board to improve signals integrity and remove signal coupling problems. Indicator LEDs on the board inform the user that all the voltages are at the correct level. Analog data being sent to ADC board is first conditioned and filtered before being passed to the ADCs for digitisation. ADCs can then digitise and serialise the data from each channel and send it to the FPGA. A 240 MHz bit clock is also created by ADCs and sent alongside the data. This clock is used is used for the de-serialisation of the samples. Once the serial data arrives in the FPGA it is de-serialised and stored in temporary buffer in the VC707 RAM. If data is seen to be interesting it will be called from the buffers and further processed otherwise it will be overwritten. Each differential channel coming from the ADCs transports data at 480 Mbps. There is 16 channels in total coming from the ADC board leading to a total data transfer rate of 7680 Mbps.

Firmware
Custom firmware was developed for the FPGA so it can control and manage the data readout of the ADC board. Firmware was described using VHSIC Hardware Description Language (VHDL) using Xilinx ISE design software. The firmware for the Prometeo is highly modular which enables different sections to be tested and used independently. As the same ADC chips were used in both MobiDick and Prometeo, similar algorithms could be used for the data readout. Figure 3: Data signals between an ADC and FPGA Figure 3 shows all the data connections needed between a single ADC and the FPGA. Each one of these signals needs to implemented and simulated within firmware. Initial testing of the ADC board involved configuring the ADCs to output the test pattern 10101010 (Deskew) from each channel. To do this serialised commands are sent from the FPGA through the control lines using a set of strict timing requirements in order to configure the ADC to send the pattern. These pattern tell us if ADCs are receiving our commands, if they are sending data at the correct speeds and if the 8 data lines are synchronised correctly. Figure 4a shows the basic order of events when doing this test.

PCB
The first prototype for the new ADC board was manufactured in 2014 in South Africa. The board went through several tests to see if it performed as expected. Chipscope IP cores were used in the firmware to allow viewing of the data signals between the FPGA and ADC board during operation. This kind of testing allows the developer to confirm that the firmware and the hardware is interacting as designed. The Figure 3b shows actual commands being sent to the ADC board to configure it to send the Deskew patten. The top 8 line show the 4 commands lines of both ADCs while the bottom 6 show the high speed data being sent back.
After testing the two prototype ADCs boards at the WITS High Throughput Electronics Laboratory many design faults could be identified. The circuit design and PCB layout of the board could be modified and improved for the next generation of ADC board. Significant changes included adjusting the power regulators, reducing noise through component and wire shifting

Conclusion
The Prometeo is in the process of being designed for the certification of the front-end electronics for the Phase-II upgrade. The high throughput system will be capable of performing all the testing needed during shut-down periods. The ADC board is still under going further testing before it will ready for use on the Prometeo system. The next generation of the board has be designed is currently being manufactured in South Africa. Once the new board is produced final firmware testing can be completed and the board will be certified for use.