Design of an FPGA-based embedded system for the ATLAS Tile Calorimeter front-end electronics test-bench

The portable test-bench for the certification of the ATLAS tile hadronic calorimeter front-end electronics has been redesigned for the present Long Shutdown (LS1) of LHC, improving its portability and expanding its functionalities. This paper presents a new test-bench based on a Xilinx Virtex-5 FPGA that implements an embedded system using a PowerPC 440 microprocessor hard core and custom IP cores. A light Linux version runs on the PowerPC microprocessor and handles the IP cores which implement the different functionalities needed to perform the desired tests such as TTCvi emulation, G-Link decoding, ADC control and data reception.


Tile Calorimeter
The ATLAS experiment [1] is one of the four main detectors working in the Large Hadron Collider (LHC) at CERN. The ATLAS experiment is composed of several subdetectors and one of these is the Hadronic Tile Calorimeter (TileCal) [2] detector which is shown in figure 1.
The TileCal is a hadronic sampling calorimeter made from steel plates and plastic scintillator tiles which covers the central region of the ATLAS detector. It is composed of four sections in 3 cylindrical parts along the beam direction: one central barrel (Long Barrel A and Long Barrel C) and two extended barrels (Extended Barrel A and Extended Barrel C). Each of the Long and Extended barrels are segmented azimuthally into 64 wedges, so-called modules. These modules contain 23 cells for the long barrels and 16 for the extended barrels, where each one is read out -1 -  using 2 photomultiplier channels. The total number of channels needed to read out the entire calorimeter is about 10,000.
When a charged particle passes through the plastic scintillating tile, light is produced and guided by wavelength shifting fibers to photomultiplier tubes (PMTs). The PMT signals are digitized and stored in the front-end electronics. Then, upon the reception of a Level 1 trigger accept signal (L1A), the data of the selected event is transmitted through optical fibers to the Read Out Driver (ROD) modules in the counting rooms. Figure 2 shows a diagram of the present readout architecture in TileCal.

Front-end electronics
All the electronics required for the readout of the PMT signals are located inside super-drawers at the outermost radius of the TileCal modules. The 3-in-1 cards are directly connected to the PMT and provide shaped PMT signals with two different gains to the digitizer boards, analog trigger outputs to the adder boards and integrated PMT currents for channel calibrations which are read out using the ADC-I board. The digitizers convert the analog signals and store them in pipeline memories, while the adder boards send a tower trigger analog sum to the L1 Calo system. Once the digitizer receives a L1A signal via a Trigger and Timing Control receiver (TTCrx) [3], the interface -2 - board collects the corresponding data from the digitizers and sends it to the back-end electronics using optical fibers.
In addition, the front-end electronics also include two separate CAN bus lines which are used to control and monitor the high voltage applied to the PMTs through the HVmicro board and to read out the ADC-I board. TTC commands are used to configure and control calibration circuitry of 3in-1 cards and digitizers [4]. An extracted super-drawer is shown in figure 3 during a maintenance operation.

Previous MobiDICK system
The previous version of the MobiDICK system (Mobile Drawer Integrity Checking system) [5] was designed to check the integrity of the TileCal super-drawers in situ during the installation and maintenance periods. This test-bench is capable of testing the functionality, configuring and emulating the back-end electronics. Three copies of this MobiDICK version were built between 2003 and 2007.

Hardware
This system is based on a VME system which include a set of different custom and commercial VME cards. All the electronics are contained in an aluminium box which has a width of 50 cm, a depth of 33 cm and a height of 41 cm. The weight of the complete system is about 20 kg. The previous MobiDICK is composed of the following VME modules: • Rio2 VME processor. Works as server and manages all the VME cards in the crate. Runs a Linux operating system.
• ODIN S-LINK. This card receives the digital data coming from the interface board via an SFP optical connector.
• CAN bus interface. This slow path communication is used to configure some parts of the front-end electronics and read back data from the ADC-I card.
-3 - • TTCvi and TTCex modules. Sends the LHC clock to the front-end electronics, as well as L1A signals and TTC configuration commands via an optical fiber.
• CAEN V792. Digitizes and stores the analog signals coming from the adder boards.
• High Voltage and LED driver card. This card generates an appropiate high voltage to feed the PMTs blocks during tests and short electrical pulses to drive a LED which generates a 20 ns wide pulse necessary for the excitation of a LED during the calibration and the checking of the front-end electronics readout chain.

Software
The software is composed of a server which runs on the VME processor under LynxOS and a client which runs on a laptop. The systems communicate using TCP/IP protocol via an Ethernet connection. The server, implemented in C language, performs the tests controlling the hardware upon a request from the client. Once the test is done the server returns the result to the client.
The client is a C++ program, so-called Willy, which runs on a laptop under Linux. This Graphical User Interface (GUI) allows the operator to initiate the tests, visualize and analyze the results through a friendly graphical interface. Willy software also generates and stores a report with the results of performed tests. Figure 4 shows a picture of the previous MobiDICK system.

New MobiDICK system
This new generation of portable test-bench is used for the full certification and quality checking of the super-drawers in the TileCal detector during the current maintenance period [6]. This new version of MobiDICK not only provides the same functionalities as the old system but also enhances mobility and improves the accuracy of tests. Moreover this new version is composed of state-ofthe-art components ensuring the availability of replacements during the next years. Figure 5 shows -4 -

Hardware
A Xilinx ML507 evaluation board [7] with a set of custom and commercial electronics boards implements all the previous system functionalities, removing the necessity of having an entire VME system and, thereby, reducing its weight and size.

Motherboard
The motherboard of the new MobiDICK is a Xilinx ML507 evaluation board. This evaluation board includes a Virtex-5 FPGA with a hard core PowerPC 440, embedded gigabit transceivers and programmable logic. It also includes a 10/100/1000 Ethernet port, a slot for DDR2 memories and a CompactFlash slot used for the configuration of the embedded system. The ML507 is equipped with an SFP connector providing high speed communication.

ADC board
The digizitation of the analog trigger and muon outputs coming from the front-end electronics is performed using a custom ADC board. This board contains two ADS5271 12-bit ADC chips from Texas Instruments which digitize 16 input channels at a sampling rate of 40 MSps. The ADC board sends the digitized data to the ML507 through the expansion ports using 16 LVDS channels at 480 Mbps each.

High Voltage and LED driver boards
The new system includes two separated custom boards redesigned from the old High Voltage (HV) and LED VME card: a high voltage power supply to feed the PMTs during tests and a LED driver board which generates the 20 ns wide pulse for the excitation of the LED for the tests.

CAN bus
Two commercial CAN bus dongles are used to control the HVmicro board which sets high voltage value applied to feed the PMTs, and the ADC-I board which provides integration data on the PMT currents. These CAN bus dongles are controlled using two serial ports of the ML507 board.

Mechanics
The new system is about half the volume of the old one and about 20% of the weight. It is placed in a custom aluminum box which provides a sturdy tool suitable for its use during the maintenance campaigns. This box has a width of 40 cm, a depth of 35 cm and a height of 20 cm. The total weight of the assembled system in the box is around 4 kg. It also includes a forced air cooling system reducing the possibibility of thermal failures of the electronics. Figure 6 shows a photograph of the new MobiDICK system.

Embedded system
The core of the embedded system is a hard core PowerPC 440 processor integrated in the Virtex-5 FPGA. The PowerPC processor is connected to different IP cores, both custom and from Xilinx, that are implemented using the logic resources of the Virtex-5. These IP cores are connected to the PowerPC processor as slave devices via a 32-bit Processor Local Bus (PLB). The different VME cards of the previous version are now replaced by firmware implemented inside the ML507, such as a TTCvi card and an ODIN card, obtaining an outstanding improvement in size, weight and portability. The embedded system has been developed using Xilinx ISE and EDK version 13.1. Figure 7 shows a diagram of the complete embedded system.

G-Link receiver IP core
The G-Link receiver IP core emulates the Agilent HDMP-1034 chip receiving and decoding the G-Link data stream from the interface board through the SFP module at a data rate of 800 Mbps. There are two operation modes: normal mode, where the received data is stored in 32 kb FIFO memories for later readout; and CRC mode where a CRC module implemented in FPGA logic -6 - detects data integrity errors in the received data and errors in the format on-the-fly, without storing the data into the FIFO memories.

TTCvi emulator IP core
The TTCvi emulator IP core emulates most of the functionalities of the existing TTCvi module. This IP core handles a VHDL module which Time Division Multiplexes (TDM) L1A triggers and commands using BiPhase Mark (BPM) encoding and sends the encoded data through the SFP module to the front-end electronics for configuration and control. As the original TTCvi module, this IP core can send short-format asynchronous and synchronous broadcast commands, as well as long-format asynchronous individually-addressed (or broadcast) commands.
The synchronous operation of the commands and L1A triggers is implemented in a 8 position FIFO memory which stores commands/L1A signals and the corresponding Bunch Crossing (BC) for their execution.
Moreover, this IP core has three operating modes to generate the L1A triggers: single trigger, synchronous trigger (using the FIFO memory) and burst mode, allowing different fixed L1A trigger rates: 1 Hz, 100 Hz, 1 kHz, 5 kHz, 10 kHz, 25 kHz, 50 kHz and 100 kHz.

Orbit clock IP core
The Orbit IP Core is a key element for the synchronization between commands and L1A signals with their corresponding BC and for synchronizing them with the rest of the embedded system. To achieve this synchronization, it generates a pulse train of period 89.1 µs corresponding to 3564 BC using a 40 MHz clock. With every tick of the clock within the train the sequence of L1A trigger and commands stored in the FIFO memory are executed.

ADC IP core
The ADC IP core receives and deserializes 16 ADC channels at a data rate of 480 Mbps each using the ISERDES resources in the Virtex-5 and some extra logic to achieve the correct data frame synchronization with the ADC board. This IP core uses 24 kb RAM memories to store the digitized trigger and muon signals coming from the adder boards. The ISERDES deserializes the ADC output data continuously and the deserialized data is stored into the FIFO memories once a special command is executed by the TTCvi emulator IP core.

High Voltage and LED driver IP core
The High Voltage and LED driver IP core implements a very simple remote control interface to the HV and LED boards. It allows switching on/off the HV module and controlling the pulse generation of the LED driver. The pulse generation is internaly synchronized with the ADC IP core to initiate storing data into the FIFO memories at the correct time.

Operating system
A custom lightweight version of embedded Linux runs on the PowerPC providing a flexible enviroment to run the server and the different analysis applications. This Linux version has been built using the Embedded Linux Development Kit 4.2 from DENX [8] and includes the BusyBox package providing a large number of applications.
The Linux Kernel is combined with the FPGA configuration file and the root file system in a System ACE file which is stored in a CompactFlash card. This file contains all the data needed to configure the Virtex-5, initialize the internal block RAM, initialize the external RAM memory with the Linux image and boot up the PowerPC. When the ML507 is powered on, the System ACE Controller on the ML507 boots up the embedded system from the CompactFlash card.
Moreover, the Linux image includes a set of drivers and libraries written in C++ allowing the embedded Linux to manage the hardware through the 32-bit registers allocated in the IP cores.

Software
The new MobiDICK maintains the same architecture as the previous system having the ML507 as server and the laptop as client. Willy software from the previous MobiDICK is used to manage the different tests needed for certification, sending commands to the server which controls the rest of the hardware, extracts and processes data and sends results back to Willy. A screenshot of Willy can be found in figure 8.

MobiDICK tests
The following thirteen tests provide the functionality to certify the front-end electronics: • Communication with motherboard. The first test confirms that the communication with the front-end electronics is alive. It also checks the serial number of the different boards with the ones stored in the Willy database for the corresponding drawer.
• Trigger tower injection test. This test manages the readout of the analog signals coming from the adder boards using the ADC board. The CIS generates analog signals proportional -8 - to preprogrammed charge via TTC commands which are summed in the adder boards in groups of five channels.
• Data integrity test. In this test the digitizers are configured to transmit different data patterns. The interface board receives this data and sends it via the SFP fiber using the G-Link protocol. The aim of this test is validate the complete digital readout chain from the digitizer to the optical links.
• Charge injection test. This test allows the verification of the digitizer functionalities and the digital readout using the CIS. A capacitor in the output of the PMT is charged with a programable charge and the produced signal is digitized and transmitted through an optical fiber to verify the correct response of the electronics.
• Noise test. The purpose of this test is to measure the noise levels introduced by the electronics. The signals of the non-powered PMTs are digitized and read out using the SFP fiber through the interface board.
• Fast test. In this test the MobiDICK sends L1A triggers with a data rate of 100 kHz to the front-end electronics checking in real time the integrity of the incoming data with the CRC module of the G-Link IP core.
• Noise test with HV. In this case the noise levels are measured with the High Voltage power supply feeding the PMTs switched on. Therefore, an increase in the noise levels from the PMTs are expected which is taken into account when the check is performed.
• Communication with HV CAN. Checks the status and communication with the HVmicro board using the CAN bus interface.
• Communication with HVOpto card. The aim of this test is to check the funcionalities of the high voltage distribution electronics in the front-end electronics using the CAN bus interface.
-9 - • HV regulation test. During this test the voltage value set stored in EEPROM of the HVmicro board is checked with the Willy database. The CAN bus interface is used for the readout of the EEPROM.
• Integrator readout test. This test confirms the linearity and noise level of the ADC-I board.
The CIS transmits a preprogrammed signal to the ADC-I board which is read out using the CAN bus interface.
• Integrator readout test with HV. It performs the Integrator readout test with the high voltage feeding the PMTs switched on. An increase in the noise is expected and therefore considered in the performed validation checks.
• LED test. The LED test aims to test the complete readout chain injecting a light pulse with a known shape via a dedicated fiber into the PMTs. The readout of the digitized data is performed using the SFP module and checking its integrity.
• Digitizer stuck bit test. This test configures the digitizers to transmit different data patterns. The generated data is verified to detect stuck bits at high or low logic values.

Conclusions
A new test-bench to guarantee the correct operation before and after the insertion of the TileCal super-drawers during the maintenance periods has been presented. The redesign of the MobiDICK system has been motivated by the difficulty finding replacements for some parts of the previous system and the possibility of improving the mobility of the portable test-bench. The new MobiDICK comprises a Xilinx ML507 evaluation board and a set of custom and commercial boards which replace the old system based on a VME system. The hard core PowerPC is used as processor in a custom embedded system which runs a lightweight Linux version as operating system. A set of custom IP cores has been implemented in VHDL to replace most of the functionalities implemented with VME crates in the previous system.
Currently, the new MobiDICK is being used during the LS1 to certify the proper operation of the TileCal super-drawers electronics. Moreover, it is being used to qualify and measure the noise levels in the analog muon trigger signals which will be used after the LS1 shutdown period in the Level-1 trigger muon system to reduce the large fake trigger rates.