The LHCb Silicon Tracker – Control system specific tools and challenges

The Experiment Control System (ECS) of the LHCb Silicon Tracker sub-detectors is built on the integrated LHCb ECS framework. Although all LHCb sub-detectors use the same framework and follow the same guidelines, the Silicon Tracker control system uses some interesting additional features for operation and monitoring. The main details are described in this document. Since its design, the Silicon Tracker control system has been continuously evolving in a quite disorganized way. Some major maintenance activities are required in order to keep improving it. A description of those activities can also be found here.


Introduction
The LHCb [2] experiment control system follows a very clear integrated architecture [3] which was defined early in the implementation process. These guidelines are generic but leave a lot of freedom, from the software point of view, for the design and implementation of the control system for each sub-detector. The control system of the LHCb experiment is implemented using a Finite State Machine (FSM). The FSM hierarchy and node grouping is defined following hardware and logical detector partitioning criteria; commands travel from the top of the hierarchy to the bottom and the state of a node depends on the state of its children nodes. For each sub-detector such a FSM is defined and they all follow the same domain partitioning and operation guidelines. The Silicon Tracker sub-detectors, namely the Inner Tracker (IT) and the Tracker Turicensis (TT), have two FSMs each instead of one. They both have an operation FSM that is referred to as the ECS FSM, and a paralel Safety FSM. This extra FSM is needed because the Silicon sensors need to be cooled at all times and, therefore, temperature and humidity conditions need to be checked to prevent ice formation. In order to monitor those conditions, the electronics must be powered at all times and monitored together with the temperature and humidity. Another essential task for the Silicon Tracker control system is the periodical data extraction of the operation conditions from the online databases to monitor the performance of the detector. For example, the historical data for the temperature and leakage current of the silicon sensors must be regularly extracted from the database in order to be able to perform ageing studies of the sensors. Special tools have been developed for this purpose. The initial Silicon Tracker ECS software structure has undergone many changes and 1 LHCb Silicon Tracker collaboration full author list in [1]. modifications since its first implementation [4] during the detector commissioning phase in 2007. The coding process was driven by the need to test the hardware of the detector. This lead to a very complex and inefficient code with little or no documentation. After the commissioning phase some cleaning of the code was performed for the production system but a full redesign is needed.

Detector Safety FSM
In order to protect the detector electronics, automated power off actions must be in place. The LHCb experiment uses the Detector Safety System (DSS) [5]. This system monitors dedicated sensors, such as thermo switches, water leak detectors or smoke detectors, placed close to the detector and the service electronics. The sensors are connected to a PLC system which is completely independent from the Experiment Control System (ECS). When an alarm is raised, and after a defined small delay, the DSS cuts the power to the detector electronics. The power cuts triggerd by the DSS can be harmful to the electronics. On top of this, the temperature and humidity conditions of the silicon sensors are not monitored by the DSS, because no suitable sensors were found. This is why, in the case of the Silicon Tracker, an additional safety level was introduced in the shape of a dedicated Safety Finite State Machine (FSM). The Safety FSM [6] uses the ECS cooling, temperature, humidity, low voltage and high voltage readings to detect potentially dangerous situations and power off the electronics in a controlled way.
The main tasks of the Safety FSM are: • Prevent ice formation in the detector box.
• Protect against over-current or over-voltage in the LV power supplies or LV regulators.
• Protect against high temperature of the control electronics.
• Protect against high temperature of the silicon sensors.
• Protect against high leakage currents in the silicon sensors.  To accomplish this, the Safety FSM is linked to the main operation ECS FSM and monitors state changes of some of its nodes. When a non-allowed state combination occurs, the Safety FSM executes automatically a power off action of the affected electronics. The Safety FSM hierarchy, displayed in Fig. 1, divides the detector in quadrants for TT and in half-stations for IT ,and acts only on the affected groups when an error occurs or when a not safe state combination is attempted. Changes in the main node or in the group nodes are reported and logged. A simplified description of the different state combinations can be found in Table 2.

Specific tools
The Silicon Tracker ECS is mainly used for operation, but it is also very useful for troubleshooting of sub-detector problems and monitoring of detector conditions. Using the LHCb ECS framework, specific tools have been designed and implemented to provide help with these tasks.

Automated data monitoring
Regular extraction of data from the online conditions databases is needed for fast ageing studies of the silicon sensors. The task is automated by a control manager executing a dedicated script which collects the maximum leakage current and average temperature for each sensor, producing a single file for each run. These files are then used as an input for a python script which generates ageing plots like the one in Fig. 3. This step is also automated by a cron job running in a linux machine on the LHCb online cluster.

Mapping
The partitioning schema for the sub-detectors is different for each of the domains: • High voltage distribution.
• Low voltage distribution.
• Distribution of clock, fast control and slow control signals.
In order to ease the diagnosis and troubleshooting of operational problems, several mapping tools depicting the location of the silicon sensors and the service electronics have been put in place. These tools use a graphical interface to show the link between the different hardware components.  Figure 3. Automatically generated ageing plot for TT which shows the leakage current and the temperature in the silicon sensors together with the integrated luminosity versus time.

ECS redesign during LS1
The Experiment Control System for the Silicon Tracker, in the context of the LHCb ECS framework, was first designed and implemented from 2007 to 2009 during the detector assembly and commissioning. This had the following disadvantages: • Several authors (not always overlapping).
• Different implementations for IT and TT detectors.
• No clear library structure.
• Little or no documentation.
• No version control.
• Built on an "as needed" basis.
Some effort was made at the end of 2009 and beginning of 2010 to merge the FSM hierarchy and behavioural definition domain, and the control libraries. However, the system was already in production so the extent of these modifications was limited. Since then, only some changes and fine tuning were performed to address very specific functionalities. As a result, the ECS libraries have more than fifty thousand lines of highly patched and replicated code distributed among too many files as shown in Fig. 4. Also the FSMs of IT and TT needed to be maintained separately as they are equivalent but not identical.
During the first LHC Long Shutdown (LS1), full access to the control system is possible, and therfore a redesign [7] can be implemented. The aim of this consolidation is to improve the control system in terms of speed, reliability and mantainability. The main tasks to accomplish this can be sumarized in the following points: • Define a new library structure.
• Correct, simplify, optimize and document the code.
• Implement version control on the libraries. • Adapt the user interface to use the new libraries.
• Define a comon FSM hierarchy for both sub-detectors.
• Simplify the FSM hierarchy and the behavioural definition of the nodes. The design process has finished and the implementation has been tested in a virtual environment and will soon be deployed and tested in the production system.

Summary
The control system for the LHCb Silicon Tracker, implements specific tools designed to cope with its requirements in terms of control, safety and monitoring. The design and implementation process of the core libraries for the control system was disorganized and its performance is not optimal. This lead to the need of a full redesign that will be finished before the end of the first LHC long shutdown. Although most of the system specific challenges have been addressed, the control system is continuously evolving and improvement is still possible.