Commissioning of the test system for PS and 2S hybrids for the Phase-2 Upgrade of the CMS Outer Tracker

The Phase-2 Upgrade [1] of the CMS Outer Tracker [2] requires the production of 7608 Strip-Strip (2S) and 5592 Pixel-Strip (PS) modules, altogether incorporating 45 192 hybrid circuits of 15 design variants. The module design makes the potential repairs impractical; therefore, performing production-scale testing of the hybrids is essential. Accordingly, a scalable, crate-based test system was designed and manufactured, allowing for parallel, high-throughput testing. To reproduce the operating conditions, the system was integrated within a climate chamber, which required the development of a remote control interface and the calibration of thermal cycles. The results and lessons learned from the test system integration and commissioning will be presented.

The production scale testing of CMS Outer Tracker hybrid circuits prompted the design of a crate-based test system [3], which can be divided into five main parts (figure 1): a computer running testing software, FC7 [4] FPGA cards to simulate and interpret signals, Test Crate with signal multiplexing boards (backplanes) to switch the connections between 12 test slots, test cards (TC) to operate hybrid-type specific functional tests, and, finally, various jumpers and adaptors to accommodate the 15 distinct hybrid variants.Overall, six types of test cards were designed and produced.The total system quantity, accounting for spares, consists of 110 backplanes, 645 test cards, and around jumpers and adaptors.To ensure the correct operation of the final system, each of the components had to be inspected and commissioned before integration.

Commissioning procedures
Owing to the system complexity a bottom-up approach was selected for integration testing, which begins at the level of each significant component.The initial activities within the scope of functional tests were the assembly of mechanical sockets on the test cards, with the fitting and clearance checks followed by the basic powering tests and e-fusing of the cards with unique ID strings.Advanced commissioning procedures for backplanes and test cards included a subset of functional tests with reference hybrids, allowing the verification of all signal connections and the component-specific functionalities.
The mentioned functionalities vary depending on the purpose of the tested hybrid, therefore we can distinguish the test cards for front end hybrids (figure 2), handling the data from silicon sensor, and -1 -test cards for service hybrids (figure 3), providing powering and communication services.Moreover some of the functions are shared between hybrids for strip-strip (2S) and pixel-strip (PS) sensors, allowing the test cards to reuse the testing procedures between different card types.
The most notable test procedures, with the referenced articles explaining them in more detail, are listed below along the test cards that implement them.
• Short and open finder procedures, which use antenna injections [5] to verify the silicon sensors input lines.Utilized by the Front-end hybrid [6] (FEH) TC (figure 2).• High voltage circuits and filters    -2 - The commissioning time depended on the component type and for a single test card it ranged from 35 to 70 minutes depending on the card complexity.The complete preparation for the quantity of 645 test cards was expected to require around 460 hours.However the estimate didn't account for logistic delays and the need to debug some of the hardware issues encountered during commissioning.

Results of system commissioning
Commissioning results were divided into categories depending on the tested requirements.During the mechanical checks, all test card types were shown to be compatible with hybrids and the test crate system.No clashes occurred when inserting cards in the crates, and the hybrid mounting was safe and secure.The basic powering and e-fusing of test cards provided an initial yield of 95% with the exception of the PS-POH TC, which required manual rework because of a design issue causing a shorted via that resulted in a 75% yield of the first batch.(Rework of remaining cards is in progress.)The most common issues included failures of the USB-SPI bridge chip and shorted power lines within high density connectors.Full functionality testing of recently delivered test cards has been on hold owing to mechanical assembly delays, but the preliminary results lead us to expect a > 90% yield.Recent testing experience with prototype hybrids revealed issues with the test system software, and this required some investigation.Nevertheless, the main scope of the functional tests is finalized.The graphical user interface has been successfully used with the multiple crate and mixed test card types during the testing campaigns, but exhaustive reliability tests are still required and planned for the coming months.

System integration within the climate chamber
The hybrid circuits need to be tested at the target operating temperature of the tracker and the warmest temperature envisioned (in the event of a cooling failure).Consequently the tests are performed at low temperature (-35 • C) and high temperature (+40 • C) using an EXCAL2 4025-HE1 climate chamber (figure 5).Therefore the test system including test cards and backplanes have to perform reliably in these conditions as well.In order to support the test system within the chamber, a custom-made frame was procured and mounted to the walls of the chamber (figure 5(a)).The test crates fit inside the volume of the chamber along with cables and optical fibers, which were routed through a hole to the service rack on the side (figure 5(b)).To ensure safety of the electronic system, the relative humidity and dew point temperature must be monitored to prevent the development of condensation in the chamber volume while the tests are performed.A remote-control interface was developed (figure 6) to decrease the need of human supervision of the functioning chamber, allowing for control, monitoring, and error handling within the test procedure.The control module was added to the graphical user interface, and this allows for the monitoring of the system and the automation of test procedures.

Problems discovered during system integration
During the initial operation of the system, with thermal cycling implemented, there was an incident in which ice formed inside of the climate chamber (figures 7(a) and 7(b)).The incident led to a pause in the integration effort and the eventual identification of two main causes of the ice formation: insufficient sealing of service hole for cable routing, and blocked drainage holes.Those issues resulted in excessive values of the relative humidity (%RH) and the dewpoint temperature, which caused humidity condensation and droplet formation on the hardware.-4 -

Applied solution
The leakage issue was eventually solved by replacing the cable passthrough with a ROXTEC R frame1(figure 7(c)).The chamber's dry air inlet was connected and multiple characterization cycles were executed (figure 8) with different dry air flow rates.In each test, 3 cycles were executed in sequence with short 5 min breaks to open the chamber doors to simulate a real use case scenario.The increased intake of dry air was lowering the relative humidity during the cycle.It was also observed that the performance had improved drastically for the subsequential cycle execution (figure 8).Additional tests showed that the best environment for the thermal cycle is achieved after executing an initial cycle with dry air injection and high temperature to remove moisture from the climate chamber components.

Summary and future work
Most of test system components, with the exception of the PS-FEH HV and POH test cards, have been delivered and assembled; their commissioning is now being finalized.The climate chamber has been extensively tested to ensure its safe operation for electronics inside.Remote control modules for climate chamber and power supplies were implemented.Two installations of the system are prepared and ready for first test campaigns with the expected production lot of hybrids.Plans for future work include: installation of the test system at the hybrid manufacturer premises followed by testing campaigns conducted on a large scale to exercise the quality control processes and logistics during hybrids production.Finally, improvements to the software and graphical user interface will be implemented based on initial feedback from early users of the system.
(a) Backplane (b) Test card flexible jumpers

(a) 3 Figure 5 .
Figure 5. Integration of the test system within climate chamber.

Figure 6 .
Figure 6.Screenshot of the remote control module for the climate chamber integrated in the GUI.

Figure 7 .
Figure 7. Effects of ice forming incident.

Figure 8 .
Figure 8. Example of 3 sequential thermal cycles (C1-C3) executed with the same dry air intake configuration.Measurements of dewpoint (Dewp), temperature (TProd), and relative humidity (RH) are done by sensors within the climate chamber.