CERN’s ATLAS project: A CAN control network of epic scale

As scientific challenges go, you can’t get much bigger than recreating conditions as they occurred immediately after the Big Bang. This is this mission of the CERN Large Hadron Collider (LHC), which smashes protons to further humanity’s understanding of a wide range of theoretical physical phenomena, including answering the question of what is mass, investigating extra dimensions and the particles that make up dark matter.

During the LHC’s first run (2008–2012), among the successes chalked up was identification of the elusive Higgs boson. During the second run (2015–2018), the energy used to produce the collisions was increased and more data was collected from the two particle detectors, ATLAS and CMS, in order to observe more clearly the interplay between particles. At 46 m long, 25 m high and 25 m wide, ATLAS is the largest of the two detectors, comprising six detecting subsystems arranged in layers around an inner chamber.

Aside from the data created by the experiment itself (recording the collision paths, momentum and energy of the particles), monitoring and controlling ATLAS is a data challenge in itself that is entirely run on CAN. During the LHC’s first run, hundreds of Kvaser’s PCIcanx boards played an important role within ATLAS, providing the link between the 7000-tonne detector and the 100 or so PCs that control and supervise it in ATLAS’ electronics rooms. Each of the PCs had up to three Kvaser cards, controlling over 5000 CAN nodes on more than 300 CAN buses and using CANopen as the high-level communication protocol.

The CANbus nodes were designed and developed by ATLAS in collaboration with the National Institute for Nuclear Physics and High Energy Physics in Amsterdam and the Petersburg Nuclear Physics Institute. Referred to by the researchers as Embedded Local Monitoring Boards (ELMBs), Kvaser worked with ATLAS staff to optimise the PCICanx boards to suit the ELMBs needs.

Asked why Kvaser’s boards were chosen, Dr Burckhart, in charge of the ATLAS Detector Control System (DCS) project at the time said:

“Kvaser’s initial form factor already fit well with our requirements. And following our suggestions they made a new version of their board, which exactly matched our needs. An additional plus was that they had CAN interface cards in USB and PCMCIA form factors which support the same application software. Kvaser was also very responsive when initial software fixes were needed.”

Among the physical requirements determining CERN’s choice of a CAN interface board was that each port would need an independent buffer and be independently controlled (so resetting one port wouldn’t affect another). From a software perspective, it was necessary to support Windows and Linux, and that a simple and intuitive API was needed.

“We were very happy with Kvaser’s PCICan cards and they ran very stably,”

confirmed Dr. Stefan Schlenker, now responsible for the ATLAS detector controls at CERN. Dr. Schlenker took over from Dr. Helfried Burckhart, who led the specification of the system. As Dr Burckhart explained, CAN was an automatic choice:

“It is robust, has good industrial support (including at the chip level), is inexpensive and provided adequate functionality.” Dr Schlenker confirms that ATLAS’ CAN system “is foreseen to operate for at least another 15 years with minor modifications.”

In 2013 and 2014, during CERN’s first long shutdown, the control system was updated as it was becoming more difficult to find PCI-based servers, and it was necessary to increase CAN port density. ATLAS moved to rack-mounted CAN to USB interfaces, with two USB ports serving 16 CAN ports.

As speeds continue to increase within the LHC, so the detector control system (DCS) must evolve too. Whilst no major control system changes are planned during CERN’s second long shutdown (2020/2021), with many parts of the detector programmed for renewal during the next shutdown in 2025 – in particular, a move to an all silicon inner detector – more fundamental updates to the control system will be needed at that time. The original DCS monitored some 200,000 slowly changing parameters, such as voltage, current, temperature and pressure. Beyond 2024, the volume of data increases to millions of readouts per few seconds, so a move to CAN-to-Ethernet cards is scheduled for that time.