What spoiled the CERN party

NINE days after the spectacular switching on of the worlds biggest and most powerful particle accelerator, the Large Hadron Collider (LHC), at the European Organisation for Nuclear Research (CERN) near Geneva (Frontline, October 10, 2008) on September 10, 2008, which was watched by an estimated one billion people and more across the world, a completely unexpected mishap struck this international scientific endeavour, the worlds biggest.

Built over nearly a decade and a half at a cost of about $8 billion by an international consortium of thousands of scientists, the gigantic, 27-kilometre-long circular machine is located in a 3.7-metre-diameter tunnel 100 m underground that straddles the Franco-Swiss border. The aim of the LHC is to reveal the fundamental structure of matter by causing counter-rotating beams of protons to collide at very high energies and temperatures, recreating the particle processes of the very early universe.

The incident was the perfect spoiler of the long ongoing party at CERN to celebrate getting things right well-collimated proton beams in both directions that remained stable over several hundred orbits in the very first attempt itself, not an easy task by any means. The builders of the machine, led by Lyn Evans of CERN, and the large community of physicists involved in the six experiments, with its mammoth particle detectors, were hoping to have the first proton collisions by November 2008 and for the physics to begin as soon as data started pouring in.

The setback has meant that the accelerator is likely to be up only by July 2009, according to the report of the committee that went into the cause and the nature of the incident. Indeed, if all contingency measures to make the machine completely secure against a similar incident in the future are to be put in place right away, it could mean that the serious physics will only start as late as 2010.

After the first successful beam injection on September 10, 2008, the next step in the commissioning process was to bring in the radio frequency (RF) system that keeps the beams bunched and prevents them from spreading around the beam pipe and to rev up their energy gradually from the initial 450 million electronvolt (MeV) to 5.5 tera, or trillion (1012), electronvolt (TeV) by the year end when the physics was set to begin. Eventually, of course, the beams would have been brought over a period to the rated peak value of 7 TeV.

Each of the eight 3.3-km-long sectors of the accelerator ring consists of a periodic lattice of twin-aperture superconducting dipole and quadrupole magnets installed in a continuous cryostat. The beams pass through the two apertures in the magnets. Each elementary cell of the lattice is 107 m long and includes six dipoles and two quadrupoles.

The dipoles use niobium-titanium superconducting cables operating at the extreme low temperature of 1.9 Kelvin, or K (271.3 C). This temperature is maintained by superfluid liquid helium at a pressure of 1.3 atmospheres in an enclosure surrounding the magnets. Two consecutive cells constitute a subsector.

Even before the first switch-on, seven of the sectors had been tested at the high nominal current of 9.3 kiloamperes (kA) that they are designed to carry, which corresponds to a beam energy of 5.5 TeV. Only the sector designated as 3-4 remained to be similarly tested and commissioned. The dipole circuit of this sector had only been powered up to 7 kA corresponding to a beam energy of 4 TeV before September 10. Sector 3-4 houses 154 dipoles and 55 quadrupoles. In all, S3-4 contains 66 contiguous magnet interconnections.

Very high currents of many thousands of amperes can safely pass through superconducting circuits, but even a very small electrical resistance, above a certain tolerance limit in the circuit, can prove to be disastrous. An electric resistance (which would show up as an electrical voltage across the resistive region) means the dissipation of electrical energy in the circuit as heat, and very high currents would result in an enormous amount of heat being generated. This is precisely what happened on the morning of September 19 as the engineers were ramping up the current in the main dipole circuit of S3-4 at the nominal rate of 10 amperes per second (A/s) to attain the desired 9.3 kA.

A report of the committee investigating the September 19 incident, which was released on October 16, concluded that the cause was a faulty electrical connection between two of the accelerators magnets, which melted at high current. This faulty electrical connection, it turns out, was nothing more than bad soldering! This led to severe mechanical damage and the release of helium from the magnet cold mass into the accelerator tunnel. Following detailed studies, this interim determination has now been confirmed by the final report, which was released on December 5.

The helium leak is not the first incident to happen with the LHC. After the LHC was turned on by successfully sending a clockwise beam (beam-1) on September 10, which circulated for about an hour and made three revolutions, the counter-clockwise beam (beam-2) was sent through the accelerator the same night and it made hundreds of turns. But the operations of the inaugural day were carried out without the periodic kicks with the RF pulses that compensate for the beams energy losses. The next night the RF was turned on, and beam-2 was successfully captured into a stable orbit.

On September 12, the same operations were to be repeated on beam-1, but late that evening the LHC developed a hitch. A 30-tonne transformer that cooled part of the collider failed. This led to the switching off of the main compressors of the cryogenic system for two sectors. Since there was no spare with the Electrical Engineering Group of the LHC, a spare transformer from one of the LHC experiments (called CMS) was refurbished and used as a replacement. While the transformer was being changed, the cryogenic system had to be kept in the standby mode, with the two sectors kept at 4.5 K (268.7 C), instead of 1.9 K. The week that followed was used to bring the sectors back to operating cryogenic conditions.

Though this was not a major event compared with the later one, inexplicably, details of it were revealed only on September 18. The journal Scientific American, in fact, commented that perhaps the group spearheading the LHC commissioning did not want to spoil the celebrations. But, of course, CERN scientists were little aware that the real spoiler was waiting to happen the very next day.

The first signs of a problem in S3-4 were detected at about 11.19 a.m. on September 19 when, with the current raised to 8.7 kA, a resistive zone developed in the electrical bus in the interconnect region (see picture) between a dipole and a quadrupole. The evidence for this was the appearance of a voltage of 300 millivolt (mV) in the interconnect region, which rapidly grew to 1 V within 0.39 s and triggered a runaway collapse of the system.

Unable to maintain the current ramp, the S3-4 power supply tripped at 0.46 s. At 0.86 s, the energy discharge switch opened to bring dump resistors into the circuit. The massive air-cooled dump resistors are designed to remove as much as 1 gigajoule, or thousand million (109) joules, of energy build-up from the dipole circuit. An estimated huge amount of 200 megajoules (106) of energy was released into the dump resistors following the incident. (One joule is the energy gained by 1 kg in falling from a height of 1 m.)

Within the first second, an electrical arc developed in the interconnect and punctured the helium enclosure. As a result, a large amount of helium was released into the insulating vacuum. This also led to a rapid pressure increase inside the LHC magnets, and a large pressure wave travelled along the accelerator in both directions for a few hundred metres. After three or four seconds, the vacuum in the beam pipes degraded to atmospheric pressure.

While the cryostat vessel of the magnets contains helium at a pressure of 1.3 atmospheres, the helium enclosure of the interconnect regions is designed for a pressure of 20 atmospheres. The helium enclosure is protected from overpressure by quench relief valves, which are installed every 107 m and are set to open at 17 atmospheres. These self-actuated quench relief valves on the helium enclosure reportedly opened at 17 atmospheres as per design and contained the pressure rise to below 20 atmospheres except in one subsector where it reached a maximum of 21 atmospheres.

Similarly, the spring-loaded relief discs two in every vacuum subsector which are set to open at a pressure of 1.1 atmospheres, opened as per design when the pressure in the vacuum enclosure exceeded atmospheric pressure and let the helium into the tunnel. But they were unable to contain the pressure below 1.5 atmospheres in the vacuum enclosures of one of the subsectors.

This resulted in large forces being exerted on the vacuum barriers that are located every two cells to separate the vacuum enclosures of neighbouring subsectors. These forces displaced dipoles in the affected subsectors from their cold internal supports. The sudden burst of gas also knocked the cryostats housing the quadrupoles from their external support jacks by as much as 50 centimetres. In fact, in some locations the support jacks broke away from their anchors in the concrete floor of the tunnel (see picture).

The displacement of the cryostats also damaged the cryogenic plumbing in some places. The collateral damage from the point of initiation seems to extend over a few hundred metres.

In the immediate aftermath, as much as two tonnes of helium was estimated to have escaped and released into the tunnel. The resulting helium cloud triggered the oxygen deficiency hazard detectors and alarm systems in the tunnel. This caused an emergency trip that switched off all electrical power and services in the S3-4 zone of the tunnel. In the continuing leakage of helium from the open circuits until the power supply was restored in the tunnel and the cryogenic valves were reactivated, another four tonnes of helium was estimated to have been lost. That is, of the 15 tonnes of helium in the sector, about six tonnes was lost. This corresponds to about 30,000 m3 of gas under normal atmospheric conditions. Firemen had to enter with oxygen masks because of the resulting severe oxygen deficiency, which lasted for several hours.

While the anomalous resistance in the cell that triggered the event would have led to the immediate quenching of that particular dipole, the sequence of events that followed the helium leak, the pressure build-up and the consequent loss of cooling in the magnets and vacuum degradation triggered the quenching of magnets in other sections of the sector in a cascade. (A quench is the phase transition from the superconducting to the normal state. Quenches can be initiated by energies of the order of a few millijoules, which can be caused by movement of the superconductor or current losses or cooling failures or any other heat source.) It is believed that as many as 100 dipoles suffered quenching following the incident.

When a magnet quenches, the conductor becomes resistive, which can lead to excessive local heat generation that can damage the magnet. In order to protect the magnet, the quench must be detected, the energy in the circuit of the magnet quickly extracted and the magnet current switched off within a fraction of a second.

The quench detectors in the accelerator dipole circuit had a sensitivity of 100 mV but no resistive voltage was registered by any of them. This, according to the report, rules out quenching of a magnet itself rather than of the interconnect in the busbar as the initial cause of the incident. In the sequence of events, the quench detection, power converter and energy discharge systems behaved as expected, says the report.

According to Jrg Wenninger of the LHC Operations Group, unlike the quench detectors provided for individual magnets, interconnects are not individually protected. The entire busbar, consisting of all the magnet interconnections of a given electrical circuit, is protected by a global protection system that includes the switching off of the power supply and the bringing in of the dump resistors for discharge of the excess energy in the circuit. The design of the busbar quench protection system was, therefore, inadequate. It was the inspection of the quench protection system data that revealed that the most likely cause was an electric arc due to rupture of the interconnection. Unfortunately, pointed out Wenninger in a presentation he made in November, this is difficult to prove since the whole dipole interconnect vaporised during the event.

The busbar interconnection between magnets involves a thin niobium-titanium strip from each magnet being connected manually by brazing using a tin-silver soldering alloy layer (see picture). Nominally, the electrical resistance of this interconnect should be no more than 0.35 nano-ohms (n). Post-incident analysis based on logged cryogenic data has revealed that the implicated cell in S3-4 had a resistance of about 200 n. It is believed that bad brazing of this interconnect was the culprit.

The favoured hypothesis for the triggering event in sector 3-4 can now be summarised as follows: An excessive resistance (of 200 n) led to a temperature increase. The superconducting interconnect in that particular busbar location quenched and became resistive at high current, causing a further temperature increase. The top-most copper sheath in the interconnect, which is cooled by helium, can handle the heating only up to a point. Beyond that, as a result of the runaway temperature increase, the layers in the interconnect melted, and the splice, which is soldered and not clamped, opened, resulting in an electrical arc.

Interestingly, a post-incident check to see whether there were any early warning signs in the other seven sectors showed that perhaps there was a hint in a cell of S1-2, points out Wenninger. Controlled calorimetric measurements at different currents in the remaining sectors have now revealed an anomaly in S6-7 as well. But in these cells, while the interconnects between the magnets were found to have resistances below the threshold of 0.35 n, the industry-build dipoles themselves were found to have internal resistances of 100 n and 50 n respectively. Within the magnets themselves, the superconducting cables have welded interconnections between the different coils, similar to what is done between the magnets. Those two magnets have therefore internally a bad interconnection, Wenninger clarified in an e-mail to this correspondent.

But the environment in terms of cooling flow is different, and in addition, the magnets are protected by the quench protection system. The magnets were also tested above [the] nominal on test benches. That is why nothing dramatic happened in them, Wenninger points out. These dipoles will be replaced with spare ones, according to him.

While a lot was learned in the beam start-up, which was excellent, it was not enough to be sure that the rest of the early commissioning would proceed as well as the first three days, says Wenninger. According to him it was just pure chance that the problem occurred with the last sector to be commissioned, which was also apparently the one to be cooled to 1.9 K. The S3-4 incident, he points out, revealed a weakness in the installation quality assurance and a weakness in the magnet protection system, which did not cover dramatic busbar/interconnect incidents.

According to the final report of December 5, though the precise origin of the fault will always remain a matter for speculation, defects similar to the inferred original fault have been reproduced in the laboratory by application of large non-conformities in the procedures on samples of busbar interconnections. As regards the collateral damage, this report says that the helium discharge encountered on September 19 was considered beyond design, and the safety relief devices on the cryostat vacuum enclosure had been designed for a lower discharge.

Overall, says the report, the discharge cross section will eventually be increased 40-fold to cope with a helium discharge twice as high as that of September 19. Similarly, the axial forces resulting from overpressurisation of the vacuum enclosure exceeded the yield point of magnet supports, resulting in their displacement and secondary damage. In the changed design, these anchors in the concrete floor will be further strengthened.

For detailed investigations and repairs to begin, the entire sector had to be warmed up to room temperature. This itself took about two months. According to the report, the in situ inspection has now revealed that a total of 53 magnets 39 dipoles and 14 quadrupoles need to be removed from the tunnel and brought to the ground level, where they will be repaired and reinstalled. This process is expected to take up most of the shutdown period until about the end of March 2009.

A survey of the contamination of beam pipes by soot from electrical arcs and chips of multilayer insulation has indicated that there was no contamination by soot outside the region of the 53 magnets. The contaminated sections of the beam pipe will either be replaced or cleaned. According to the report, contamination by chips of multilayer insulation, which fortunately were deposited only on the beam pipe surface, has apparently been found far away from the position of the original incident.

In addition to the new, sensitive systems that will be put in place in all LHC sectors as part of the recommissioning procedure, a dedicated system for the detection of abnormal resistance and heat sources on the high current busbars and interconnections will be implemented on the whole machine, says the report. This implies the manufacture and installation of over 2,000 additional electronic crates and the pulling of some 160 km of signal cables during the shutdown period.

To prevent the occurrence of a similar incident in the future, the quench protection system for all interconnects will now be upgraded. The worst-case scenario has also been now revised in the light of the incident. Accordingly, the number and size of the pressure relief devices on the cryostat vacuum vessels will be increased. A large port will also be cut into every dipole cryostat and fitted with a full-flow relief device. This, according to the report, will first be done on the warm sectors and gradually implemented on the whole machine. If this has to be implemented on the entire machine, all sectors will have to be warmed up.

Asked whether there was any continued risk given that all the interconnections were unclamped, Wenninger said: Youll get different answers depending on whom you ask. Clamping would, of course, be ideal but cannot be added easily. He put the repair and upgrade costs at around 30 million Swiss francs (20 million or $28 million). As of now, it is planned that the LHC will restart in late July 2009, with the beam energy and intensity limited to minimise risk. On the other hand, if it is decided that a complete upgrade of the pressure relief system first be implemented on all sectors, the accelerator cannot be recommissioned in 2009. The decision in this regard is expected to be taken in February, according to Wenninger.

However, Robert Aymar, the outgoing CERN Director General, stated on December 5: The top priority for CERN today is to provide collision data for the experiments as soon as reasonably possible. This will be in the summer of 2009.

We have a lot of work to do over the coming months, said LHC project leader Lyn Evans, but we now have the road map, the time and the competence necessary to be ready for physics by summer. It, however, remains to be seen whether the new Director General, Rolf-Dieter Heuer, sticks to the same road map or chooses to recommission the LHC only in 2010.