Operation of the PSI Accelerator Facilities in 2016
Andreas Lüdeke, Anton C. Mezger, Angelina Parfenova, Marco Schippers, Mike Seidel, Division of Large Research Facilities, PSI The Department of Large Research Facilities has responsibility for the operation and development of the three accelerator facilities at PSI. These are: the High-Intensity Proton Facility, the Swiss Light Source, the Proscan medical accelerator. This article covers operational aspects of the facilities, as well as performance highlights and new developments achieved in them.
High-intensity proton accelerator (HIPA)
The year 2016 was, inspite of a couple of challenges, quite successful for HIPA operation. The overall availability of the facility was close to 90%. The license to run the facility up to 2400 µA in normal operation and with 2600 µA in test mode was obtained from BAG by beginning of May.
The user operation in 2016 (KW19) was started as scheduled on May 9 with an average production current of almost 2000 µA extracted from the Ring cyclotron. Figure 1 shows the distribution of the weekly availability over the year 2016, the weekly rate of the beam trips and the averaged beam current at the meson production target E, as well as services with machine development intervals. During the first week of operation the number of trips was 490, which is slightly higher than usual. In addition a faster increase of the radiation level at the ionization chambers MRI3 and MRI11 than in a year before was noticed.
Until KW24 the facility was running smoothly at approximately 2200 µA with a 93-97 % weekly availability. On June 9 high current operation at 2400 µA took place. KW24 began with the breakdown of target E and a water leak in the Ring extraction
magnet AHA. The damage of the target E was caused by the crumbling of the ceramic ball bearing and spontaneous clamping on its axis. Thanks to the efforts of the technical team the challenging exchange of AHA magnet was successfully carried out. This outage resulted into weekly availability of 28% and impacted the yearly availability. After the target E and AHA magnet were exchanged, centering problems at the target E resulted in further reduction of the overall availability. KW25 was also accompanied with an increased number of EEC-discharges.
On June 25 a flow reduction in the cooling system of SINQ target was detected, which resulted in a complete shut down of the SINQ. The resolution of the problem required the exchange of the target. The operation was switched to the beam dump with a lower beam current of roughly 1700 µA, so that the availability could be kept at the 97% for other experiments. In KW30 (a service week) the first attempt to install a 4D emittance measurement in the 870 keV beam line was undertaken;
unfortunately it led to a vacuum problem. This problem could be solved, but the installation is still ongoing.
Figure 1: Operation of the Proton Facility: availability (purple), beam trips (red), average beam current at the target E (black) and services with machine developments (green).
The operation on the beam dump was unusually long for almost four months due to the non-availibility of SINQ. The last month of operation on the beam dump was affected by a water leak in the region of beamdump. On October 9 an increase in vacuum pressure up to 10-3 mbar was detected at GH23 so that the beam current at the beam dump had to be reduced first to 1200 µA, afterwards even to 1050 µA. The beam optics in the beam dump region was slightly modified to bypass the first segment BD1, while shifting the beam to the second BD2 and the third BD3 segments. This allowed to raise the beam current again up to 1400 µA.
On October 31 the new SINQ Target was commissioned. The beam optics was slightly modified aiming at a safer operation. In particular, the beam divergence at target E was increased in order to lower the beam current density at the SINQ target in the unwanted case of beam bypassing target E. Moreover, a larger beam footprint at SINQ was adopted in order to match the aperture monitors located at the SINQ collimator system. The beam was centered by means of profile monitors, while the temperature of the SINQ target was continuously monitored. The beam current was stepwise increased under permanent monitoring and optimization of beam losses. According to BAG regulations, the beam current had to be limited to 1700 mA. The time period from KW44 until the end of operation on KW51 was characterized by availabilities 91-97% with a rather stable operation. Note that periodic EEC-discharges every 25 minutes stayed an issue until the end of operation on December 23. The operational data 2016 of the facility are given in Table 1.
Table 1:
The various relative contributions to the downtimes in 2016 are shown in Figure 2. Major contributions came this year from the water leak of the AHA magnet (52.4%) and the target E exchange (24.1%). The old analog electronics or Ring’s beam phase probes
MRF were replaced with digital electronics in 2015, which triggered a measurement campaign in order to prove their functionality and calibration continued in 2016. The existing correction algorithm based on phase response matrix used to correct the magnetic field profile to keep the phase of the beam as close as possible to the RF-phase was reviewed [1]. As a result of the joint effort with the beam dynamics team an old control room application for the phase correction for Injector2 and Ring cyclotrons was replaced by a new one based on C++ Qt.
Figure 2: Downtime characterized by category of HIPA outages longer then 5 minutes (ca. 462 hours).
HIPA consumes almost a half of PSI electrical power. By switching off beamlines (IP2/UCN/IW2/SINQ/PK1/PK2) that are not used longer than for half an hour, a notable amount of electricity could be saved within the facility. An application (SLEEP) for putting facility beamlines and cavities into standby during outages with no beam longer than 30 minutes was commissioned in 2016 and successfully run over the year. The total amount of energy saved by December 23 was 2871.3 MWh, where 1700 MWh were saved due to the breakdown of SINQ.
Precise on-line measurements of the number of turns in Ring cyclotron have been performed using a fast oscilloscope and a LabVIEW program. A standard application to be used by the operators should be ready for the beginning of the 2017 operation.
Proton Therapy PROSCAN
In 2016 the cyclotron and beam lines for the proton therapy facility at PSI have been operating with an Up-Time of more than 7100 hours, similar as in 2015. However, the availability of 95.6%
is a few percent lower due to a major event in the cryogenic system of the s.c. cyclotron. As can be seen in the figure, approximately half of the unscheduled down time was due to this problem. The other causes show a similar distribution as last year. No components have contributed exceptionally to the unscheduled down times.
Beam-time statistics for HIPA 2016 Total scheduled user beam time 4812 h Beam current integral
• to meson production targets
• to SINQ
• to UCN
• to isotope production targets
7.96 Ah 2.49 Ah 0.148 Ah 0.0093 Ah Outages
• total outages (current < 1 mA, time > 5
min) 462 h
Availability 90 %
Figure 3: Operating hours per year, availability of PROSCAN (left) and unscheduled downtime by causes.
The figure of Up-Time reflects the time that cyclotron and beam lines have been in the status “ready for beam delivery”. Down times due to interlocks from the patient treatment side have thus not been included in these statistics.
The problem in the cryogenic part occurred after a routine service of one of the cryo-coolers in the helium supply-cryostat of the cyclotron. The 50 K parts of these two-stage coolers are mounted on a connection to the heat shield. When replacing the cryo- cooler for a service, this connection is routinely heated to prevent ice formation. However shortly after this replacement the insulation vacuum has broken down and the helium in the coil and supply cryostat has evaporated. We assume that this has been caused by an ice-bridge, formed after heating and recooling the 50 K support of the cryo-cooler. Air that has been leaking into the insulation vacuum in the cryostat during the ten years since the cyclotron has been delivered at PSI, forms the ice. This leak (≈10-9 mbar.lit/sec) is not exceptionally large, but it has been enough to build up ice-bridges between the cold parts and the external wall of the coil cryostat. The heat transfer due to this connection has initiated an avalanche of events, causing a temperature increase of approximately 100 K of the coil, evaporation of the helium, evaporation of some of the ice in the insulation vacuum and the loss of the insulation vacuum.
During the following 6 days we have restored the insulation vacuum by adding extra pump capacity. Then we could cool down and refill the coil again with helium and restart operation.
Fortunately the beam dynamics did not show any indication of a change in the magnetic field due to this event. We had to cancel 4 days of patient treatments during this period.
After consulting Varian ME, the supplier of the cyclotron, we have decided to plan a shut down in 2017 to heat up the coil and insulation vacuum to 300 K, so that all ice remnants will be evaporated and pumped out. We expect to have the cryogenic system in a much better condition then. This period will also be
used to service, replace and improve several vacuum and cryogenic components connected to the insulation vacuum.
Installation of some extra pump capacity will also help to deal with the small helium leak (<10-7 mbar.lit/sec) from the coil into the insulation vacuum.
Swiss Light Source
The operation statistics of the Swiss Light Source overtopped all former years. The beam availability of 99.1% and the Mean Time Between Failures of more than 8 days both were the best in the history of the SLS.
Two events caused about a third of the downtime, both triggered by disruptions of the mains. The first was a power cut of PSI west of about 15 minutes. The beam was recovered in 6.5 hours – this was only possible because the incident happened during normal working hours, when dozens of people were right there to help restarting all subsystems. The second was a power glitch that caused the main switch of one RF station to trip. The failure recovery took over nine hours. A water sensor was not backed up by an UPS, resulting in a false alarm. The RF expert did spend the night searching for a leak, while it was just the water sensor being out-of-power. The main power switch will now be included in the plant supervision, to avoid similar problems in the future.
Figure 4 shows the number of beam outages of the past years grouped by failure categories. The total number of beam outages reached an all-time-low in 2016. Only four categories had more than a single event: RF (12 events), magnet power supplies (3 events), mains (6 events) and unknown causes (4 events). All these are excellent numbers compared to the past. It is worth as well to mention the absentees on the outage list: since three years in a row there was no single beam loss during user operation from neither an operator fault, nor from a controls problem. Neither water cooling, diagnostics devices, nor thunderstorms caused any outages in 2016.
0 h 1000 h 2000 h 3000 h 4000 h 5000 h 6000 h 7000 h 8000 h
95 % 96 % 97 % 98 % 99 % 100 %
2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
Availability and Up-Time
Figure 4: Beam outage count per failure category.
Figure 5: Number of beam distorting events of the past years.
Figure 5 shows the numbers of the different events for the past years that contributed to the Mean Time Between Distortions. The number of beam interruptions as well as the number of beam- drops has been the lowest in the history of the SLS. With a standard rate of orbit feedback outages we reached the largest Mean Time Between Distortions that was ever recorded for the SLS: 67 hours! Figure 6 shows that the largest fraction of beam outages in 2016 were from trips of the RF. Disruptions of the mains became a significant contribution as well. The 12%
outages from magnet power supply failures is a rather good value, considering the fact that more than 600 magnet power supplies are installed in the SLS. A leak from a water-cooled taper of an in-vacuum insertion device required a 3.5 hour intervention for a provisional repair, causing 8% of the downtime in 2016. This water cooled, moving tapers showed to be a weak point in the design of our in-vacuum IDs. Since it is known now from experience that an operation without water cooling of the tapers is possible, we will detach the water cooling from the tapers, one-by-one in the next shutdowns. A re-design of the
tapers was already planned in the context of SLS 2, the upgrade project planned after the year 2020. Table 2 summarizes the SLS operation statistics of 2016.
Figure 6: Beam outages per failure category in 2016 Table 2: SLS Operation Statistics
Beam Time Statistics 2016 2015
Total beam time 6864 h 78.1% 6872 h 78.6%
• user operation 5016 h 57.1% 5056 h 57.7%
- incl. compensation time 160 h 1.8% 160 h 1.8%
• beamline commissioning 800 h 9.1% 728 h 8.3%
• setup + beam development 1048 h 11.9% 1088 h 12.4%
Shutdown 1928 h 21.9% 1896 h 21.6%
User operation downtimes 25 46
• unscheduled outage duration 45 h 0.9% 49 h 1.0%
• injector outage (non top-up) 1 h 0.0% 11 h 0.2%
Total beam integral 2497 Ah 2497 Ah
Availability 99.1% 99.0%
Availability after Compensation 102.4% 102.3%
MTBF 193 h 108 h
MTTR (mean time to recover) 1.8 h 1.1 h MTBD
(mean time between distortions)
67 h 35 h
Statistical data about the operation of a facility is important to evaluate the reliability evolution over time. But it is as well often used to compare the reliability of similar facilities around the world. The recent study [2] questions the validity of such benchmarking. A meaningful comparison requires the operation metrics of the different facilities to adhere to a common standard.
The authors propose a specific metrics to evaluate the reliability of storage ring light sources across facilities in an objective way.
References
[1] A.S. Parfenova et al., Measurements of the Beam Phase Response to Correcting Magnetic Fields in PSI Cyclotrons, Proc.
IPAC 2016, Busan, Korea, TUPMR019, pp. 1271-1273.
[2] A. Lüdeke et al., Common operation metrics for storage ring light sources, Phys. Rev. Accel. Beams 19, 082802 (2016).
