Production of superconducting 1.3-GHz cavities for the European X-ray Free Electron Laser

The production of over 800 1.3-GHz superconducting (SC) cavities for the European X-ray Free Electron Laser (EXFEL), the largest in the history of cavity fabrication, has now been successfully completed. In the past, manufacturing of SC resonators was only partly industrialized; the main challenge for the EXFEL production was transferring the high-performance surface treatment to industry. The production was shared by the two companies RI Research Instruments GmbH (RI) and Ettore Zanon S.p.A. (EZ) on the principle of “build to print”. DESY provided the high-purity niobium and NbTi for the resonators. Conformity with the European Pressure Equipment Directive (PED) was developed together with the contracted notified body TUEV NORD. New or upgraded infrastructure has been established at both companies. Series production and delivery of fully-equipped cavities ready for cold rf testing was started in December 2012, and finished in December 2015. More than half the cavities delivered to DESY as specified (referred to “as received”) fulfilled the EXFEL specification. Further improvement of low-performing cavities was achieved by supplementary surface treatment at DESY or at the companies. The final achieved average gradient exceeded the EXFEL specification by approximately 25%. In the following paper, experience with the 1.3-GHz cavity production for EXFEL is reported and the main lessons learned are discussed.


I. INTRODUCTION
Superconducting-radio-frequency (SRF) cavities are key components of modern particle accelerators producing high-quality beams. Thanks to their ability to operate with high duty cycle or even in continuous-wave (CW) mode, exploitation of SC cavities has been steadily growing during the last three decades [1].
The European X-ray Free Electron Laser (EXFEL) [2] facility is 4th-generation light source which will produce x-ray pulses with the properties of laser light at intensities much brighter than those produced by conventional synchrotron light sources. The almost 2.1-km-long superconducting linear accelerator of the EXFEL will accelerate electrons to a maximum energy of 17.5 GeV by using a total of 808 SRF cavities installed in the three main linac sections. The cavity design performance requires an average accelerating gradient E acc ≥ 23.6 MV=m at a Quality Factor Q 0 ≥ 1 × 10 10 .
Before the EXFEL production, only the mechanical fabrication has in general been done by industry, with the final surface treatment being performed by research institutes.
The production of more than 800 SC cavities for the EXFEL facility represents the largest deployment of this technology to date, and presented several major challenges. The large number of cavities fabricated within 3 years required a clear production strategy, including instruments for the procurement, set up, and qualification of the infrastructure at the vendors, and quality control (QC).
The collected experience is doubtless very important for subsequent large projects, like the upgrade of the Linac Coherent Light Source (LCLS-II), or the European Spallation Source (ESS). The International Linear Collider (ILC) project can also consider the EXFEL production as a feasibility check and profits from the high statistical significance of the test results of the cavities.
The preparation phase of the EXFEL cavity production started in 2006, and continued for approximately five years until the beginning of September 2010 when the production contracts were placed to RI and EZ. The shipment of serialproduction cavities took place from December 2012 until December 2015. Important preparation measures [3] for the procurement are completion of the: (i) R&D process; (ii) development of a list of qualified vendors; (iii) documentation; (iv) in-house technical review; (v) development of the procurement strategy; (vi) definition of the delivery rate and completion date; (vii) identification of key project personnel.
Several aspects of the requirements for industrialization of the EXFEL cavity production have been developed based on experience with single-cell and nine-cell prototype cavities. A dedicated EXFEL single-cell program established at DESY [4][5][6][7][8] was an important first step. The program was particularly helpful for the qualification of new suppliers of high-purity niobium, the work on niobium specifications, and in developing the specifications for cavity mechanical fabrication (e.g. requirements checking for welding preparation of parts and storage).
After the single-cell program, approximately 50 prototype nine-cell EXFEL cavities have been produced at two European companies, ACCEL (since 2009 RI) and EZ, under the supervision of DESY. 1 Mechanical fabrication was done by industry, while subsequent surface preparation, integration into the helium tank (HT) and rf testing were done at DESY [4,[9][10].
During the production phase, the main effort was dedicated to the implementation of a strategy for setup and qualification of the new infrastructure for surface treatment in industry; the ramp up of the fabrication capability necessary for the desired production rate; and the development of the procedures and instruments for production monitoring and QC.
During the R&D phase, some effort was also devoted to large-grain (LG) cavities. Production of superconducting cavities from LG discs (also called ingot material) was considered as a possible option for EXFEL. Several singleand eleven nine-cell cavities were produced from LG material. The test results satisfied the EXFEL requirements (for details see [11][12]). However, it turned out that the availability of the required >20 tons of material could not be guaranteed, and in order to minimize the risk for the project it was finally decided to use conventional fine-grain material for the EXFEL.
Some aspects of the EXFEL cavity production have been presented and discussed at several conferences and workshops and can be found in the respective conference proceedings [4,[13][14][15][16][17][18][19]. The aim of this paper is to summarize all the important production issues and to discuss the adopted strategy for the cavity procurement and quality management. While transferring the laboratory's knowledge and experience to the two companies has been challenging, the successful results have demonstrated that the EXFEL cavity production has been an excellent example of technology transfer to industry.

II. MAIN REQUIREMENTS AND BOUNDARY CONDITIONS FOR THE EXFEL CAVITIES
The requirements and boundary conditions for the EXFEL cavity production are described in detail in [20].
Here we present only the key points.

A. Cavity design and fabrication
EXFEL uses the so-called TESLA cavity, a nine-cell cavity build from solid niobium with the pi-mode frequency of the lowest TM-mode pass band at 1300 MHz [21][22]. Minor changes in comparison to the original TESLA design have been made for EXFEL, resulting in a moderate cost reduction and some simplification of fabrication [4].
Amongst the more critical requirements are the mechanical tolerances for the shape accuracy of the cells (AE0.2 mm), cavity length (1059 AE 3 mm, between reference rings), and cavity straightness (cell eccentricity < 0.4 mm). The HT is built from titanium (grade 1). The mechanical fabrication had to be done in accordance with the European Pressure Equipment Directive (PED-97/23/EC).
Industry was required to carry out the following steps: the fabrication of the cavities and HT; integration of the cavities into the HT; cavity surface treatment according to the EXFEL specification; assembly in ISO4 (ASTM 10) air quality conditions; and finally shipping to DESY under vacuum. For the cold rf performance tests the cavities had to be equipped with two beam tube flanges, two highorder mode (HOM) antennas, one High-Q fixed antenna, a pick-up antenna, a CF 40 angle valve, the bellows clamp and the tuner ring. A special bellow clamp had to be mounted in order to prevent cavity deformation and keep the cavity resonant frequency constant during transport under vacuum. Figure 1(a) shows a depiction of the cavity integrated in the HT with the helium service pipe (HSP) ready for shipment for the performance rf test at 2K (fully-equipped cavity); in Fig. 1(b) can be seen the inside of the cavity in HT. All cavities underwent at least one cold rf performance test at DESY before being handed over to CEA Saclay for string assembly and module installation. 1 Several of these prototype cavities have been installed in FLASH (Free-electron LASer in Hamburg).

B. Helium tank (HT) issues
The HT welding process was developed at DESY (for more details see [23,24]). A special Field profile Measurement System (FMS), based on a bead-pull field profile measurement, was developed and tested. The FMS allows in situ frequency monitoring while maintaining the cavity in ultraclean conditions. Assembly and disassembly of the FMS has to be done inside an ISO 4 clean room.

C. Rf requirements
The fundamental-mode frequency at 2K is specified as f 0 ðTM010; piÞ ¼ ð1299.7 AE 0.1Þ MHz, and the flatness of the field distribution [1] after HT welding must be better than 90%.
An important rf aspect of the EXFEL cavity is the HOM damping efficiency: The HOM field distribution and damping, generally depending on the accuracy of the mechanical fabrication, are not formally specified in the EXFEL requirements, but have been steadily monitored at DESY. Originally the loaded Q-values (Q load ) of the HOM coupler was expected to be lower than 10 5 (see Sec. VIII).

D. Treatment and assembly
Surface treatment of prototype cavities established a reliable and reproducible process fulfilling the EXFEL performance requirements. An overview of the main preparation and assembly steps is shown in Fig. 2. Initial treatment consists of the removal by electrochemical polishing (EP) of a 110 μm surface layer (or 140 μm depending on the final treatment), followed by an ethanol rinse, outside buffered chemical polishing (BCP) etching, and 800°C annealing under high-vacuum conditions. There are two options for the final treatment, both of which have been tested [9]: "Final EP" with 40 μm of material removal by EP and a subsequent ethanol rinse; or alternatively, a "BCP Flash" with 10 μm of material removal by BCP. Both cases are followed by high-pressure ultra-pure water rinsing (HPR) and a 120°C bake.
The outgassing of all UHV-components for assembly, including the cavity itself, has to be free of hydrocarbons. This means that in a leak-free system with a total pressure below 10 −7 mbar, the sum of the partial pressures of masses above mass 45 has to be less than 10 −3 of the total pressure. The largest permitted leak rate was specified as 10 −10 mbar l=s.

E. Transport
To keep the cavity mechanically in tolerance and also to reduce the risk of degradation of the cavity by particle movement during transport, special transport boxes were developed [25]. The boxes were designed for transport on standard trucks with the cavity in a horizontal orientation. Different foams are glued inside the box and its lid, which effectively hold the cavity sandwiched by the upper and lower halves when the box is closed. Mechanical resonance frequencies in the range of 4-200 Hz were identified and suppressed successfully by the choice of the foam. Transport on normal roads by standard trucks, as well as additional defined shocks up to 6g in the positive and negative directions, were simulated at a test area on one of the prototype cavities in a transport box. No influence on the cavity's fundamental frequency and field profile distribution was found, nor was any degradation of the performance observed.
Fundamental mode spectra and high-Q antenna transmission have been controlled before and after transportation. Comparison of the data for all produced cavities did not indicate any mechanical deformation or problems with antennas arising from the transport.

F. Parts in circulations
Some accessories (beam tube flanges, high-Q fixed antennas, angle valves, bellows clamps) are only foreseen for transportation and cold vertical test. In order to save costs, it was decided to purchase approximately a quarter of the required amount and let these parts be recycled between the labs and industry [parts in circulation (PIC)]. Circulation consists of the following steps: parts in circulation are attached to the cavities at the vendor; vertical rf tests are done at DESY; PIC are disassembled during string assembly at CEA Saclay; PIC are then shipped back to DESY for QC; and finally the PIC are shipped back to the cavity manufactures for reuse. This procedure requires reliable scheduling by both industry, DESY and CEA Saclay in order to guarantee the timely return of the PIC to the cavity producers. In the end about half of the required total amount of 800 had to be purchased due to delays in string assembly.

G. Cold rf performance test
Upon arrival at DESY every cavity underwent an incoming inspection to verify conformity with the specifications, which included mechanical, electrical and vacuum checks. The cold vertical rf acceptance tests followed a standardized procedure, which included the measurement of the unloaded Q-value (Q 0 ) as a function of the accelerating gradient E acc at 2K, as well as the frequencies of the fundamental modes.
Two aspects have to be emphasized with respect to the rf testing of fully equipped cavities. First, diagnostics are restricted: T-mapping is not possible on a cavity already integrated into the HT, and with the HOM antenna installed the pass-band mode measurement is limited to low power.
Second, in the past all cavities were vertically tested in CW mode without output lines connected to the HOM couplers, i.e. without HOM feedthroughs. (The HOM coupler feedthroughs were attached later during the assembly process). During the CW test, the heat load on the HOM couplers increases by a factor of 100 compared to EXFEL nominal operation. In order to avoid overheating, a pulse acceptance test with typically 5-20 s long pulses and an off time of approximately 50 s was used [26].

H. Performance acceptance criteria
The original acceptance criteria for the cold rf tests have been defined in [27]: (a) the maximum gradient > 26 MV=m (b) the unloaded quality factor Q 0 ≥ 10 10 (c) the radiation (X-ray) level < 10 −2 mGy= min.
The gradient of 26 MV=m gives a margin of 10% compared to the designed operation gradient (23.6 MV=m at Q 0 ≥ 10 10 ) of the EXFEL [2]. Practically, cavity performance was quantified by the use of a so-called usable gradient, defined as the lowest value of: (i) the quench gradient; (ii) E acc at which Q 0 drops below 10 10 (if at all); (iii) E acc at which the one or both x-ray signals exceed the specified threshold (if at all).
For the last criterion, two x-ray monitors located above and below the vertically tested cavity were used. The cross-calibrated thresholds are 0.01 mGy= min and 0.12 mGy= min respectively. In general, if the usable gradient was considered too low, then a case-by-case decision was taken on the remedial measures to apply to improve its performance (see Sec. X).
As noted, the original acceptance criterion for the usable gradient was set to E acc ≥ 26 MV=m. However, during production at the end of project, because of the relatively high average performance, a decision was made to reduce the threshold to ≥20 MV=m, resulting in a reduced number of retreatments and associated additional vertical rf tests. It should be noted that the developed for EXFEL waveguide distribution with asymmetric shunt tees [28] allows operation of the cavities in cryomodules at maximum gradient.

A. Vendor selection and qualification
During the preparation phase, industry was asked to provide studies regarding large-scale production and prototyping. The companies had to show that they were capable of producing the HT, the SC resonators and the cavity semifinished parts (SFP) (mainly of high purity niobium with residual resistivity ratio (RRR) of 300, see Table I). Furthermore, they were required to demonstrate that they were able to fulfill the necessary pressure equipment regulations and certification concerning pressure loaded parts.
As a result of this process, the companies RI and EZ were qualified for the cavity production; RI and Henkel Lohnelektropolitur in Germany were qualified for the first step of EP (main EP); Ningxia OTIC (China), Tokyo Denkai (Japan), Plansee (Austria) and Heraeus (Germany) were qualified as suppliers of niobium and niobium-titanium alloy SFP. C.S.C. S.p.A. and EZ (both companies located at Schio, Italy), together with Graeven Metalltechnik in Germany, were qualified for the fabrication of the HT and its accessories. For the production of the remaining parts (HOM coupler antennas, pick-up antennas, power couplers and accessories), various collaboration partners in the EXFEL consortium were qualified.

B. Contracting
The specification documents for serial production were reviewed in a Production Readiness Review meeting in April 2009, with the participation of invited international experts. After reviewing, the specification documents were sent to the qualified vendors as a basis for quotations. The call for tender was based on the production of 800 serial cavities, 8 dummy cavities (DCV), 8 reference cavities (RCV), and 24 so-called ILC-HiGrade (HG) cavities, as well as options for fractions of those amounts. DCV and RCV were used for qualification of the infrastructure at the company sites (see below). The HG cavities were foreseen for R&D purposes at DESY, and during production were also used as a tool for QC [13].
Two possibilities were considered for specifying the contracts: "build to print" and "performance guarantee." For the second option, formal acceptance of a cavity from the company would have depended on it achieving the specified rf performance in the vertical acceptance test (i.e. achieving the specified usable gradient). Given their lack of experience with the final surface preparation, both companies added a significant "risk premium" to the quotes provided. For this reason, the option ("build to print") was adopted. In lieu of an ultimate performance guarantees, the companies were obliged to follow and document every step of the fabrication process as defined by DESY. As a consequence, DESY accepted the cavity irrespective of its final performance, providing all mandatory fabrication steps were completed in spec. and formally documented.
After several negotiation steps, contracts were placed with both companies for one-half of 560 of the 800 serial cavities in September 2010. Each of the companies had to deliver 280 serial cavities, 12 HG cavities, 4 DCV and 4 RCV. The remaining total order of 240 cavities was initially held back as a competitive incentive. This so-called "cavity option" was intended to be placed only after the companies had demonstrated the required infrastructure and specified production rates of four cavities per week. After successfully setting up the infrastructure the "cavity option" was allocated at the beginning of 2013 to both companies in equal amounts (120 cavities to EZ and 120 cavities to RI).
As part of the tender, the choice "Final EP" or "Flash BCP" for final surface process step was left up to the companies. As a result, EZ chose "Flash BCP", while RI chose "Final EP".
The HT were fabricated at Ettore Zanon S.p.A. and C.S.C. S.p.A. in Schio, Italy. While EZ produced HT for their own cavity production, DESY ordered and provided all HT for RI cavity fabrication.
DESY has taken over the procurement, QC, documentation, and shipment of material to producers of the EXFEL cavities. Four qualified companies produced about 25,000 SFP from high-purity niobium and niobium-titanium alloy in a period of three years. Close contacts with the manufacturers contributed to successful production. After scanning of all niobium sheets for half-cells by eddy current at DESY, it was stated that in order to define the rf side, 26% of the sheets were scanned on both sides. About 2% of the material was rejected. Most of the eddy current signals indicated foreign material inclusions or demonstrated topographical flaws. Material inspection based on eddy current scanning of the delivered material, and supplemented by nondestructive detailed investigation like x-ray element analysis and digital microscopy, allowed to avoid diminished rf performance of cavities caused by material. The details of QC are described in [14].
Discussion of the manufacturing cost is beyond the scope of this report. However, two general points are worth emphasizing: First, only by not demanding a performance guarantee from the companies was an acceptable cost for the project achieved; second, a scenario of sharing the work between the companies for cost reduction was proposed, but was rejected by the companies: The concept was that Nb55%Ti Rod Nb-55%Ti, D147 mm, length 1000 mm, forged, annealed, surface turned Flanges one company would do the mechanical fabrication, while the other the surface treatment. However, each company wanted to independently implement the complete production cycle, preferring to develop the necessary infrastructure and expertise required for the full production technology. This eventually proved beneficial, since the advantage for the project of having two full and independent production facilities in terms of more reliable time schedule and shipment rate was evident.
The possibility to organize mutual support between the cavity manufacturers was very helpful for the production. For instance, during the initial production phase RI performed the main EP for EZ, until EZ had established its own EP facility. Similarly, EZ supported RI in removal of the HT, until RI had qualified its own procedure.

IV. PRESSURE EQUIPMENT DIRECTIVE
The cavity integrated in the HT is a pressure loaded part, subjected to the danger of implosion and explosion and as such has to follow the Pressure Equipment Directive of the European Commission-(PED 97/23/EC).
The required certification was done by a contracted notified body (TUEV NORD) and DESY according to the conformity evaluation procedure (Table II) for modules B1 (EC design-examination), B (EC type examination) and F (EC product verification) [19]. Each of these steps will be discussed below.

A. EC design examination
A finite element analysis (FEM) was performed to localize the position of the highest stress in the structure. It turned out that the maximum stress is located at the welding connection of the stiffening ring to the half-cell and at the welding connection of the bellow unit to the tank tube. However, the calculated stresses are still below the maximum allowed for the material.
All fabrication drawings were checked by the notified body, according to DIN EN 13445-unfired pressure vessels. Material suppliers for the pressure bearing parts were qualified according to the PED, annex I, paragraph 4.3. All materials have been confirmed by a 3.1 or 3.2 material certificate according to EN 10204. Particular material appraisals (PMA) for niobium with RRR of 40 and 300, titanium grade 1 and 2 and niobium-titanium alloy were obtained from the notified body.

B. EC type examination
Qualification of welding processes and further PED relevant processes (annealing, deep drawing, and forming) was performed at the companies. Necessary witness and hold points in the production have been determined in the Quality Control Plan by the notified body. The company personnel responsible for the welding and testing were verified with respect to valid qualification and certification.
An important stage of this activity was the qualification of production by a so-called test piece. The test piece was composed of two cavity-cells with a helium vessel, without the non-pressure-loaded end groups, and represented all pressure bearing parts and welding connections. It had to be built using exactly the same manufacturing methods, welding parameters and possible repair procedures that would be later used in the series production. Two test pieces per company, one for each electron beam welding (EBW) machine, have been produced and successfully qualified in accordance with the ISO 15613. Visual inspection, liquid penetrant tests, micrographs, hardness measurements and measurement of the wall thickness have been performed on each welding seam of the test piece. Finally, the production qualification was done on the first eight serial cavities (called pre-series cavities, or PCV). The notified body was present during all the above production steps.

C. EC product verification
After the successfully passing modules B1 and B examinations, a more simplified procedure (module F) could be applied to serial production, consisting mainly of visual inspections, control of documents and a final pressure test for each cavity (see Table III). The notified body advised and tracked the process.

D. Traceability
Adequate procedures were established and maintained to provide traceability from the raw material used for the pressure loaded components, through the semifinished products, and finally up to the completed cavity with HT. The process was realized and ensured by using an Engineering Data Management System (EDMS).

E. Exchange and repair of helium service pipes (HSP)
The longitudinally-welded HSP ( Fig. 1(a)) was fabricated from titanium (grade 2) by two PED qualified HT suppliers (C.S.C. S.p.A. and EZ). During the first assemblies of cavities into strings, the x-ray examination of the HSP connecting welds showed that the longitudinal welds contained pores larger than allowed by the DESY specification (no pores > 0.4 mm) and occasionally also out of PED requirements (no pores > 0.6 mm). To remedy this, it was decided to replace the extremities of the service pipes by seamless titanium tubes, both on "naked" helium tanks as well as on tanks with cavities already installed. Rf tests of cavities after cutting and welding the service pipes showed that cavities did not change their rf performance [29]. In addition, approximately 180 HSP could be exchanged completely by seamless pipes during the stillrunning HT production, while all other HSP needed a replacement of tube parts. Pores located in the region between the exchanged extremities (chimney area) were locally repaired by EZ.

A. Monitoring of the production
The "build to print" fabrication scenario requires no performance guarantee from the companies, but meant that the cavities had to be fabricated and treated in strict accordance to the EXFEL specifications. This requires a very precise specification and a comprehensive monitoring of the production.
The main monitoring actions consisted of work according a well-defined Quality Control Plan, internal Quality Assurance (QA) and Quality Management system at the companies, as well as Non-Conformity Reports (NCR) and monthly progress reports provided by the companies. In addition, a "Project Meeting" took place at the company sites (approximately once per month), with in-between visits to the companies by the DESY/INFN expert teams. Tele-and videoconferences were held as needed to cover all problems and routine information exchange in a timely manner.
The initial intention to deploy an inspector permanently in attendance at the companies was later waived. It was impossible to recruit experts able to cover all comprehensive issues of cavity production. Frequent visits of DESY and INFN experts (depending on production progress and quality) proved to be more fruitful, for which several teams responsible for general coordination, material, mechanical cavity fabrication, HT and PED issues, treatment, rf, vacuum, quality management and documentation were created.
In order to insure the "build to print" production DESY/ INFN executed an "external QC" or production control procedure in addition to the internal QA processes of the manufacturers.
Formal acceptance for delivery of a fully-equipped cavity was split according to the EXFEL specification into three acceptance levels AL (AL1-AL3) that define the contractual hold points (see also Fig. 2): (i) AL1: release the cavity for surface treatment after the mechanical fabrication without HT. (ii) AL2: release the cavity after dressing it with HT for further treatment. (iii) With AL3 the XFEL expert team gives release for shipment to DESY for the vertical rf test.
It was decided that the EDMS in use at DESY for several years, would be applied for EXFEL as a central repository of all engineering information. This included all documentation of the manufacturing process from niobium sheets to the cavity integrated in the helium tank [18,30,31].
An automated and completely paperless transfer of the quality management documents was developed and implemented. Documents/data from the companies' Enterprise Resource Planning systems were automatically transferred to the DESY EDMS, after which an automated analysis gathered statistics and usage as reference for the measurements during cavity incoming inspection at DESY. Key relevant data were then selected from EDMS and transferred to the EXFEL database [32]. The cavity manufacturers only had access to documents and data relevant to them. The acceptance procedure for AL1 is presented as an example in Fig. 3.
In order to keep the series fabrication and treatment processes running smoothly it was necessary to define a practical limit for the review duration for the AL release procedures. A limit of two working days was mutually agreed upon between the vendors and DESY/INFN.
According to the technical specification, a NCR had to be generated by the cavity supplier whenever a deviation from the specification occurred. The report had to include a detailed description of the nonconformity as well as a proposal for its correction. Cavity suppliers used their own templates to create a single NCR. This report had to be uploaded to the DESY EDMS by a tool provided by DESY, at which point the automated workflow shown in Fig. 4 for the cavity nonconformity life-cycle was initiated.
The run-time of the cavity nonconformity life-cycle was limited to five working days, after which a self-release took place. After review by the designated experts ("Approver"), the NCR (and specifically the proposed remedial action) is either accepted or rejected, and the decision automatically sent to the company via email by the DESY EDMS.
The NCR categories which typically appeared during EXFEL cavity production are listed in Table IV.

A. Experience
The described documentation methods, acceptance release processes and NCR life-cycle were successfully used for the cavity fabrication and treatment. More than 80,000 quality management documents were transferred from the cavity manufacturers to DESY. Automated release processes helped to fulfill the production schedule requirements and maintained a continuous flow of cavities to DESY. Over the whole production only eight cavities had to be rejected and replaced by the vendors.
EXFEL experience has shown that for a cavity series production of this scale, the need for such automated processes supported by an EDMS is mandatory to manage the documentation in a reliable way. It provided a fast, paperless and well-structured storage of documents and traceability to manage the QC procedures. It was indispensable for QC and QA, especially when the rules regarding traceability of PED have to be followed.
The disadvantage of using a system like DESY's EDMS is that it needs an extensive development and maintenance support team, ensuring its availability for the entire project duration. The required effort for the preparation and set-up of the documentation as well as the release processes were underestimated at the beginning of the project, resulting in considerable ramp-up and parallel development work during the early phases of production.

B. Exchange of information
In order to guarantee the exchange of all technical information between cavity producer and the DESY/ INFN team it was decided to use a request tracking system. The chosen software (Request Tracker), an open-source tracking system (http://bestpractical.com), was customized and supported by DESY's IT group.
The advantage of using such software was that the cavity producer had to send e-mails to only one single address (so-called "queue"). The incoming message created a ticket with a unique ticket number, stored the incoming message and distributed it via e-mail to all members of the queue.
The experience with the request tracking system has been very positive; the exchange of information was significantly simplified, while the message storage and search functions allowed the requests to be easily traced. Furthermore, it was guaranteed that each request could be answered in a reasonable time.

VI. SETUP AND QUALIFICATION OF THE INFRASTRUCTURE
An essential part of the work was dedicated to upgrading the existing infrastructure and also creating new infrastructure for cavity production at both companies [15][16][17]. The infrastructure improvements were partially implemented through equipment developed and constructed by DESY/INFN and placed at the companies, and partially through technology transfer of techniques (see Table V).
Infrastructure at Ri. Some infrastructure, like the EP facility (qualified for main EP), the BCP in open basins, EBW machines, inside control of cavity welding seam, 3Ddimensional control, and equipment for HT integration, were already available before contracting the EXFEL cavities. The following infrastructure had to be installed: ISO 4 cleanroom of 120 m 2 with two HPR stands; slow pumping/slow venting vacuum units (allowing particle free pump down and venting of ultra-high vacuum (UHV) systems [33]); ethanol rinsing facility; drying and assembly areas integrated in this cleanroom. The automated wash  Cavity mechanical damage.
In several cases the cavities mechanically damaged from outside. Repaired by individual procedure by replacement of some subcomponents. 7 Deviation from vacuum requirements. The rule that sum of the partial pressures of masses above mass 45 has to be less than 10 −3 of the total pressure, was in many cases not kept. The tolerable partial pressure relaxed during production on a case by case basis. 8 800°C annealing errors. Applying a wrong venting, wrong annealing parameters. 5 affected cavities rejected due to PED regulation. 9 Errors in transfer measurement (measurement of the reference coordinates that are used for the transfer of the electrical axis of the cavity and the alignment of the individual cavities in the module string).
Process control intensified. Data examination improved. Personnel trained. Repeated measurement of the reference coordinates.
10 Damage of the bolt and nuts during assembly for shipment.
Bolt and nuts of blind flanges damaged several times. Cavity sent back for repeat of the final HPR and reassembly. Improvement of tools and usage of adequate torque wrenches applied. 11 Cold and warm leaks.
Cavities showing leaks at room temperature and leaks in cold test with indications of problems at sealing areas were returned for reassembly and repeat of the final HPR. The repair of all other leaks at 2 K done at DESY. 12 Total Organic Carbon (TOC) values exceeded the spec requirements.
These have been accepted after discovery of the reason and source of the high TOC, if considered harmless for the cavities. 13 Maximum E acc below 5 MV=m. Considered as a deviation from spec requirements during production. Asked for repeat of the final HPR and reassembly. system for bringing cavities into the clean room, ultra-sonic (US), ultra-pure water (UPW) rinsing basins and FMS integration are also located in the ISO 4 area (Fig. 5).
A new high-vacuum oven for 800°C annealing of niobium cavities, three 120°C baking ovens and a safety area for pressure tests were also installed. The EP system was upgraded for "final EP" and additionally qualified.
A new ISO 7 cleanroom of about 15 m 2 was set up next to the EP facility, where US cleaning with detergent, UPW rinsing equipment, and the low-temperature drying oven with ISO 4 air quality were also installed (Fig. 6).
Special clean transport boxes were designed and built for transfer of cavities between the different buildings.
Infrastructure at EZ. For the EXFEL project the company set up a refurbished manufacturing hall with ISO 10 air quality, where a completely new infrastructure for cavity preparation was created. A 160-m 2 ISO 4 cleanroom together with a further 220-m 2 ISO 7 clean room was set up. Semiautomatic US cleaning and UPW rinsing line, nitrogen gas lines, two BCP closed loops (one for the BCP flash chemistry and one for outside etching) were also installed (Fig. 7).
UPW rinsing and ethanol rinsing facility were integrated into the ISO 7 area. In the ISO 4 area, slow pumping/slow venting vacuum unit and two HPR stands are incorporated. FMS integration, cavity drying after HPR, assembly of accessories and vacuum checks are also performed in the ISO 4 clean room. A new 800°C annealing oven (Fig. 8), four 120°C baking ovens, the company's second EBW machine, a semiautomatic chemical plant for parts etching before EB welding and a local BCP bench for accessory etching were set up inside the hall.
In addition, the hall also housed the new equipment for HT integration, 3D-dimensional control, borescope system for inside surface control, and the safety room for pressure tests. During the start-up of the cavity preparation, a new EP facility for "main EP" was designed in-house and set up [34].
Infrastructure qualification. The DCV and RCV cavities were used for the commissioning and subsequent qualification of the new infrastructure. The DCV were used at the companies for operator training, mechanical testing of devices, process parameter adaptation, infrastructure setup and ramp-up, final treatment tests, tuning tests, etc. The DCV remained with the companies up to the end of the serial production and were periodically used for infrastructure checks (e.g. after repair).
After infrastructure setup using the DCV, five steps were defined for qualification of the surface treatment infrastructure using the RCV: (1). Transportation between DESY and company. (2). Slow pumping/slow venting including leak check and rest-gas analysis (RGA). After initial treatment at DESY, the RCVof both companies reached an accelerating gradient E acc ¼ 30-35 MV=m. The RCV were then used for stepwise qualification of new infrastructure. After each step, the respective RCV was sent to DESY for cold rf testing. Each qualification step was repeated in the case of failure.
The EP facility at EZ was qualified separately. After verifying the EP facility by one RCV, two serial cavities passed the complete treatment cycle of BCP flash with main EP done at EZ. Both cavities reached E acc above 30 MV=m without field emission. The 800°C furnaces were qualified using niobium samples.
After successful completion of the qualification procedures, the RCV remained at the companies for infrastructure requalification (e.g. after some repair action, or before restarting series production).

VII. RAMP UP OF THE PRODUCTION AND DELIVERY RATE
The logistics and preparation sequences foreseen in the specification were adapted and optimized at both companies according to the available infrastructure [15][16][17]. Ramp up of serial production in both companies was made using the PCV; these cavities were the first to pass through the complete cavity production process and be equipped with all accessories.
Delivery of fully-equipped cavities and subsequent preparation for the cold rf test started at the end of 2012. Production of all 800 serial cavities and the 24 HG cavities was completed in the 4th quarter of 2015. Figure 9 shows the integral of weekly cavity shipments of RI and EZ. On average the production rate was close to 3 cavities per week per company, although for several long periods the specified production rate of 4 cavities per week per company was successfully achieved.
The contracts were allocated to both companies simultaneously; however, RI was only able to start serial production three months after EZ due to delays in the set up and qualification of the new infrastructure.

VIII. RF ISSUES
The specified mechanical tolerances for the shape accuracy of the cells (AE0.2 mm) are rather tight due to several reasons (e.g. spring back during deep drawing, shape distortion during iris and equator welding, shape distortion during welding of the stiffening ring and of the connecting flange).
In addition, achieving the cell-shape tolerances does not necessarily guarantee that the primary cavity requirements will be met (i.e. cavity length, fundamental-mode frequency, flatness of field distribution and cavity straightness).
To address these tolerance issues, a length adjustment procedure was applied which allowed all the primary conditions to be consistently met. The procedure was originally developed and qualified during the cavity production for FLASH [35], and takes advantage of the fact that the frequency shift due to a change in cell volume has the opposite sign for the high magnetic-field and high electric-field regions. The process consists of the following steps: (1) Frequency measurement on half cell (optional).
(2) Frequency measurement on dumbbell and end groups. The rf measurements are very sensitive to mechanical deviations and do not require mechanical contact on the inner cavity surface [35,36]. The influence of surface layer removal by the treatment was taken into account.
Three semiautomatic rf measurement machines (HAZEMEMA) [37] were designed and built at DESY to provide the measurements for a large volume of cavity parts. The machine performed an easy load of the parts and decreased the test duration considerably (up to 80%) as FIG. 9. Integral of weekly shipment of EXFEL cavities to DESY for the rf cold test. compared to manual operation. Both RI and EZ were each given one HAZEMEMA machine for the series cavity production.
The adjustment of the main parameters (length and frequency) was successfully done for the complete EXFEL cavity production. Figures 10 and 11 show the example of the length and fundamental-mode frequency measurements respectively for cavities produced at RI [38]. Similar results are achieved by EZ.
During treatment the cavities had to be tuned, their frequency and length being measured several times before being integrated into the HT. Semiautomatic cavity tuning machines (CTM) for 9-cell TESLA-type cavities were designed and fabricated by a collaboration of DESY, FNAL and KEK [39][40]. CTM were installed at both RI and EZ and successfully operated over the entire production period. These machines reduced the tuning time from 2 days down to as low as 4 hours, and proved decisive for insuring the required production rate for the EXFEL project. During the production period, DESY was responsible for reliable operation, maintenance and repair of the HAZEMEMA and CTM located at the companies.
Another important rf aspect of the EXFEL cavity production was the HOM damping efficiency. The HOM parameters for the EXFEL cavities are described in [41]. During EXFEL cavity production the value of Q load for the strongest modes was analyzed (TE111, TM110 and TM011, for details see [42]). Based on Tesla Test Facility (TTF) experience Q load values for these HOM should be lower than 10 5 . The first two dipole modes for all EXFEL cavities (TM111 and TM110) always fulfilled the Q load < 10 5 original requirements. However, the second monopole TM011 proved problematic. This mode does not cause a critical beam perturbation but does increases the cryogenic losses in the accelerator. The zero-mode for TM011 has the highest frequency (the ninth peak in the spectra, referred to as TM011_9). The measurement results of Q load (TM011_9) for EXFEL cavities under cryogenic conditions are presented in Figs. 12 and 13 [38]. Almost all cavities from the company EZ achieved values of ≤10 5 .
Only one wrongly trimmed cavity demonstrated significant deviations. The shape of this cavity was corrected by reducing the equator diameter and subsequent retuning (for more details see [43]).
It was observed that the damping of the second monopole mode (TM011) for RI cavities showed the largest variation, which was sometimes up to 2-3 times lower than the originally allowed limit (Fig. 13). It was determined that this TM011-damping degradation was caused by cavity geometry deviation within specified mechanical tolerances. Some analysis of cavity shape deviation relative required profile can be found in [44][45]. Subsequent corrections of the trimming procedure proposed by DESY remarkably improved the HOM suppression in the second half of production (blue line in Fig. 13) [38].

IX. OPTICAL INSPECTION: FLAWS IN THE EXFEL CAVITIES AND THEIR CORRECTION MEASURES
Extensive diagnostics and detailed surface studies of the serial EXFEL cavities was very restricted due to the tight production and cold rf testing schedules, as well as the presence of the HT and installed HOM antennas. As a result, the only available diagnostic method was surface inspection. The inner surface of all cavities was visually inspected by the companies after the EBW operation. A detailed analysis of surface quality was applied at DESY for some suspicious and bad-performing cavities. Around 80 EXFEL and 24 HG cavities were investigated by the high-resolution optical system OBACHT, many of which were also investigated by replica surface profilometry [46]. These inspections were carried out after at least one vertical acceptance test. The mechanical extraction of a defect area from a cavity for detailed investigation was never done in order to not sacrifice a cavity.
Several types of surface flaws were identified: welding errors; etching irregularities; foreign material inclusions; mechanical damage; and surface pollution. Details are presented and discussed in [47]; in the following we give a short overview.
Such types of fabrication errors were unfortunately much more prevalent at EZ than RI. However, EZ invested considerable effort in developing successful repair techniques. A local grinding machine constructed at EZ with the support of INFN and DESY was particularly beneficial [48]. The machine provided efficient grinding of weld spatter and other weld irregularities in the equator regions as well as scratches in the iris areas. (i) Welding errors, for example incomplete penetration, rough welding seam surface, rough overlapping, were predominantly observed at the beginning of the production, but were later reduced by an optimization of the welding parameters. (ii) Six cases of burned-through holes caused by the electron beam occurred. A procedure for fixing such holes was developed by EZ, qualified by the notified body and successfully applied. Experience shows that accelerating gradients up to 35 MV=m are reachable after proper repair. The repair was especially successful when made before taking the cavity out of the EBW chamber. (iii) weld spatter (Fig. 14), observed in 20 cavities, appeared either because of insufficient stability of the EBW parameters or because of some inadequacy during preparation of subassembly parts for welding (deviation from tolerances or from required cleanliness). Performance of up to 35 MV=m could be reached by locally grinding these defects away (followed by a subsequent 20 μm BCP, HPR and baking in the standard way). (iv) Etching erosion had already been seen on the prototype cavities produced at EZ. Several serial EXFEL cavities demonstrated similar surface flaws and show quenches (breakdown, BD) at rather low gradients without field emission (FE) (E acc of 11 to 18 MV=m).
Quench at the etching erosion location (Fig. 15) was confirmed for one cavity by the use of a second sound (SS) set up for dressed cavities [49]. Efforts to cure this type of defect on a prototype cavity (by additional EP or post purification with titanium [50]) had no success. Two serial cavities with similar defects were ground locally and treated with 20 μm BCP followed by 20 μm EP; only negligible improvement of final performance was obtained.
Unfortunately, the phenomenon is not yet fully understood. Our hypothesis is that the defects are most probably caused by foreign material contamination occurring before or during EB welding. The localized contamination then causes differential etching during the EP process, resulting in a surface with little bumps and etching holes (surface with etching erosion). (i) Foreign material inclusions have been suspected on the inner surface of several EXFEL cavities. As mentioned above, all niobium sheets for EXFEL cavities have been eddy-current scanned to exclude the presence of inclusions relevant for EXFEL performance [14]. However, inclusions can also be introduced during one of the cavity production steps, such as deep drawing, clamping for machining or during assembly for EBW. Presence of foreign material inclusions in cavities could not be always checked: however, the images of the resulting defects were very similar to ones discovered on niobium sheets, as well as those observed on cut-outs of some prototype cavities. These observations, together with rather low quench fields (typically ∼15 MV=m) gave us confidence that the root cause was the presence of an inclusions on the surface. A dedicated X-Ray Fluorescence (XRF) diagnostic tool able to analyse the material composition of the cavity inner surface was developed and used [51], albeit only toward the end of the production. In one case iron was identified as the inclusion (for details see [51]). In several cases it was very plausibly observed that defects like foreign material inclusions located close to iris caused radiation. (ii) Scratches on the iris (e.g. Fig. 16) are caused by contacts or collisions of some parts with the iris surface during assembly for EBW, removal of the cathode for EP or by HPR lance touch. The main impact of such defects on the cavity performance is excessive x-ray radiation due to the electron field emission, enhanced on the resulting sharp edges and spikes in the scratched regions. The scratches were successfully removed by local grinding. (iii) In a few cases OBACHT could not find any plausible reason for limitations leading to quenches at relatively low acceleration gradients (below 20 MV=m). It can only be speculated that in these cases the niobium had reduced purity in some specific location in the welding seam or in the heat affected zone [50]. This can happen by inappropriate preparation of subassembly parts for EBW.

X. PERFORMANCE
The detailed statistical analysis of the cavity performance will be discussed elsewhere. Only the key results will be summarized here.
All cavities were vertically tested at 2K to their maximum achievable gradient, which was limited either by quench, by excessive FE (as measured by the x-ray monitors), by the available forward power, or by problems with the HOM coupler warming up [52][53][54][55].
Most of the cavities tested for the first time immediately after delivery ("as received") exceeded the specification requirement, as can be seen in Figs. 17 and 18, where the average values per month of the maximum and usable accelerating gradient E acc for cavities produced at RI and EZ are presented.
It was very desirable to have the "as received" cavity performance data as quickly as possible after delivery, to provide timely feedback to the companies so that corrective action could be quickly started when needed. Unfortunately, the time between the end of mechanical production and cavity shipment was normally about ten weeks. This meant that before any correction could be applied, several cavities with potentially similar faults would be delivered to DESY. This aspect is visible in Figs. 17 and 18; for example, the sudden decrease of both maximum and usable gradients at RI below the EXFEL specification in summer 2014, caused by enhanced FE, required almost two months for correction. In the meantime, many cavities were unfortunately affected.
Similar reduction of performance in April-May 2014, also caused by FE can be seen for EZ cavities. The last period of cavity production (autumn of 2014 to the end of 2015) was rather stable at EZ (Fig. 18) and still less stable at RI (Fig. 17).
Table VI summarizes the statistical analysis for "as received" cavities.
The table shows that the average usable E acc comfortably exceeds the average operational requirement for EXFEL (23.6 MV=m).
For RI the average usable gradient is 2.6 MV=m higher than for EZ. The results from RI clearly indicate that application of final EP provides stable behavior of Q 0 versus E acc without a pronounced high gradient Q-slope. Furthermore, the RI results showed that the presumed risk of disturbing the surface quality by the integration of a completely treated cavity into a HT without follow-up surface material removal was negligible. In comparison to "final EP" treated RI cavities, the "BCP Flash" treated EZ cavities demonstrated a stronger high-gradient Q-slope (clearly seen in the examples shown in Fig. 19). As a result, the usable gradients of EZ cavities were more frequently limited by Q 0 than the RI cavities, resulting in the observed lower average value.
It is known [1] that 120°C baking works better for EP than for BCP treated cavities. Our data shows that BCP removal of even a rather thin layer (approximately 10 μm) can be sufficient to destroy the EP surface structure and affect the E acc performance. 71.3% of the "as received" cavities were accepted for string assembly. The remaining 28.7% of cavities have to be split up in two categories [56]: Nonconformity after delivery from vendor. About 90 cavities showed a mechanical, vacuum, electrical or other nonconformity, which required a retreatment at DESY or at the vendor before the first vertical test. These cavities did not have an "as received" test.
Performance. Most of the low-performing cavities have been retreated in order to improve their performance. The predominant reason for retreatment was the excessive FE. The following retreatments procedures have been applied separate or in combination: HPR (six times high-pressure water rinsing and drying for 12 hours in the ISO 4 clean room) [57]. Additional HPR worked well as a main cure for high FE load, caused by some lack of cleanliness during different steps of preparation and assembly. On average the usable gradient was improved by approximately 8 MV=m after retreatment by HPR.
BCP, HPR and 120°C (chemical treatment with maximum removal of 10 μm by BCP, ultra-pure water rinsing and one time HPR; subsequently six times HPR and drying for 12 hours in the ISO 4 clean room, 120°C baking) [57]. The procedure works sometimes, when the simple HPR did not help.
Retreatment of cavity with HT reintegration: around 6% of the cavities from the production had defects that could  19. Examples of the Q 0 (E acc ) curves of some of the best cavities, either treated at RI using "EP final", or at EZ using "BCP flash." FIG. 21. Histograms and yield plots of the usable gradient in the "as received" and "after retreatment" test for EXFEL cavities. not be fixed by HPR rinsing and light BCP polishing, and required a more comprehensive repair.
Repair of a cavity integrated in a HT was very restricted. For layer removal greater than 10 μm a retuning is required, which could only be done after removing the HT. A special procedure of HT removal and refurbishment was developed, qualified, and successfully implemented at EZ. Grinding in combination with successive 20 μm BCP (removal of grinding residuals) and 20 μm EP (restoring of the inner surface finishing) followed by the standard cycle (10 μm BCP, HPR and 120°C baking) allowed recovery of the cavities. Usable gradients of up to 35 MV=m without field emission could be reached after repair of the defects.
A history of the cavity recovery can be seen in Fig. 20. Finally, after retreatment the E acc was notably improved (see histograms and yield plots in Fig. 21 and Table VII), with the average usable gradient for all cavities achieving 29.9 AE 5.2 MV=m. This is about 25% higher than the specification requirements.

XI. CONCLUDING REMARKS
The experience gained from the EXFEL cavity production has demonstrated the ability of industry not only to perform the mechanical fabrication of superconducting cavities, but also to implement the complete cavity preparation procedure required for high performance. A total of 800 series-production cavities dressed with HT and ready for cold rf acceptance test have been successfully manufactured within three years at two companies: RI Research Instruments (RI) and Ettore Zanon S.p.A. (EZ).
The industrial production demanded broad knowledge transfer from both DESY and INFN to the companies, and also needed investment in new infrastructure requiring extensive qualification. Beyond the standard series production, several cavity repair scenarios were effectively implemented at both companies.
The following aspects have been critically important for successful implementation of the EXFEL cavity production: (i) Varied tasks in the preparation phase (elaboration of detailed specification for required infrastructure and its qualification; clearly defining all steps of fabrication, surface treatment, assembly and shipment of fullyequipped cavities). (ii) Procurement strategy (vendor qualification, choice of "build to print" contracting, activating the competition between companies by implementing a "cavity option"). (iii) Concept of the quality management (creation of expert team, systematic monitoring of production, acceptance release processes and NCR life-cycle procedure, paperless documentation supported by EDMS). (iv) Setup and execution of the PED model.
(v) Tight cooperation with industry and comprehensive exchange of technical information between all participants. (vi) Precisely defined requirements for cavity performance and well-organized cold rf testing, allowing timely feedback to the companies.
The EXFEL industrial production represents the largest deployment of the TESLA superconducting technology to date, and the experience gained will provide the SRF community with invaluable input for projects such as LCLS-II and ESS. The high average performance of the cavities (≈30 MV=m with Q 0 ≥ 10 10 ) has also demonstrated the feasibility of the technology for possible future projects such as the ILC.