Challenger: Home for Christmas

Mark Yates examines the engineering behind Apollo, and highlights the continuing importance of science and R&D teams on the ground

FIFTY years ago, on 14 December 1972, the single fixed-thrust hypergolic engine on the ascent stage of Apollo 17’s lunar module Challenger burst into life, rocketing its two astronauts back into lunar orbit. This small 16 kN engine was the only method for all six Apollo lunar landing crews to depart the lunar surface and begin the journey back home. Each engine had to work perfectly. There were no backup options. The Grumman Apollo lunar module was indeed a truly unique vehicle. Each flight article was built by hand and each first flight would also be the last – they were designed for a single landing and takeoff. Many other firsts are rarely acknowledged, such as the use of a solid-state radar, a fly-by-wire control system in a rocket-powered craft that would launch and land vertically, and a novel rocket engine that could be partially throttled to control the rate of descent.

The ascent propulsion system

For many of us, December is a time for Christmas breaks when thoughts turn to returning home to be with our loved ones. This was also true for Apollo 17’s Gene Cernan and Jack Schmitt, who had just completed the final three days of lunar exploration within the Apollo programme. They had stayed longer and travelled further than the previous five landings and now, tired and grimy with Moon dust, it was time for Cernan and Schmitt to leave the surface, re-join their colleague Ron Evans who had spent the three days orbiting the Moon in the Command and Service module America, and head home.

In 1961, President Kennedy’s goal had been clear from the outset – put a man on the Moon by the end of the decade and return him safely to the Earth. A quick review of launch statistics at that time is sobering and emphasises just how bold Kennedy’s goal was. In the first three years of the American space effort, failure rates from 1958 to 1960 were 59%, 42% and 45% respectively. Worse still, these were rockets launched from Earth attended and fussed over by hundreds of engineers and technicians, something that was clearly going to be impossible with two people on the Moon’s surface, 386,243 km from home.
What was striking when researching this article was the tireless commitment between 1962–1972 from key members of the 450,000-strong Apollo team, not only to meet Kennedy’s goal but to maximise their chances of success, ever mindful that if they failed to do so, they and their colleagues would personally carry the burden of failure with them for the rest of their lives.

Design of the ascent propulsion system

NASA’s guidance requirements called for an engine design that would produce a modest 16 kN of thrust and ignite for a duration of 550 s (9.1 min). Once the start button was pushed it had to develop 90% of that rated thrust within 0.450 s, while transient pressures had to be less than 177% of operating pressures due to structural limitations of the lunar module ascent stage. Similarly, thrust at shutdown had to decay to 10% within 0.5 s.

Redundancy was the core engineering premise throughout Apollo. All phases of the round trip to the Moon were tolerant of some failure, from the mighty first-stage engines which lifted the Saturn V rocket off the pad, to the three parachutes that would slow the tiny spacecraft before splashdown in the Pacific Ocean. There were just a few instances where a redundant system was not feasible, none more so than for the ascent from the lunar surface, where weight restrictions constrained the design to just a single system. For this reason, the ascent propulsion system clearly stands out as the one critical component that kept everyone from the drafting board to the White House acutely aware of the risks involved.

Little surprise then that the engine’s weight and performance was a secondary consideration given the imperative that it simply had to work. Safety mattered more than performance. Period. It had to work flawlessly to reach lunar orbit and, once there, be capable of restarting if required to enable precise rendezvous with the orbiting mother ship and the only ticket back to Earth. As we saw in the design of the PLSS in TCE July 2019, simplicity and reliability go hand in glove, for what you do not have cannot go wrong.

Grumman would be awarded the contract for the Apollo lunar module in November 1962 and would in turn award a subcontract to Bell Aerospace for design of the ascent propulsion system.

The engineers and managers were acutely aware just how critical the ascent propulsion system was to crew survivability from the outset, prompting a strong emphasis on three key topics: design simplicity, robust safety factors, and confidence hard won through extensive ground testing. One notable proviso was that the ascent engine chosen for a flight would not be test-fired ahead of time.

A review of failures in propulsion systems identified active systems such as controls, solenoids and valves as being the main culprits, rather than the more passive components such as injectors and thrust chambers.

The Grumman leadership team championed a policy of “There are no random failures” with its workforce and those of its subcontractors, including Bell and later Rocketdyne. The point being that every failure encountered during a test would have a root cause that had to be found and corrected to avoid a recurrence. To the credit of all involved, the programme would be left with only 22 unexplained failures out of a catalogue of 14,247 from the entire lunar module programme.

Safety mattered more than performance. Period.

Bell did not design the engine from scratch, drawing on its experience with the proven but unmanned Agena rocket engine. Propellants with a track record in space were selected, with the purpose of spontaneously igniting when brought into contact with each other (hypergolic), thereby removing the additional complexity and mass of an ignition system. Ambient helium would pressurise the propellant tanks, removing the need for heavy forwarding pumps with their associated power and control systems. The required oxidiser-to-fuel mixture ratio was 1.6 to 1. The oxidiser was nitrogen tetroxide and the fuel was even quantities of hydrazine and unsymmetrical dimethylhydrazine (to reduce the freezing point of hydrazine in space).

An ablatively-cooled rocket engine was selected to eliminate intricate liquid cooling ducts and provide a passive thermal coating that was more tolerant of wider mixture ratios, propellant temperatures and chamber pressure operational limits. The engine was fixed to the ascent engine structure for reasons of simplicity; there would be no mechanical gimbals or actuators because the rocket could be steered using the 16 smaller reaction control system (RCS) engines mounted on four clusters on the ascent stage.

The control system would also be simplified, as a fixed thrust further removed valves, associated plumbing and potential leak paths. It was either on or off and had to work at the push of a single button or computer command; every time, on time. The only moving parts were the eight redundant ball valves that opened to introduce propellants into the injector and thrust chamber.

When asked what kept him awake at night ahead of the Moon landing, Neil Armstrong spoke of his concern about the reliability and performance of the little ascent engine, his engineering experience seeking areas where even further simplification might be possible. This simplification process did not stop at the engine but extended to the fuel storage and manifold systems. The initial design assumed four spherical titanium tanks, with one pair for oxidiser and the other for fuel in keeping with the aeronautical doctrine of maintaining design symmetry. Again, simplicity won the day, with a two-tank design being sanctioned once NASA had been reassured that the lunar module could still be flown in the event the ascent propulsion system performance was sub-optimal. Not only did this carry a significant weight saving in tanks, pipework, connections and structural supports, it also minimised potential leak paths.

The result was a relatively small, 106 kg engine, about 51 inches (1.3 m) high, no larger than your average outdoor domestic dustbin. Despite its small size it had to be very tough, because the first few minutes at launch from Earth would subject it to forces ten times greater than at lift-off from the Moon, stressing and testing every single joint, weld and wiring connection.

The engine primarily comprised four main components: a bi-propellant valve package; injector assembly; combustion chamber; and nozzle extension.
The bi-propellant valve assembly housed the only moving parts of the entire engine. Two independent parallel supply lines contained a fuel and oxidiser ball valve mounted on a common shaft, with a dedicated pilot valve and actuator. All eight ball valves actuated simultaneously, driven by four pilot valves using fuel as an activation fluid. Any fuel discharged was vented overboard through separate vents. Vented seals and shaft seals prevented any unwanted contact between the propellants that could have led to a spontaneous explosion.

The engine’s outer structural components comprised the combustion chamber, its mounting and a plastic assembly that included the peripheral nozzle extension. This plastic assembly had three layers, including an outer structural filament winding and an inner ablative lining. Sandwiched between these two was an insulating layer of sufficient thickness to keep the chamber skin temperature within design limits. The outer nozzle extension had a single ablative layer. 

Small engine, big problems

So why was such a simple design plagued with problems, making it one of the single biggest threats to overall lunar module delivery? Despite heroic efforts, these problems persisted for years and would only be banished uncomfortably late into the development and testing phases. The problem was one of combustion instability, which plagued rockets of all sizes. Even the giant F1 engines in the Saturn V first stage were not immune.

Combustion instability is a tangle of chemistry and acoustics. If the fuel and oxidiser are not carefully mixed, combustion instability occurs, whereby the rocket engine can behave like an organ pipe, inducing irregular combustion with a corresponding decrease in thrust and localised heating of regions in the engine bell. This in turn leads to premature failure, a predicament that no one wanted to face more than 386,000 km from home.

No matter what configuration was evaluated, the engineers at Bell just could not shake off the instability problem.

By 1967 the lunar module’s descent engine was ready, but prequalification had not even started for the ascent engine. Astronauts might be able to land on the Moon, but they would be unable to leave. NASA’s senior management had been closely monitoring the programme and wanted results.

In June 1967, NASA put out a proposal request for a backup programme for an alternative, verified injector design. Rocketdyne was one of three contractors shortlisted and began work in July – at its own risk – the moment its proposal was submitted. A month later, it had secured the contract. Its responsibilities did not stop there, also encompassing the overall integration of the engine’s existing components such as the valves and thrust chamber. It was also handed final responsibility for the entire engine system, including documents, drawings, specifications, maintenance manuals and running the sequence of design reviews.

Rocketdyne joined the effort with greater resources and a deep understanding of the instability problem, having encountered it during an earlier research and development programme. Then, its senior management team had not been content merely to accept the engineers’ claims at having fixed a similar problem, challenging them instead to go back and fully understand the physics behind its underlining causes.

With Rocketdyne now working alongside Bell, a highly-intensive work period followed, each day of the week starting with a stand-up meeting at 08:00, reporting on the results from the three shifts the previous day and reviewing the forward plan. Such intensity enabled a preliminary design review in the November and a critical design review at the end of the year. The beginning of 1968 would see the first full-duration altitude test, with the conclusion of design feasibility tests in March.

Rocketdyne’s design solution was a fixed-orifice injector plate assembly that used a baffle and a series of perimeter slots (acoustic cavities) to damp combustion disturbances. Made from aluminium, it weighed just over 4.5 kg and looked like a shower head, with 515 finely-machined holes arranged in precise radial patterns.

Engineers developed a methodology and test stand to initiate combustion instability at the injector, with a small explosive device that could be positioned and triggered at will inside a hardened steel chamber opened at one end. Although an effective methodology, it doubled the testing times, further compounding the time pressures felt by all. The new injector design was found to be stable, validated through hundreds of tests, and perfectly demonstrated in a final bomb test where the acoustic cavities were deliberately blocked: the instability demon resurfaced immediately.

One critical test finding that had not been anticipated was the susceptibility of the injector plate to even the smallest changes in the shape of each orifice, arising from either manufacture, accidental damage, or other means.

Given this new risk, the team now had to ensure that each of the 20 production engines would contain an identical copy of the injector. Electrical discharge drilling (where electrodes pierce the aluminium block with repeatable precision), left behind a surface free of blemishes or pitting. Electron beam welding permitted greater precision in targeting exact locations and avoided the risk of damage to any surrounding areas. Both techniques were innovative at that time and a tribute to the bold management decision to integrate emerging technologies, despite the additional risk to an unrelenting schedule.

Thrust chamber erosion problems persisted, with tests showing repeatable and localised erosion of the ablative material lining. The traditional fix here was symmetrical film cooling at a cost of engine performance, but for this little engine the engineers broke with convention and targeted only those areas that needed it, minimising any performance losses.

“Hard starts” (overpressure which can result in damage) from detonation when propellants come into contact in rocket engines are a constant problem to designers. The ascent propulsion system was no exception, but changing the diameter of the fuel line and adding a little extra volume at the end resolved the issue, an effective if inelegant solution that gave the oxidiser a 50 ms head start before the fuel arrived, resulting in smooth engine starts every time.

For Grumman and all its subcontractors, weight was a persistent headache throughout the lunar module’s development programme. Even the ascent propulsion system, so critical to crew safety, was not immune, with the team achieving an impressive reduction of 15% to the design, permitting an additional 15 kg of Moon rocks to be returned on each of the six Moon landings.

Testing, testing, testing

A conventional flight test programme was not a realistic option due to the costs involved with spaceflight. Instead, testing and development was Earth-based and began at the most fundamental level – single components, then on to the modular level and then to full-scale system tests.

Most of the prototype testing was conducted at the US Government’s White Sands Test Facility, New Mexico, where Grumman established a test site with 340 people. The ascent stage test rig was constructed from aluminium and incorporated the ascent propulsion system and reaction control systems (RCS) within a crossover capability.

Testing was the key to demonstrating high engine reliability. It was not confined merely to reproducing normal flight conditions but had to span a comprehensive array of failure modes, operating conditions, and variations in injection pressures. Fire-in-the-hole (FITH) starts simulated lunar launch conditions and evaluated protective heat shields, confirming no structural damage or adverse effects on engine performance. These tests would also be repeated in a vacuum chamber to better represent actual flight conditions. When Rocketdyne joined the effort, additional test sites at the Santa Susana Field Laboratory, and critical resources such as diagnostics were made available.

A conventional flight test programme was not a realistic option due to the costs involved with spaceflight. Instead, testing and development was Earth-based and began at the most fundamental level – single components then on to modular level and then full-scale

Farewell Moon

The precise timing and duration of ignition was fundamental. A precondition of any orbital rendezvous is that lift-off can only occur at precise times, or “windows”, to ensure that both spacecrafts can synchronise their orbits and dock successfully. If the ascent propulsion system did not fire, there were alternate windows for lift-off, permitting time to evaluate the problem and refer to numerous contingency checklists and procedures. Whilst the total loss of pressurising helium or propellant would have been catastrophic, key components such as valves, guillotines and pyrotechnics could be armed and activated from a wiring network connected to independent batteries.

In the minutes before firing the ascent engine, the astronauts manually armed explosive valves to pressurise the propellant tanks, one at a time, with high pressure helium. Mission Control in Houston monitored their telemetry, and any sign of a leak would initiate an immediate launch to mitigate any further losses. Crews would wear their spacesuits, to mitigate the risk should the pressure hull be compromised due to the pressure wave at ignition. This must have provided some comfort, as the top of the engine protruded through the floor of the ascent-stage pressure cabin.

Next came direction from Houston on which of the two guidance computers had control and that the updated trajectory information loaded into its memory checked out correctly.

Physically arming the ascent engine allowed the eight ball valves in the bi-propellant package to open, with the computer displaying a flashing “VERB 99”, signalling all was well. All that remained was for the commander to manually depress the “Proceed” button.

The first firing of the ascent engine would occur as a fire-in-the-hole, while the ascent and descent stages remain mated, although no longer mechanically secured to each other. Manually depressing the “Abort Stage” switch separated the ascent stage from the descent stage via a carefully-orchestrated pyrotechnic firing, using miniature guillotines and explosive bolts to sever interconnecting cables, plumbing and the four structural supports.

Crews departing the lunar surface were surprised at the quantity of debris and the flat trajectory it followed over significant distances, thanks to the low gravity and absence of air resistance. Ascent felt smooth, as if they were riding a high-speed elevator, whilst through their helmets it sounded like the wind whistling. The steep angle of ascent provided a bird’s-eye view of the lunar surface which they had explored so briefly. But there was no time for sightseeing as two pairs of eyes were intent on monitoring engine performance, burn duration and the ascent stage’s ever-increasing velocity. Only once sufficient velocity (660 km/h) had been attained would the computer shut down the engine and the intricate process of orbital rendezvous begin.

Both the single unmanned lunar module test flight and all eight manned lunar module flights performed within the predicted ranges; the only anomaly being encountered on Apollo 9, the first manned test flight in the relative safety of Earth orbit. This arose from a failure of the primary system’s helium regulator, brought about by contamination when a component was swapped out in vehicle checkout prior to launch. The flight of Apollo 13 would push the capabilities of the lunar module and its larger, throttleable descent engine far beyond what it had originally been designed for. The ascent engine would not be used in this configuration, as the lunar module’s lifeboat mode required the batteries and consumables in the descent stage be used right up to the point of jettison before atmospheric re-entry.

In the event of Moon disaster

Despite herculean efforts to ensure the crew’s safe return from the lunar surface, NASA was prepared for the worst. The third crew member, circling 100 km above the lunar surface every 60 minutes, would train for a solo return to Earth. Michael Collins would refer to it as his “secret terror”.

For Apollo 11, (then) President Nixon had a carefully-crafted speech ready to read to the world. It would have been little comfort to the two stranded explorers but explained that: “Fate has ordained that the men who went to the Moon to explore in peace will stay on the Moon to rest in peace.” At the end of the televised Presidential address and once NASA had severed communication with the crew, a clergyman would have conducted the same procedure as a burial at sea, commending their souls to “the deepest of the deep” and concluding with the Lord’s Prayer.

Nixon was nervous about potential failure during later missions, even influencing the launch of the Apollo 17 mission to avoid it interfering with his re-election.

Legacy

So, what of the legacy of the ascent propulsion system? Future missions to the Moon, Mars and beyond will distil the lessons of Apollo, carefully documented in their series of technical notes; lessons learned, written for a future which has still to arrive more than 50 years later. The crew debriefings, carefully transcribed over hundreds of pages, provide a first-hand account of the operational challenges and anomalies that were encountered and overcome.

These documents are of tremendous value to the engineers of today designing the spacecraft and rocket engines of the future. Aside from their technical content, they represent something that those Apollo engineers could only dream of – a blueprint to show how it can be done.

But documents can only tell us so much, so we need to turn our focus to the human and not the hardware. 

The flawless flight performance of the ascent propulsion system is quiet testament to the vigilance and leadership of the teams involved. Their calibre, tenacity and work effort obviously played a major part, but it started at the top, in this case the founder of Grumman. From the earliest days of building aeroplanes, Leroy “Roy” Grumman instructed his managers and engineers: “You bring the pilot back one way or another.” Combine that with a belief that there are no such things as random failures and a mantra of reliability through rigorous testing, and the conditions for success were set.

The story of the ascent propulsion system reminds us of what we already know: that risks are inherent in any bold activity and that they must be identified, assessed, reassessed, and constantly communicated across all levels and organisations. Creative engineering can eliminate some of those significant risk contributors whilst accepting that others will persist, no matter how rigorous or prolonged the engineering and testing programmes.

Organisations must have the courage to introduce new risks as novel or unproven engineering practices and construction techniques are embraced, despite known impacts to cost and schedule

As for Apollo 17’s intrepid crew, the ascent engine worked perfectly, ensuring they made it home for Christmas. But that is only half the story. Just as significant is the part played by those on Earth who dug in and delivered

Lastly and most significantly for the ascent propulsion system, individuals were empowered to investigate and, through their own initiative and actions, solve problems they encountered. Personal responsibility was key, enhanced by scheduling regular visits with the Apollo flight crews – direct engagements with those whose lives would be reliant on their work to get back home. One such example was the secondment of astronaut Charlie Duke into the Rocketdyne team. The team would review themselves at key milestones with the question: “Is it good enough for Charlie?” Duke himself would walk on the Moon, returning safely in April 1972.

As for Apollo 17’s intrepid crew, the ascent engine worked perfectly, ensuring they made it home for Christmas. But that is only half the story. Just as significant is the part played by those on Earth who dug in and delivered.

References

1. Thomas J Kelly, Moon Lander, How we developed the Apollo lunar module, Bravo, 2009.
2. Apollo Experience Report, Ascent Propulsion System, NASA TN D-7082, March 1973.
3. Richard Thruelsen, The Grumman Story, Praeger, 1976.
4. David Baker, The Rocket, The History and Development of Rocket and Missile Technology, New Cavendish Books, 1979.
5. John M Logsdon, After Apollo? Richard Nixon and the American Space Program, Palgrave Macmillan, 2015.
6. Andrew S Erickson, Lessons from the lunar module: The Director’s Conclusions, Acta Astronautical 177 (2020), p514-536.
7. Josuha Stoff, Building Moonships: The Grumman lunar module, Arcadia Publishing, 2004.
8. SKYLINE vol 27, No 3, 1969, Published quarterly by the Aerospace and Systems Group, North American Rockwell.
9. Remembering the Giants, NASA History Division, December 2009, NASA SP-2009-4545.
10. Michael Collins, Carrying the Fire, Farrar Straus Giroux, 1974.