"Captain Kirk Forgot to put the Machine on Stun"

Pushkin. Nobody really wants to be deliberately bombarded with radiation, but if you have cancer, radiation therapy might just save your life. That must have been what a young man called Voyin Ray Cox hoped as he made visit after visit to the East Texas Cancer Center. Just thirty three years old, Ray, to his friends, was young to be a cancer survivor. He had a tumor cut out of his shoulder, and now March nineteen eighty six, he was there for his ninth session of radiation therapy designed to ensure that no traces of the cancer remained. Despite his bad luck, Ray was a cheerful, resilient man. He knew the drill, press his bare chest and stomach onto the cold metal treatment table, chat to the operator while she

maneuvered him into position underneath the looming bulk of the THEAK twenty five radiation therapy machine. The operator knew the drill too. In his account of the case, the ergonomics expert Stephen Casey calls her Mary Beth, although that's not her real name, will do the same. Mary Beth cheerfully caught up with Ray as she used a console control to precisely position him under the THAK twenty five's radiation beamgun.

Then she walked down the corridor to the control room, which was at a safe distance. Ordinarily, the control room and the treatment room would be linked up by CCTV and microphones, but neither the cameras nor the audio were connected that day, and that didn't seem to matter. Normally, they'd be useful for some reassuring chat or to give the patient a word of instruction, and Ray, being an old hand, didn't need any of that. Mary Beth typed the treatment instructions into the computer a series of letters.

She pressed X to choose the mode, then straightway realized her mistake. Ray needed the other mode. She deleted the X and pressed E should check the instructions. They were all correct, beam ready, the computer told her. She pressed B to administer the treatment. Down the hall. On the treatment table, Ray Cox heard a sizzling sound and saw a blue flash and then agony. It was like someone had thrust a hot skewer through his sh shoulder. This wasn't right. He knew it couldn't be right. The last eight treatments had been nothing like this. Back in the control room, Mary Beth couldn't hear Ray's cry of pain, and she couldn't see his body contorting on the treatment table.

All she saw was a bland little notification malfunction fifty four. It wasn't clear what that meant. The machine would often pause and produce an unexplained error code, sometimes thirty forty fifty times a day. As one operator later commented, I can't remember all the reasons it would stop, but there were a lot of them. The machine indicated that Ray had received only a tiny fraction of the intended dose, and Mary Beth had been assured that the Therak twenty five had so many safeguards it was almost impossible to overdose a patient. It only took a single keypress for her to reset the machine and try again. I'm Tim Harford, and you're listening to cautionary tales. Mary Beth was an experienced operator of for Ferrak twenty five. She must have

seen it crash and pop up an error message ten thousand times or more. Although she didn't know what those error messages meant. How could she? The machine's manual didn't explain them, it didn't even mention them. Because she'd been in that position so many times. It took her mere seconds to reset the machine for another try at giving Ray his treatment. That wasn't enough time for Ray to get off the table. He'd rolled onto his side, but mary Beth, of course couldn't see that, and she couldn't hear his agonized yelling. The machine fired again, another flash of blue light, Another sizzled, and this time a hot skewer went through Ray's neck. He was in too much pain even to scream. Then the agony started to fade.

He gulped in some air and blew it out again. Tried to calm himself, Hey, are you pushing the long bottom? But mary Beth couldn't hear him. Something had gone wrong with the Tharak twenty five. But what loyal listeners to cautionary tales will be familiar with the Swiss cheese model of accidents, made famous by the psychologist James Reason. Imagine slices of Swiss cheese with those distinctive holes in them.

Each slice represents some kind of safeguard against an accident. Maybe it's a fail safe in the hardware of a system, so it simply won't work if the right pieces aren't in position. Maybe it's a subroutine in the software, monitoring what the system's doing and shutting it down if another part of the software happens to glitch. But no safeguard is perfect. Every slice of cheese has holes in it.

In James Reasons model, an accident becomes possible when all the holes line up. That means that every safeguard in the system becomes simultananeously vulnerable to the same kind of problem. To prevent accidents, then get extra lines of defense and try to strengthen the defenses you already have. To put it another way, get more slices of cheese with fewer holes in them. So did the THEAC twenty five need better hardware or better software to prevent the accident that happened to ray Cox? As it turns out, yes, but that's also the wrong question. We should be looking at a different kind of cheese slice. Altogether. Nine months before ray Cox's excruciating experience, in July nineteen eighty five, forty

year old Francis Hill arrived at the Ontario Cancer Foundation Clinic for her twenty fourth round of radiation treatment for cervical cancer. The clinic was using a twenty five machine, but there was a problem the machine didn't seem to be working. Every time the operator tried to fire the radiation beam, the machine paused, produced an error message and reported that no dose had been given. The operator hit the pea key to proceed, and the same thing happened.

The operator hit p again, and it happened again. We've all been there, clicking an icon on a screen, finding that nothing seems to happen, and then clicking it again. After four attempts, the operator called a technician who couldn't find anything wrong with the Thearact twenty five. Francis Hill left and the machine was used successfully on half a dozen other patients that afternoon. That sort of thing wasn't particularly strange, the operator reflected. The fair Act twenty five would often seem to glitch like that, produce mysterious error messages, and then suddenly working again for no particular reason. But while glitches didn't seem strange to the operator, something seemed strange to Francis. She could feel a kind of burning, tingling,

electric shock kind of sensation in her hip near where the therapeutic beam had been aimed. When she came back for another round of treatment three days later, her doctors immediately diagnosed a radiation burn in her hip, which was painfully swollen. They called the machines manufacturers to report a suspected radiation overdose. The manufacturers were Atomic Energy of Canada Limited AECL of radiation overdose. AECL had never heard of anything like that before Strange. They sent a engineer along to investigate. The FARACT twenty five could be used in two different modes. The electron mode attacked cancer near the surface of the patient's body. The machine emitted a beam of electrons spread out by an array of magnets. The

X ray mode attacked cancer deep inside a patient's body. The magnet array would be moved aside and replaced by a device called a flattener, which focused the X ray beam precisely on the cancer. The flattener absorbed a lot of energy, which meant the X ray beam had to be very powerful. The components which diffused the electron beams or focused the X rays on the FAACT twenty five were positioned on a turntable. As the machine was programmed to fire either electrons or X rays at the patient's tumor, the turntail would rotate automatically to fix the right component into position. At least that was the idea. Two types of radiation beam then one that needs diffusing, one that needs focusing. If this sounds like an accident waiting to happen,

well it was. But machines like the THEAK twenty five are expensive, and this dual purpose design meant that hospitals got more bang for the buck. As long as the right component was in place for the right beam, there would be no problem. The THEAK twenty five was fully controlled by a computer, unremarkable these days, but radical for the mid nineteen eighties. Its predecessors, the THEAK six and the THEAK twenty allowed a human operator to physically position the magnets or the flattener. On the THEAK twenty five, this manual position was replaced by servo motors, computer controlled to quickly and precisely put everything in position. When AECL investigated the incident with Francis Hill, they weren't actually able

to reproduce the error, but they suspected that the turntable system hadn't worked properly. The turntable had three tiny switches designed to measure when it was in position, but it emerged that a single bit of error the computer glitching and mistaking a zero for a one could produce a faulty reading of the turntable's position. So AECL told the clinics that used the THEAAK twenty five to visually confirm before each procedure that the turntable was in the correct position, just as a precaution until further notice. They tightened up the software, making it more robust to a small error like that. Then they got back in touch with the clinics.

No need for those visual checks anymore. We've just made the machine five orders of magnitude safer, which in plain English means it's about one hundred thousand times safer than before. And it was safe already. But one thing AECL don't seem to have done is to have notified the clinics that an accident had happened and that a patient had been injured by the machine. The clinics, at least say they weren't told of any injuries. When AECL announced that they'd fixed the problem, it was September nineteen eighty five.

A month later, they were sued by a woman named Katie Yarborough. AECL had never heard of Katie Yarborough, who was Katie Yarborough. Cautionary tales will return after the break. Katie Yarborough's injury happened seven weeks before Francis Hills, early in the summer of nineteen eighty five. She was being treated at the Kenniston Oncology Center in Marietta, Georgia. Katie was sixty one. She'd had a malignant tumor removed from her breast, and now she needed follow up treatment to destroy any secondary tumors which might have spread to the lymph nodes under her collar bone. That treatment, of course, would be provided by a Therak twenty five machine. But when the technician fired up the Therak twenty five, Katie

felt an agonizing pain, a tremendous force of heat. This red hot sensation. You burned me, Katie, exclaimed the technician, who was puzzled. That shouldn't be possible. The technician said the twenty five was safe, and there wasn't any sign of a burn. Perhaps Katie's clavicle was a little warm to the touch, but otherwise nothing seemed to be a miss.

The physicist at the Keniston Center was a man called Tim Still. When he was informed, he was just as puzzled. Tim knew about the two treatment modes, the powerful X ray fire through the flattener and the gentler electron beam fired through the magnet array. Katie Yarborough had been treated in the electron mode, but Tim still wondered if something had gone wrong with the array. He called the manufacturers AECL with a question, was there any way the electron beam could be fired directly at a patient without the magnetarray in position? After three days, AECL replied no. They explained that was simply impossible. But somewhere along the line AECL didn't seem to get the message that a patient had been injured. Maybe Tim still didn't tell them, or

maybe he did but the message didn't get through to the key decision makers. It's quite possible that still didn't even realize that Katie Yarborough was injured. After all. At first she seemed fine, But Katie Yarborough wasn't recovering from that mysterious burn. In fact, her symptoms were getting worse.

The skin above her left breast had reddened, her shoulder would freeze up, and she suffered excruciating spasms. Her doctors were baffled. They continued sending her for theac treatment. After all, with malignant breast cancer, you can't afford to take risks. But when her skin started to fall off, Katie refused to continue lying on the treatment table underneath that machine.

And when Tim Still later examined Katie, he noticed something strange. Not only did Katie seem to have a severe burn to her upper chest, but her upper back we was starting to Redden two. It was as though whatever had burned her had passed right through her body and caused an exit wound. When Francis Hill was injured, her clinicians didn't know about the injury to Katie Yarborough, and as hospitals across North America continued to use the THERAK twenty five, they didn't know about Francis Hill or Katie Yarborough. They only knew that the machine was already safe and had just become one hundred thousand times safer. And that makes what happened next almost inevitable. In December nineteen eighty five,

Dora Moss, a patient at the Yakima Valley Memorial Hospital in Washington State, complained that her right hip seemed red and inflamed in a distinctive striped pattern. Dora's doctors were puzzled. It wasn't clear what could have caused the inflammation, although possibly it was a perfectly normal reaction to the course of radiation therapy she was having on her hip. Which device was being used, funny you should ask. It was a THERAC twenty five, But because the hospital staff at Yakima weren't aware of the history of accidents, they were baffled.

Some of them wondered whether a slotted component on the THERAK twenty five might explain the striped pattern. Others suspected that it was a burn caused by Dora's habit of sleeping with an electric heating pad. Maybe those heated wires had slowly burned her skin, although on closer inspection the arrangement of wires in the heating pad didn't actually match the sore stripes on Dora's hip. So they contacted AECL who responded.

After careful consideration, we are of the opinion that this damage could not have been produced by any malfunction of the FARAC twenty five or by any operator error, So that was it.

Then, Officially, the cause of that stripeye burn was cause unknown, But if the cause was unclear. The consequences were stark. Dora Moss needed surgery and skin grafts to patch up her ulcerated skin and treat her chronic pain. Maybe it wasn't the FAAC twenty five. It certainly wasn't a burn from a heating pad. The Yakima hospital staff were even told that there'd been no other incidents with the THERAK twenty five. Was anyone putting all these incidents together and spotting a pattern, It seems not, although it's hard to be sure. Nancy Levison, software safety expert and the author of a definitive account of the affair, explains that because there was never an official investigation, it's often unclear who

exactly knew what and when they knew it. At the East Texas Cancer Center in March nineteen eighty six, three months after the injury to Dora Moss, Mary Beth was puzzled. She'd tried twice to administer the treatment to ray Cox, apparently without success. Third time, lucky she hit again. Ray Cox had been trying to ease himself off the table, but when that searing skewer feeling hit him for a third time, jabbing through his neck and shoulder. He leapt for safety, barged open the door, and ran to the nurses station. When Mary Beth emerged, Ray was obviously shaken by what had happened. He told her that he felt like had been given three separate powerful electric shocks. How strange, Mary Beth reassured him that the machine had automatically shut

down and according to the computer's display panel, Ray had only received one tenth of the intended dose. Mary Beth informed Ray's doctor and the center's physicist, Fritz Hager about the electric shocks. They came to examine the machine and Ray. There seemed to be nothing wrong with either of them, but that's the nature of a radiation overdose. It's invisible and at first the injuries it causes who are invisible too.

Ray looked fine, but he really wasn't. Hagar called the manufacturer of the machine AECL to apport the incident. Then he ran through some tests and since everything seemed to be in order, pronounced the machine good to go for the afternoon patients who were waiting for treatment, and everything went smoothly. That's what happened. Remember, the THERAK twenty five often produced mysterious error messages. And then suddenly started to work again for no obvious reason. Three weeks after the strange incident with Ray Cox, Mary Beth had a new patient, sixty six year old Vernon Kidd, who had a tumor on his ear. As soon as the treatment beam was activated, Vernon cried out and started moaning for help. This time,

the audio link was working. What happened, asked Mary Beth.

Fire.

He replied, fire on the side of his face. When the physicist Fritz Hager arrived on the scene, Vernon elaborated he'd heard a sound like frying eggs and a flash of light and then pain. He was confused and upset. What happened to me? We'll find out what happened to Vernon Kidd after the break. September ninth, nineteen forty seven, computer scientists at Harvard University get to the bottom of why their fancy computer, the Mark two, is malfunctioning. It's a bug, a literal bug, a moth in fact, which has crawled into the mass of electrical relays in the room sized computing machine and caused a short circuit. The logbook tells the tale. Handwritten on blue gridded paper are the words relay seventy panel f moth in relay. Next

to those words, the bug itself is preserved under a short length of yellowing sticky tape underneath the dry remark. First actual case of bug being found. Software bugs are the bane of programmers, although as that punchline implies, this wasn't the first time the word bug had been used to describe a device malfunctioning. It was just the first time that an insect had been obliging enough to turn the metaphor into reality. In fact, software bugs are older than computers. The first computer program is widely thought to have been written by Ada Lovelace, an English mathematician and friend of the engineer Charles Babbage. In eighteen forty three, Lovelace published an algorithm that would enable Babbage's proto computer,

the Analytical Engine, to calculate a particular sequence of numbers. What's so striking about Lovelace's algorithm is that the Analytical Engine had not been built, nor would it ever be. Babbage's designs were just too ambitious, and nobody would be able to construct a general purpose computer for another century.

But modern analysis concludes that if Lovelace's program ever had been run on an analytical engine. It wouldn't have worked, not first time anyway, because there was a typo. Not only had Lovelace published the first software she had also published the first software bug. Anyone who's had the experience of writing computer code, or even for gen x as like me, of typing in a computer program printed in the pages of a magazine will know what it's like to have a bug. In some cases, the bugs are easily fixed. You run the program and it doesn't work.

Maybe the computer even tells you where things went off the rails, exactly where the error was, or maybe not, because some bugs are more like an unsuspected hole in one of James Reasons slices of cheese. They sit there, hidden by other slices, causing no trouble until the holes and the slices aligne and the computer crashes. You may not know why it crashed. You may be able to restart the program and find it runs with no trouble, but somewhere under the surface, the bug is still there.

Two of Fritz Hager's patients had apparently been electrocuted, and while the Farak twenty five seemed to be working perfectly well, he wasn't going to risk a third. He shut the machine down, notified AECL, and then tried to figure out what had occurred. It wasn't easy. After each incident, the machine seemed to be working just fine. Hager and mary Beth worked together trying to figure out what had triggered the malfunction fifty four message. Eventually they succeeded. Mary Beth recalled how she had originally typed X to use the THURAC twenty five in X ray mode, then realized she should have typed E. Quickly using the cursor keys to move to the correct box. She corrected the entry to E, repeatedly hit return to accept all the other treatment variables

which were correct, and awaited the message beam ready. What she hadn't known, what nobody had known, is that a quick edit like that confused the computer's subroutines, which checked the setup only at certain moments. The result, the beam was set to full X ray power, but the flattener that would absorb most of the radiation wasn't in position.

To make matters even worse, the Thorax software was confused by this dangerous setup and didn't correctly monitor the dose administered. It was a particular sequence of keystrokes that caused the problem an unlikely sequence, but not an inconceivable one. It shouldn't be particularly surprising that experienced operators such as Mary Beth might type the wrong mode, notice the error, then swiftly correct it. It was the swiftness of that correction that bewildered the software. Fritz Hager explained to AECL that he could now reproduce the error on demand. The AECL engineer couldn't until Hager explained that all the keystrokes had to be entered in less than eight seconds. The next day, the AECL engineer called back, Yes, he could now replicate

the error, and he had bad news. If the beam was fired in such conditions, the patient would receive a dose of twenty five thousand rads, more than one hundred times more than intended, which potentially could be fatal. Over the course of three weeks, sixty six year old Vernon Kid moved from disorientation to a coma to death. The autopsy revealed that the section of his brain running from under his right temple to behind his right ear had been withered by a high dose of radiation. It's natural to describe the problem with a Tharak twenty five as a software bug, and while that's true, it doesn't really help us understand the problem or prevent similar problems in future.

As Nancy Levison writes, virtually or complex software can be made to behave in an unexpected fashion under some conditions. Demanding software with no bugs is like demanding a slice of Swiss cheese with no holes. It's in the nature of Swiss cheese that there will always be holes, and it's in the nature of complex software that there will always be bugs. The question is what happens when a bug appears. Perhaps another part of the software is able to spot the problem. A separate slice of software cheese that didn't happen With a Therac twenty five. The computer would tell the operator what dose the patient had received.

There was no direct measurement of that dose. The bug that led to the overdose also led to the software failing to report the overdose, or perhaps there are failsafes in the hardware. Again, not with the Therrak twenty five, the machine relied on the software being perfect. The shielding components were put in place by the software. The decision to allow the electron gun to fire or not was made by the software, and the dose the patient received was reported by the software. If the software was wrong about one of these things, it could easily be wrong about the others. There is a different approach. The THEAK twenty, the predecessor of the THERAK twenty five, was designed to operate with or without a computer, and it was built

with mechanical interlocks. If you tried to fire the beam without the right component in place, the machine just wouldn't do it. It was only after the THEAK twenty five's problems became widely known that something began to dawn on THEA'AC twenty users. Sometimes the machine's fuse would blow after quick edits I was annoying and a bit mysterious. The machine would have to be switched off, the fuse replaced, and everything restarted. The fra AC twenty software had been built on the same codebase as the fair AC twenty five, and on closer examination it became clear that the THEA ACT twenty had the same software bug. Because of the mechanical interlocks, which would physically prevent the machine from working

unless it was correctly set up, the bug was never anything more than an annoyance. The most precious thing that was damaged was a fuse. The Fair Act twenty five five needed better software, and it needed better hardware. But as I said earlier, that's also the wrong place to focus. Another kind of cheese slice was missing. There was no proper process for noting anomalous incidents, suspected malfunctions, or possible injuries.

Hospitals should have been reporting both the strange incidents and the later mysterious injuries. Someone either AECL or a regulator should have been collecting and analyzing the reports. Because that didn't happen, every new incident caused a new wave of muddle and bewilderment. Of course, we should try hard to eliminate bugs, especially when lives are at stake. But the real lesson here is that safety is not a function of good software alone. It's the function of the whole system, and the system goes beyond the software. In fact, it goes beyond the machine. The system includes the network of people who make the machine, use the machine, and regulate the machine. In this case, they should have been keeping

each other closely informed. They weren't, and as a result, it took months for anyone to assemble the pieces of this awful puzzle Voin. Ray Cox was young and strong, but he had taken a triple blast of high dose radiation to his back, shoulder, and neck. Before long, he was starting to spit up blood. The awful radiation burns on his back and neck turned into yawning lesions as the skin and flesh started to peel off his body.

While over the weeks that followed, the damage to his spinal column paralyzed his left arm, both legs, and his vocal cords, ray tried to keep his sense of humor. Before he lost his ability to speak, he would joke to friends and family Captain Kirk forgot to put the machine on stun. Ray Cox died in August nineteen eighty six, a few months after the accident, and five months after that, Glenn Dodd, a sixty five year old cancer sufferer, was given radiation therapy at Yakima Valley Memorial Hospital, where Dora Moss had acquired those mysterious striped radiation burns after treatment from a THERAK twenty five. Glen Dodd was being treated by a THEAK twenty five, he received a fatal overdose.

Dodd was terminally ill, but doctors concluded that the injuries he had suffered from the malfunctioning machine had killed him before his own cancer could. Why hadn't aecl fixed the software fault? Well? As it happens, Glen do Odd wasn't killed by the glitch that had killed ray Cox and Vernon Kidd, but by a different software error. You see, there's always another bug lying in wait for the moment to strike. This isn't the first cautionary tale we've done about radiation overdoses. Two of my very favorite episodes of the podcast were Glowing Peril, The magical glitter that poisoned a City, and How the Radium Girls Fought Back, two unforgettable episodes which came out as a pair late in twenty twenty three. You may enjoy listening to them. I

first heard about ray Cox's case from Stephen M. Casey's book Set Phases on stun while the definitive authority on the THERAK twenty five case is Nancy Levison's investigation, written with Clark Turner. For a full list of our sources, see the show notes at Timharford dot com. Cautionary Tales is written by me Tim Harford with Andrew Wright. The show is produced by Alice Fines with Marilyn Rust. The sound design and original music are the work of Pascal Wye.

Sarah Nix edited the script. Cautionary Tales features the voice talents of Ben Crow, Melanie Gushridge, Stella Harford, Gemma Saunders and Rufus Wright. The show wouldn't have been possible without the work of Jacob Weisberg, Ryan Dilly, Greta Cohne, Eric Sandler, Carrie Brody, Christina Sullivan, Kira Posey and Owen Miller. Cautionary Tales as a production of Pushkin Industries. It's recorded at Wardoor Studios in London by Tom Berry. If you like the show, please remember to share, rate and review. It doesn't really make a difference to us and if you want to hear the show ad free, sign up to Pushkin Plus on the show page on Apple Podcasts or at pushkin dot Fm, slash plus, st

"Captain Kirk Forgot to put the Machine on Stun"

Kapitelklicken springt direkt hin

Shownotes

Transkriptvom Podcast bereitgestellt · Zeitmarken springen hin