Thursday, August 17, 2006

stories from the field....

This did not happen to me, but I've helped out the engineers who were there when this tool blew up on other issues, and worked on software for a tool of this vintage. It is pretty much the worst-case scenario of what you can do to an ion implanter accidentally and still survive; if they were using old-style gas bottles that were at positive rather than negative pressure and those had blown off, there would also have been a significant poison gas release, and if the tool was more recent or had a linear-accelerator unit on it, the explosion would have been that much bigger.

The tool in question is of a type that ceased manufacture shortly after I was born, and is pretty much out of the industry. However, all names of customers and suppliers have been redacted to protect those not at fault. The implanter designation has also been redacted, because it's not like these things blow up all the time. This should be seen as we in Product Support saw it, as a cautionary tale about the Extremely Bad Shit that can happen when you work with a system that produces poisonous vapors and excessive voltage by design, and explosive precursors as a side affect of operation.

----

Let me start by saying that all that was necessary to change out the [supplier] supply was to shut off the breaker to High Voltage, and Isolation Transformer in PD2 in the TERMINAL PD only. This could have also been locked, or tagged out to fulfill safety requirements. By shutting off power to the entire terminal, the customer actually created an out of control set of conditions that resulted in the explosion.

Shutting off terminal power without first going to a vacuum safe condition does several things, none of which are good, by the way.

First, the power is removed from RP1, which backs the diffusion pump as well as the remote vacuum controller. The DP is still hot and will begin to backstream oil. V4, V2, V5 and V17 will NOT close, as the vacuum controller has just become 100% dedicated to trying to communicate with the remote vac controller only. This communication was interrupted when the power was shut off to the terminal.

So, as the mechanical pump stops, a leak to atmosphere will happen, quickly raising the pressure through the source beamline and resolving housing, and once the pressure exceeds ~2x10-4 torr the ion gauges on P2 and P3 will extinguish (due to limitations on the emission control pcb, and not due to any commands from the vac controller -- in this particular case, this also removes the possibility of an ignition from the cryo ion gauges). Next, the cryogenic pumps will ice ball and begin to warm up due to heavy loading from the leak condition described above. In this case, due to the pumping load, the cryos will be overwhelmed and begin to warm up even though they were bypassed and the compressors and cold heads continued to run. Once they warm up to ~ 20 Kelvin, hydrogen will begin to be released from the array; a little warmer and oxygen is released from the array also.

Because all of the valves are open through out the system, H2 and O2 will began to fill the resolving housing, beamline, source housing and diffusion pump. The slight draw on RP1 exhaust may have facilitated pulling more gas into the DP through its foreline, concentrating the amount of gas in this location.

The system is now primed and ready, and waiting for any spark from an ignition source. Once terminal power was restored, and DC power restored to the remote vac controller, it would begin to re-establish communications with the vacuum controller; however most likely before that could take place, the remote vac would bloom the DP ion gauge, because the logic and gauge power are coming up at the same time. Normally this is not a big deal, and even if the DP is cold, it would only result in a ½ second pulse and immediately shut off the gauge filament power. However, if you happen to have the vacuum system filled with some amount of H2 and O2 the resulting explosion could be quite impressive.

Note: from the pictures [originally attatched, obviously redacted] you can see that the optical baffle over the DP is bent towards the electrode, indicating that at least part of the explosion (also believed to be the ignition point) was in a direction out of the DP into the Source housing, then out through the source bushing as the holding dogs broke, and the source and source mounting flange fell away. At the same time that the gas in the beamline and resolving housing ignited, it would have caused the damage to the cryos, the housing blank off plate, and the accel bushing, and even move the terminal a little; as I said, very impressive.

Thankfully, no one was injured, as things could have been much worse.



The [obsolete tool] explosion I told you about earlier is actually worse than first thought. Evidently when the terminal power was off it also shut down the cryos. When I turned on terminal power I suspect the ion tubes over the cryos lit and exploded the hydrogen released when they started warming. The source and flange blew out of the housing. The cryos blew off the mounting flange, the terminal shifted and blew open the muff coupling. No telling what other damage was caused.

When the [customer] electrician was swapping the [supplier] tanks from 137 to 167, his supervisor told him turn off all breakers. The machine was still in hivac. I think what happened when he did this was that he inadvertently hit the terminal power off button. Neither [other FSE] or myself were involved in the [supplier] swap. We were concentrating on the [unrelated tool] problem which was priority with [customer] and is positioned right next to the [obsolete tool] that exploded. When the [customer] electrician had the [supplier unit] connected he asked me to check all the connections, which were all ok. After he turned on all the breakers he had turned off, I noticed the terminal power wasn't on. I hit the terminal power on button, I noticed that the fluorescent light in the terminal came on, and a split second after that is when we all heard a great bang. I immediately pushed the EMO button on the terminal power distribution panel.

I wanted to write this as soon as possible while it was still fresh in my mind. [Other FSE], if you can add anything have at it. I just want to add, since I was standing right next to the machine when this happened, it thoroughly scared the living crap out me. In all my years as an FSE with this company, this has never happened to me.

-----

Tool detonation is an uncommon but extant problem in the semiconductor equipment field. It's unknown how many implanters or other devices with cryo pumps on them blow up every year, because both customers and equipment manufacturers never want to admit that their tools blow up, or that tools blow up in their fab, and jealously guard the information. It still happens, and is the reason that we insist that our cryo pump suppliers detonate one of their pumps for us before we purchase any, so that we can be sure that the unit will contain the explosion and not throw shrapnel into anyone that's standing around. Of course, one of the major side benefits of designing a system that will bear borderline hard vacuum (edge-of-the-atmosphere pressures) without imploding is that it's also unlikely to disassemble itself when you get a mini-Hindenburg inside.

Just glad it wasn't me standing in front of it....

No comments: