MM wrote: I was working for a computer component manufacturer and we were moving to a new state-of-the-art facility complete with a brand new server room with special fire precautions and brand new clean rooms. On our first walk thru of the building, I’d asked about the electricity and was assured we’d have plenty of dedicated power to the server room. We went ahead, marking locations for coax jacks to be placed based on the office and cubicle floor plans at the time. A few more go rounds were needed as we made adjustments and each time I’d asked about electricity. The day before move in I went in for a final walk thru and cable test of the ports and ran into the head electrician. I asked him to show me the power panels and found he hadn’t labeled any of the switches. I managed to talk him into figuring out which switches controlled the server room and clean rooms so we could label them as “don’t touch” switches. While he ran around figuring out which switch did what I noticed that 3 of the 4 panels looked “odd”. My grandfather had been a general contractor and I worked in some construction companies during my college days so this didn’t seem right at all. When the electrician got back I pointed out the differences in the panels and told him I didn’t think they were grounded properly. The electrician shook his head and assured me everything was grounded properly and that I shouldn’t worry my “pretty little head about such technical stuff”. Strike 1 for Pointy Haired Electrician (aka PHE).
Fast forward a few months. We’d been experiencing oddly timed outages of our Prime system which was on the conditioned line. My NetWare 3.12c server, complete with dual 5.25″ drives and 2 full-height 600MB hard drives (okay, so it was a LONG time ago) was plugged into a UPS which would report power outages that coincided with the Prime system’s outages. I demanded we haul the electricians back in to check the power panels. Junior electrician comes in, says something looks odd, but he has to call his boss. The PHE assures him, over the phone, that all is fine and I’m just being “hysterical”. Strike 2 for PHE.
We call in our local utility company to do a weekend test of the power in our server room. Comes back all hunky dory, but my team notes that the test occurred when all the engineers that would have been using the clean rooms were off on a team building weekend. So we schedule to have them come back during a busy work week. We see some fluctuation but no outages, so utility company calls PHE who again insists all is fine.
My gut tells me to get in a “spare” server for “just in case”, so I go to my PHB-CFO to request said server. This was back in the days when NetWare servers were pretty much high-end workstations and ours was a custom ordered Everex (anyone remember them) which had taken 3 weeks to get, and had a 1.2GB Gigatrend tape backup unit in it. PHB says it’s “too expensive” and anyway he’d talked to the PHE just that day and was assured that it wasn’t a power issue. Strike 3 for PHE.
Fast forward another month. Major disaster…power spike takes out the Prime system, NetWare server AND our UPS. Battery in the UPS had exploded and the surge was so bad to the server that it melted the daughter boards to the motherboard, nothing could be removed but the drives. A few parts later on the Prime and our primary manufacturing application was on-line by the next business day. The NetWare box was dead in the water. PHB-CFO tells me – oh, just take another workstation and put the hard drives in and make it work. I try to explain to PHB-CFO that a white box PC with a RLL drive is NOT the same as an Everex with an MFM controller and speciality card for the tape back unit. He doesn’t listen. I do my best to try to find parts in town to get the hard drives to work in a white box but it’s all no go as this was NetWare 3.12 and you couldn’t easily swap out parts without having to do some major rebuilding. And it had taken me over a week to compsurf the drives to begin with. Every day for a week I walk into PHB-CFO’s office with a quote in hand saying “just buy this already it’s cost you XXX dollars in downtime already). At the end of the week PHB-CFO finally gives in and I get my server that day, thanks to my local VAR who’d ordered it the first day I’d called.
I tried the old drives in the new server, which had identical drives already installed, held my breath and punched the power button. By pure luck and the grace of some guardian angel the server booted up just fine and we hadn’t lost any data. And our tape drive had been spared, which was even better. I tell the PHB-CFO the good news in the hallway as the company President is walking by. President asks what I thought the cause was and I tell him about the power issues we’d been having. President gets on the phone and calls in one of his golfing buddies who happens to own an electrical services company.
New electrician comes in and asks me to show him what I think is wrong. I show him to the power panels and tell him “they don’t look they’re grounded to me”. Electrician shakes his head, groans, and says “That’s because they’re not grounded at all! You’re lucky you didn’t lose all the clean room equipment too.” End result was an apology from the PHB-CFO to my team, a lawsuit for damages to the original PHE, and a newfound respect for DR planning.