Weathering the Hurricanes of 2004
Novell Cool Solutions: Feature
Digg This -
Posted: 26 Jan 2005
The 2004 hurricane season affected many Cool Solutions readers living in Florida as well as Jamaica, the Cayman Islands, and other Caribbean locations. Hurricanes Jeanne, Ivan, Frances and Charley caused as much as $26 billion in damage, according to AIR Worldwide Corp. The storms marked the first time in Florida's recorded history that four hurricanes hit the state in the same year, according to the National Oceanic and Atmospheric Administration.
We wondered what happened to the networks of all the people involved in the hurricanes, and we held a contest to find out how people fared. We know everyone has a Disaster Recovery Plan, but there's nothing like 120 mile per hour winds to expose the flaws in such plans.
Here are the winning entries. Winners received Solar Navigator Lanterns.
Michael Holmes, Seminole Community College
Encounters of the Windy Kind?
I work for Seminole Community College (SCC) in the role of Director for Academic Computing and User Services. SCC is a public, multi-campus community college located in the city of Sanford which is in the heart of Central Florida about 20 miles north of Orlando. The area is known as the I-4 High Tech Corridor. We natives lovingly refer to the area as "The Silicon Swamp". After this year's storm season, swamp is quite apropos.
We've endured not one, not two, but three hurricanes this year. Charley, Frances and Jeanne each knocked on our door; all within six weeks. What a hellacious six weeks it was. Charley passed directly overhead while the eye of Frances and Jeanne brushed within 40 miles. Forty miles may sound like a far distance but not when the eye of the storm is 35 miles wide. Our site was on the north-east side of Frances and Jeanne which is always the worst place to be.
About our network:
SCC does use "other" OS's on the network but the backbone is NetWare and is eDirectory driven. It's been rock solid unlike "another" OS. We currently have about 18,500 student accounts and about 2000 faculty and staff accounts registered in eDirectory. SCC also uses GroupWise for faculty and staff e-mail and NetMail for student e-mail systems.
SCC's physical network is somewhat unique for school of our size. Through a partnership with Seminole County Government the college leases approximately 65 miles of dedicated fiber from the county's Traffic Division. The leased fiber gives SCC a gigabit triangle that connects its three campuses. It's this high speed link that helped us in hurricane preparedness.
Round One: Hurricane Charley
Charley paid his visit on of all things, Friday, August 13th. We were fortunate that Charley hit between the college's summer and fall terms therefore classes were not in session during this first go around. Before the college closed and battened down its hatches the college DBA and fellow director backed up the PeopleSoft databases and shipped it off to one of the remote campuses. The gigabit link between the campuses made the backup and replication a breeze.
It was decided to leave the core servers up and running so that the college could make critical information available to faculty, staff and students. The PeopleSoft portal, GroupWise, NetMail and web servers were all left up and running while the others were shut down. We knew we were going to lose power at some point however, and we were confident that our data center generator would be able to supply the power needed until the power company could get the entire campus back on line. Charley visited, his eye passing directly over Sanford. Not since 1960 has a hurricane passed directly overhead. It was a scary night. The sound of the howling winds is something you never forget. We were fortunate that the storm was fast moving and was here and gone within an hour. The aftermath however is another story.
The college's network was still up and running the day after. E-mail and web services were still on the job and we still had our internet connectivity. However the college suffered physical damage as Charley took part of the Adult Education building's roof off and uprooted many large oak trees. That was Friday night and the college was open for business on Tuesday. Our staff was back at the college on Monday and brought up the remaining servers without incident.
While the college was back up and running, surrounding neighborhoods were without power for up to five days. Some areas were without power for seven days; and it was a very hot August.
Round Two: Hurricane Frances
Here we go again. September 5th, not more than three weeks later, here comes Frances. While Charley was a fast moving high wind storm, Frances took her time, pounding the area with sustained winds of 60-65 mph for 12 hours straight. She just wouldn't go away. The anxiety of watching the local meteorologists talk about the storm and waiting for its arrival wore at the nerves. It's unnerving too as you sit at home during the storm and watch 50 foot oak trees do the limbo, no pun intended, praying that one doesn't fall toward your house.
The college was prepared once again and we had our hurricane script ready. Good old GroupWise and NetMail kept our people informed as the storm passed. This time however, the damage to the college was much more extensive. What Charley started Frances finished when the remaining portion of the roof came off the college's Adult Education Building. Having toured the aftermath it looked like a war zone inside the building as furniture and computers alike were covered in acoustic tile splat. The proof is in the pictures.
The building's network infrastructure was powered by two Cisco switches. The north wiring closet managed to escape significant damage but the south closet was not so fortunate. Ever see a grown network analyst cry? They didn't but could have as they assessed the closet's 4500 switch, water weeping from the ports. It was this wiring closet that housed the college's link to the county fiber system. The network team was able to quickly bypass the 4500 switch and reconnect the college to the county fiber system thus keeping the college's two other campuses connected.
There were about 200 computers inside the building and all but a precious few were very water logged. The Wednesday after the storm our team was crawling on their hands and knees as we unplugged cables and wires from the damaged computers. Suddenly one of our notorious Florida summer afternoon thunderstorms cracked overhead. It was a rather wicked electrical storm that dropped an additional 4 inches of rain on the area and most of it I think was coming through the roof as we rushed to retrieve what computers we could.
The college lost the contents of the building and it displaced about 4000 students. Here again, eDirectory and NetWare proved itself as administrators rushed to find new classroom space. The student's didn't have to worry because no matter where they ended up, they could log into the network and do their work. Classes are now being held in every type of room you can think of.
Round Two ?:
"The Silicon Swamp" was spared from Ivan's wrath unlike our fellow college in Pensacola. However we were not totally untouched. Ivan went up the Appalachian's and circled back coming back across central Florida as a tropical storm. This time though all we had was more rain?. more rain?.
Round Three: Hurricane Jeanne
Like Frances, Jeanne was a slow moving storm. On September 26th we again endured 12 hours of hurricane force winds. By the time Jeanne came to visit it seemed most of the limbs and other debris had already been blown down or cleared out by the previous storms so local power outages were not quite as widespread. State-wide, however, there were 3.5 million people without power.
The college was closed for two days while the campuses were cleared of debris and safety was assured. Once again our systems performed flawlessly. This time all the servers in the college data center were left up and running without incident.
Damage to the college data center was virtually non-existent due, in part, to the fact that it is in a concrete building with a concrete roof. The exception to damage was a small leak where two concrete beams meet in the roof. The Adult Education Building is under re-construction. Some college staff members were not as fortunate as they had extensive damage to their homes. You could not complete repairs quick enough before the next storm arrived. Should you fly into Orlando International Airport even today, you will still see a patchwork of blue tarps as people try to reclaim what the storms attempted to take away.
I must tell an amusing anecdote about one of my team members: Here he was; the hurricane is raging outside, there's no power and he's surfing the internet tracking the storm. How could he do that you ask? He has a wireless home network and several large battery backups. Amazingly his internet connection never failed. He's such a geek?
Our network and systems performed flawlessly during the storms. Thanks in part to our choice in network OS we were able to deliver the necessary information via e-mail and web services without incident. In the aftermath the same OS enabled the college to relocate classes and the students picked up where they left off before the wind started blowing.
I read an article once about NetWare. The title was "NetWare: It just works". Thank you and AMEN.
Hurricane Ivan, September 11-12, Grand Cayman.
I'm IT manager for a chain of duty-free stores in Georgetown Grand Cayman. We had several days notice that Ivan was going to be 'the big one', but we get a threatened big one each year, and as usual the general population was fairly blase. However our MD is the son of a sea captain, and at least as far as the weather is concerned is the cautious type. Fortunately. He decides we'll close Friday and Saturday to let the staff get their houses ready.
On the Thursday afternoon we are still hoping for a shift northward, they ALWAYS go northward; but just in case I send the techs up on the roofs of our three wireless-linked buildings to take down the antennas. "No one else is taking down antennas" shouts 250lb Derrick as he sways on the top of the (slightly cracked) metal ladder while lightning crackles in the distance. We decide to leave the servers up until Friday as we aren't due for any action until Saturday afternoon, and it'll probably go North....
Friday dawns, a beautiful day, downtown is a ghost town as only the hardware stores are open. The managers all work a full day nailing up ply, moving Wyse POS machines up on top of the sales counters, unplugging, packing, coiling. That evening I down the servers, take out all the disks and stow it in the (3rd floor) safe; take the extra backup and a clone Netware 5.1 server home. Friday night seems like party night, everybody still thinks it's going to miss us; it ALWAYS misses us, right?
Saturday is another beautiful day, but a little windier. Many staff are going to shelter in our office building and start moving in. Self, wife and dog vote to stay at home behind our new shutters with our accumulated hurricane supplies - fools! In the afternoon the wind picks up rapidly, we retreat behind the shutters and eat a hearty meal, watching the ladies final of the US open until the power goes off. We won't know who wins for another week.
By 2 am the wind is howling, shingles are rattling, the rain sounds like waves breaking over our second-story apartment. Ceilings start to leak and then fall in. We frantically move furniture around, gradually retreating from one room to the next as the damp patches in turn become drips, dribbles and eventually waterfalls. The phones are still working, somewhat, and we are able to call the UK to ask where the weather websites say the storm is. Unfortunately they can't answer as we've aclled them on their dialup line. By the time they've found out the answer we've lost the phone as well.
When our watches say it's Sunday afternoon the noise is still incredible, it doesn't sound like this hurricane has an eye. I open a shutter just enough to peer out and it looks like nuclear winter, I close it again rapidly. We stay up through until Monday at 6 am when it starts to get light. Feeling pretty sorry for ourselves and our wrecked apartment until we see next door has it worse, and learn our neighbour's husband has had a seizure for 14 hours until we get the door open.
We walk into town to see the office. It's intact, apparently 50 people spent the night on various floors and one desktops. Town's a mess, but we've seen the after-effects of near misses before. We ain't seen nothing yet....
One of the cars works, so we drive north to our most outlying store at the Strand. The beach area looks worse, sand everywhere, lightpoles down, roofs missing completely or torn up. The store has a single smashed door and a few things look oddly placed. We discover afterwards it had 8 feet of swamp water through it. The 160+mph winds weren't the biggest problem, the storm surge has covered half the island at some point. The boat I sold last year is sitting on her keel and embedded in a construction hut in the middle of a car park. She and the 70-ton dive boat she was moored alongside have floated 100 yards and over 10 foot fences before being laid gently(?) on top of the Hyatt staff cars.
We drive south and it's worse. The surge has blown straight through million-dollar homes, taking some out altogether and just gutting the ground floor of others. I pick up my first of 5 punctures in a month, all from roofing nails. Everywhere are wrecked cars, eventually we'll hear 50% of cars are to be written off and 30% more have major damage. 70% of homes are also seriously damaged or worse. The further from town the worse the damage, some of the traditionally hard-hit places are even harder hit this time. Three large condominiums are almost totally destroyed.
Back to the office, there's no power and water, and there are still people living in the computer room. We aren't going to see tourists for a while so the management team have a bit of time to work out what we do next. Damage assessment starts. The main building had minor leaks from other people's zinc roofs slamming into our new plastic 'waterproof membrane'. The buyer's office is leaking like a sieve, and proves eventually to be our biggest loss of equipment as desktops corrode and moulder. A few stores have minor water damage but except for the swamp victim their equipment is safe. Thank <insert diety> for Wyse boxes with no moving parts.
We have staff meetings and promise no layoffs - yet - but encourage staff (NOT tech staff) to leave the island as accomodation is a problem.
Two weeks later we have water and power in town and I'm able to make ice in the office fridge and carry it home for an evening G&T. We bring up and test all servers, everything fine except a blinking light on the main UPS and the condenser on the backup a/c unit is no more.
No tourist ships due for another month so we gradually roll through all the stores and offices fixing what needs fixing and writing down what's ruined. Our two hardware techs become adept at bringing dead desktops back to life. After 4 weeks we finally enter the Strand store and can save one Wyse box ($299) out of $10,000 worth of computer and security equipment, but the worst thing is the smell from the supermarket next door. Hazmat crews have been in there for two weeks so far, the flies were so loud they thought the generator was still running.
Under pressure from returning office staff we replace water-soaked network switches in the buyers office, only to have the temporary roof fail again in a downpour and ruin another set. Fortunately the insurance people are so slow we get to amend the claim.
While there's no life yet in any of the local computer stores and few users to keep happy, I spend hours on CDW's site buying replacements and putting together a new rack of servers. I get the orders in but getting non-relief supplies through Miami is slow. Convincing the MD that we need a cluster and a hot-standby on the next island has somehow gotten easier....
Finally we put back the antennas. From up on the roof and with all the vegetation blown away we can see for miles and we can't see any other antennas anywhere.....
Rick Elliott, Lockheed Martin Information Technology
Hurricane Ivan took a direct shot at Pensacola, Fl. Our facility, located approximately 10 miles inland on Saufley Field, was buffeted by 110+ MPH winds and torrential rain. Our 60-year-old building survived intact except for some shingles missing and some minor water damage. Down trees were everywhere! The network survived intact running on generator power for a week. Our outgoing circuits were down for four days until power was restored to the local communications equipment on the other end of the base.
Network techs do not normally work on main power circuits, but you do what you have to do to get services back up. The communications techs wired the base communications facility into a generator and all circuits were back on line by noon on the Monday after the storm. It quickly became evident that staff located at NAS Pensacola were not going to be able to enter their building due to the damage from the storm, so a "Tiger team" rebuilt an abandoned building, installing voice and data communications equipment, getting the building on line and available for use less than 24 hours after the tasking was received.
One week after the storm the NAS Pensacola staff were relocated and back on line. Additionally the FEMA distribution center for Ivan relief was operating out of our base so as a way to say thanks for all their help and the long hours they were working, the tech staff set up an Internet Cafe for the FEMA workers with PCs and phones so they could contact their families. The LAN techs and network engineering staff put in some long hours to get the staff back on line and working, but that's our job!
Todd Seagraves, Graybar Electric
My name is Todd Seagraves, I am a System Administrator at Graybar Electric. Our network itself was not severely impacted but we did use some cool technologies and intuitive thinking to help out in some of the rebuilding efforts.
Graybar Electric is a distributor of Comm/Data and Electrical equipment. So we sell almost everything you need to rebuild after a hurricane, from single electrical outlets up to large transformers. As unfortunate as a disaster like this was, it was good for business. When the hurricane hit Punta Gorda, FL it hit hard and within days Graybar had loaded up supplies into trailers and was headed to the heart of the destruction. We have locations all over Florida, but there wasn't one in Punta Gorda, so we made one.
With all the phone lines and data lines down it was almost impossible to place orders for customers there onsite. Thanks to some very intuitive thinking, we were on location conducting business. We put some wireless air cards into laptops and sent them down there. The laptops connected to our VPN via the wireless connection and our employee's were able to process orders immediately onsite in the midst of destruction. In a matter 15 days we had completely setup an emergency response trailer conducting normal business. This was a great accomplishment for Graybar and a significant help to the restoration efforts in Punta Gorda.
We are proud to know that we had a hand in helping get that city on its feet and running again.
Bonita Davey, Escambia County School District
We here in Escambia County Florida shut down all networks till we received the ok to bring them back up. We had two servers that took severe water damage. One server we had to replace. The roof leaked and the server sat in water for about a week and a half before we could get to it. The other server we got to in time to save it. Once it dried out it was ok. Both hard drives survived.
Considering all the damage we had here in Escambia County, we came through very well. We have about 70 or so centers (schools, special centers, administration buildings, etc.). We had one school close for major repairs. All other schools remained open while waiting for repairs.
It is something I don't want to have to go through for a long time to come.
Novell Cool Solutions (corporate web communities) are produced by WebWise Solutions. www.webwiseone.com