When Hurricane Sandy hit the Northeast United States in late October, towns along the coast had prepared for the worst. But few people could have truly imagined the devastation that lay ahead. As the Category 1 storm approached the shore, it became the largest Atlantic hurricane on record, with winds spanning across 1800 kilometers. After making landfall in New Jersey on 29 October, the storm moved across the area blasting 145 kmph winds. When it was all over, the storm had killed more than 250 people in seven countries along its path and became the second costliest Atlantic hurricane—at US $65.5 billion in estimated damage—behind Hurricane Katrina, which hit New Orleans in 2005.
In Sandy’s aftermath, millions of homes and businesses throughout the Northeast were left without power—some for weeks. The New Jersey shore was ravaged and almost unrecognizable, with many iconic boardwalks ripped out and a rollercoaster in the beach town Seaside tossed into the ocean and protruding out like a scene from an apocalyptic movie. In New York, subway lines were flooded and every borough was affected.
I know I’m not alone in knowing several friends, coworkers, and family members whose houses were ripped from their foundations and their possessions washed away. And when my power came back, it was utterly heartbreaking seeing the TV footage of the New Jersey shore I grew up near completely devastated and unimaginably altered.
Among those affected was IEEE. The storm knocked out power and left the IEEE Operations Center, in Piscataway, N.J., in the dark and closed for a record nine days.
The lack of electricity caused the data center in Piscataway to run on generator power, relying on daily shipments of diesel fuel, which was becoming scarce as gas stations throughout the state had trouble meeting demand—not because of a shortage but because of the electricity needed to pump the fuel. To a certain extent we all felt helpless, hoping power would be restored before fuel ran out.
Meanwhile, those of us who had laptops and electricity worked on what projects we could from home. Like many departments, The Institute and IEEE Spectrum staff held daily teleconferences to check on everyone’s status (was everyone OK? Who still had no power, heat, water, and the status of projects).
While we all waited for the lights to come back on in Piscataway, dozens of staff members worked around the clock—some even sleeping at the building—over those nine days. Their work included holding daily meetings to assess the situation, trying to secure fuel, fixing broken equipment from the storm, arranging for systems to fail over to IEEE’s remote server site in Arizona, and finally bringing all those systems back up in Piscataway once power was restored.
To help members understand what went into the preparation for the storm and recovering from it, I spoke with IEEE Chief Information Officer Alexander Pasik, Director of Facilities and Distribution Services John Hunt, and Senior Director of IT Infrastructure Systems Dave Bankowski.
IEEE began getting ready for the storm on 26 October. All scheduled IT maintenance projects were put on hold, plans were developed for operational support during the storm, and a remote IT Help Desk was established with staff members prepared to work from home on technical issues. Plans were also put in place to failover the IEEE Xplore digital library to remote servers in Arizona if necessary, and generators and the uninterruptible power supply (UPS) systems—which keep power running for 30 minutes to provide time to get the generators going—were checked.
The storm hit the area the evening of 29 October and the New Jersey offices lost power at 7:15 p.m., but the UPS and generators ran with no loss of IEEE services. Soon after, lower Manhattan lost power, including the New York office. At the time, the estimate on fuel levels for the generators was three days—more than was expected to be needed. After all, each tank held 1500 gallons. On 30 October, all systems continued to run as a state of emergency in New Jersey was declared.
But on the morning of 31 October, something unexpected happened: one of the two generators failed, causing a “hard crash” of systems when the UPS systems ran out of charge, according to Pasik. Immediately, failover procedures for IEEE Xplore to the remote server facility began and the digital library was up and running three hours later. In the meantime, the generator was repaired and the data center staff began rebooting the other systems.
Publications Technology staff spent many hours during the outage, shifting IEEE Xplore to the Arizona backup site, as well as restoring all of the Pubs applications after power was restored.
By 1 November, fuel availability had become a concern, with no power restoration in Piscataway in sight. “We kept calling for fuel deliveries but there were problems with restoring power and distributing the diesel,” says Hunt. Multiple daily diesel shipments were arriving, but because of the distribution problems, fuel was being rationed: only about 200 gallons at a time were being delivered, compared to the usual 1000 gallons.
Realizing it needed to prepare for the worst case scenario—no fuel deliveries at all—the IEEE Management Council, which was holding daily meetings, developed a plan to send more IEEE services to the remote facility. They would include Spectrum Online, the IEEE.org main site, and the member e-mail alias service.
The e-mail alias service was an especially challenging venture because it was never run in the remote facility. “We had to build a solution on the fly because the initial capacity of the remote e-mail service wasn't sufficient to support the amount of e-mail,” says Bankowski. “Once we discovered the capacity issue, the team worked to resolve it.”
In the meantime, IEEE Web pages were updated with a notice explaining the situation. And a Google form was added to make it possible for those wanting to join IEEE or renew their membership to leave their e-mail addresses so they could be contacted once the membership registration system was back up. [More than 15 000 e-mail addresses were collected].
MORE FUEL ISSUES
By 2 November, diesel fuel had become extremely scarce throughout the region because of various problems at the refineries. The generator was estimated to run out of fuel by noon the next day, so IEEE began the process of migrating some services, such as Spectrum Online, to the remote server. Staff e-mail, which runs on Google, remained unaffected because it is cloud-based.
“This situation added a whole new level to the benefits of cloud computing,” Pasik says. “Many of my arguments in favor of moving [IEEE services] to the cloud have dealt with the economics of it, but the storm proved how beneficial the cloud is when it comes to availability of services.” Pasik adds that he hopes to move most of IEEE’s systems to the cloud over the next five years. Read more about Pasik's view on cloud computing in a guest blog post he wrote earlier this year.
Over the next few days, the IEEE.org and Spectrum websites, along with the member e-mail alias system and IEEE Xplore continued to operate from the remote servers.
When power was finally restored to the Operations Center on 7 November, and thus to the other New Jersey and New York offices (which rely on the Operations Center data center for connectivity), IEEE IT staff still had much work to do. Data Center and Facilities staff worked around the clock over the next day to restore the data center and various equipment, and bring systems back up. When the rest of the staff returned to the offices on 9 November, heat had been restored, and 90 percent of the systems were up and running.
Although it was certainly a challenging and historic event for IEEE, Pasik says he is pleased with how it was handled.
“I think we handled this amazingly well and our dedicated staff took care of things according to plan,” Pasik says. “No one could have expected power to be out for that long.” Hunt agrees: “Murphy’s Law came into play. We had a perfect storm with the power and fuel problems, but they were handled very well and everyone worked as a team. The resilience of everyone involved was very impressive.”
Were your company’s operations affected by Hurricane Sandy? In addition to leaving your stories in the comments below, the IEEE History Center has created a page for members to submit their first-hand accounts of Hurricane Sandy.
The views expressed in this blog are solely those of the bloggers and do not represent official positions of The Institute or IEEE.
Photo: Mandel Ngan/AFP/Getty Images