Somewhere in the world, someone spills a cup of coffee in a data center. In millions of homes and businesses across the United States, televisions go dark just as Jordan Spieth aims his final putt in the U.S. Open golf tournament. Possible? Absolutely.
To err may be human, so preventing costly business interruptions should include the human factor as well. Even a simple spilled beverage can create a major problem, as happened in a 2015 incident that that knocked out 300,000 Bloomberg data terminals.
In fact, an estimated 22 percent of data center outages in 2015 were attributed to human error—not including cyber-crime. The price is high: the average cost of a data center outage rose to $740,357 last year —38 percent higher than in 2010 — according to a Ponemon Institute Study.
“Having a properly designed, constructed and commissioned data center building and infrastructure is the starting point in reducing human error,” observes John Walters, National Director and Senior Vice President, Project and Development Services, JLL. “The project teams that produce these critical buildings need to be experienced in the unique nature of how they need to be operated and maintained after the keys are handed over.”
The most common human errors
Accidents can happen in any work environment, but they’re especially harmful in mission-critical workplaces involving high-intensity electrical fixtures, costly equipment and sensitive materials—including corporate networks and data, scientific research or high-tech inventions in progress. Forward-looking data center designers and managers have a lot to teach the rest of the world about preventing costly or dangerous mishaps.
- Whoops! Watch out for that cable
Simply tripping over a cable can disrupt operations, whether by inadvertently unplugging a computer from the IT network or by cutting power for an entire set of equipment—or worse.
“Appropriate decisions and attention to detail in design and construction will go a long way in assuring reduced impact from human error,” says Walters. “Having a well trained staff is crucial in maintaining the uptime of critical infrastructure systems after a data center goes in to live operations.”
- Missed connections
How often have you set up a new computer or entertainment center, only to find that nothing works because the cables all look alike, and the wrong cable is in the wrong place? A 2013 Emerson Network Power survey found that 15 percent of data center outages stemmed from “Improperly deployed server/hardware in a move, add or change,” which can happen when equipment and cables are poorly labeled. While missed connections at home are frustrating, in a critical environment the error can be costly.
- Keeping systems on full power 24/7
As computers become more powerful and energy-intensive, electrical safety risks are growing. Pressure for continuous operation in high-intensity data centers or industrial plants contributes to the risk, when employees are reluctant to power down critical systems for maintenance and upgrades.
Yet, “hot” equipment can be dangerous to workers and also create significant business interruption if a disaster occurs. A particular kind of electrical incident, known as an arc flash, generates heat as high as 35,000 degrees—and can easily arise from dust, a dropped tool, accidental touching, corrosion or faulty installation involving live wires.
“It’s essential to have documented and established standard operating procedures, emergency operating procedures and methods of procedures,” notes Walters. “The proactive and predictive maintenance of the power, cooling and other building systems coupled with disciplined staff and procedures makes all the difference.”
- Power out! Accident or sabotage?
In data centers and other mission-critical environments, it’s common to see an “emergency power off” (EPO) button. One press makes it easy for anyone to shut down a particular piece of equipment, a production line—or an entire facility. This button may be poorly labeled and unprotected, which explains why unintended EPO accounted for 48 percent of data center outages in 2013. More effective is a protective case or, better yet, remove the button altogether by segregating power access in an electrical closet.
Managing smart buildings
As smart building technology becomes more prevalent, it’s reducing the amount of hands-on work needed to keep a critical facility in tip-top shape. When smart building and other systems are integrated to monitor the entire critical environment, potential disasters stemming from overloaded power circuits or failed cooling systems that can bring down sensitive equipment can be avoided. This cohesive approach enables systems to monitor and provide real-time alerts and recommended actions to prevent power interruptions, overheating and other scenarios that can cause a costly power outage or equipment failure.
While it’s not possible to completely eliminate the risks of human hazards, smart building design along with good facility management and operational practices can help to prevent the power going off when the pressure is on. . .
