British Airways IT failure: experts doubt 'power surge' claim

Experts have questioned British Airways’ claim that this weekend’s catastrophic IT failure was down to a “power surge”, as the company’s chief executive has claimed.

Álex Cruz, BA’s chairman and chief executive, said on Monday that the surge was “so strong that it rendered the back-up system ineffective”. But multiple data centre designers have told the Guardian that a power surge should not be able to bring down a data centre, let alone a data centre and its back-up.

“It’s either bad design or there’s more to the story than just a power surge,” said James Wilman, chief executive of the data centre consultancy Future-tech. “You have something specifically that you build in to a data centre called surge protection, which is there to protect against exactly this incident. You also have an uninterruptible power supply, a UPS, and part of its job is to condition the power” – ie smooth out the peaks and flows in current.

“Between those and a quality earthing system, you should be protected from power surges,” Wilman said.

Andy Hirst, specialist projects director at the data centre builder Sudlows, said it was shocking just how many data centres lacked the resilience to deal with common problems. “A number of failures could have resulted in the downtime. Some organisations invest millions in IT equipment but seem to overlook the infrastructure required to ensure the IT equipment is kept running with no power outages.”

View image in fullscreen

According to a report in the Times, SSE and UK Power Networks, the two electricity companies that provide energy to the area in which BA has its data centre, denied there had been a power surge.

The real problem, said Matthew Bloch, managing director of Bytemark Hosting, was not what happened when the power went off but what happened when it was turned back on. He asked: “When was the last time somebody test-rebooted a crucial database? Or executed a procedure like restoring a server? Was there pressure to say it’ll only take an hour when it’ll really take a day?”

Barry Elliott, director of Capitoline consultants, said power surges were easy to protect against but still a regular problem. “We have had 12 years’ experience in designing and auditing data centres and we’ve seen that after catastrophic IT problems, power problems are the second most common cause of data centre outages.”

The airline industry is notorious for running outdated infrastructure long after standards have improved. In December, for instance, it was revealed that passenger booking systems used by multiple airlines were easy prey for hackers.

Wilman said the sort of outdated infrastructure that could lead to a data centre being brought down by a power surge was unlikely to be limited to the travel industry. “If you saw the amount of old infrastructure that this country is hanging off of, you wouldn’t sleep at night,” he said.

“We were leading the communications curve back 20 years ago, and the problem is that that now means that much of our infrastructure is hanging off a 25-year-old backbone. Some data centres are reaching the end of their life. And how do you refurbish that when you can’t turn it off?”