Today's award goes to the operator(s) of a data center in Sweden. A tip o' the hat to reader Snoggeramus for sending me the link.
Having worked in the information technology industry for a decade or so, rising from computer (mainframe) operator, through programming and systems analysis, to manage a department and then be a director of a small IT company, I'm pretty familiar with commercial computer operations. This was entirely preventable, and should have been foreseen.
A loud sound emitted by a fire suppression system has destroyed the hard drives of a Swedish data center, downing Nasdaq operations across Northern Europe.
The incident took place in the early hours of Wednesday, April 18, and was caused by a gas-based fire suppression system that is typically deployed in data centers because of their ability to put out fires without destroying non-burnt equipment.
These systems work by releasing inert gas at high speeds, a mechanism usually accompanied by a loud whistle-like sound. With non-calibrated systems, this sound can get very loud, a big no-no in data centers, where loud sounds are known to affect performance, shut down, or even destroy hard drives.
The latter scenario is what happened on Wednesday night, as the sound produced by the errant release of the inert gas destroyed hard drives for around a third of the Nasdaq servers located in the Digiplex data center.
. . .
A Digiplex spokesperson told Bleeping Computer that Nasdaq only rents space in the data center, and uses its own equipment. Nasdaq said there weren't enough servers in the whole of Sweden to replace the destroyed ones, and had to import new machines.
There's more at the link.
Yes, loud noises can be devastating to computer disks. Have you ever seen a really loud woofer at full volume on the back shelf of a car? The speakers are vibrating in and out, shaking the entire vehicle. Do that to a disk drive while its heads are reading or writing data, and they'll crash into the disk surface, scratching it and damaging the read/write heads. Bye-bye, disk drive. If Nasdaq had to import servers, because "there weren't enough ... in the whole of Sweden to replace the destroyed ones", that must have been a very loud noise next to a very large number of server units.
Unfortunately, even though they should know better, sometimes that sort of known problem is overlooked when other priorities are pressing. I remember when a new fire suppression system was fitted to the mainframe computer center of a South African oil company, where I was employed at the time as a computer operator. I looked at the emergency masks, designed to allow operators to exit the room in the event of a fire. They were all smoke inhalation masks, designed to take particulates out of the air so one could breathe freely. I pointed out to the Operations Manager that halon, the gas used in our new fire suppression system, actually made it impossible to breathe at all. It was as if all the oxygen had been removed from the air. In such circumstances, particulate filters would do nothing at all to save our lives. Smoke or not, we needed something to breathe! The offending masks were replaced with respirators within a day, each with a small self-contained cylinder of oxygen, enough for up to five minutes, to let us get out alive. We called that an improvement . . . again, something that should have been foreseen, but was overlooked due to pressure of other factors.