I was once at a customer doing an upgrade on a massively parallel database environment. It was about 5 racks of little 1U servers with FCP connectivity. The
timeouts for FCP were set to 0, meaning if you lose one single SCSI command, just reboot. That was legit for local disk, but not SAN.
Ever seen 200 servers reboot in perfect synchronization? I have. I was in the hot aisle, sandwiched between the servers and the storage system. As soon as the
controller rebooted, it got very, very bright as all the LED’s on the backs of the servers lit up during self-test. Then it got very hot and loud as all the self-test progressed and all the fans spun up to maximum speed.
Ive seen that, but at scale.
Was in PS at Netapp, and I was asked to sit-in on a customer moving racks of 500s, 600s, and a few 700s from telco racks to cabinets.
At the time the cabinets were 240v only.
I was in another room...sit-in help as needed, I was not allowed in the DC.
They made a show of 'lets hit the breakers and turn on all 11 cabinets at once'.
Most of them were full of shelves with 120v only power supplies, at the time I think that was "green button" supplies.
Lots of oil..lots of fire.
Replaced 100+ supplies with switching ones, and all was good in the end.
Them was the dayz..
_________________________________
Tech Yahoo, Storage Architect, Principal
Don't stand directly behind it when you turn it on. :-)
Semi-seriously though, I once had a very old original DS14 shelf p/s
(the ones with the vertial bar handles) flame out at me. We'd just
finished re-racking the rig, so I flipped on the breakers and moved
down the row to start on the next one, when the p/s spat out a small
and brief burst of smoke and spark.
It wouldn't have been harmful and probably wouldn't have even left
a mark, but it was still a surprise.
On Wed, Apr 15, 2015 at 04:39:04PM +0000, Jeffrey Mohler wrote:
> This could be fun...and old F760 with say, 4 shelves of 36G drives..
>
> Replace the NVRAM battery, check power supplies for leaking capacitor oil...reseat cards and DIMMS...
>
> What're the odds it'll work, and any other fire-prevention steps to take??????? :)