;)
Had a humorous thought...we've had to replace a lot of motherboards and memory in the last 2 weeks for various reasons, and I've been getting pretty good at it. I was looking back at the logs at the actual downtime for these various tasks. My best times so far have been, where "downtime" is measured from the filer messgages:
System shut down with "halt" command.
to
NetApp Release 5.w.xRyPz boot complete.
memory swap: 3 minutes, 57 seconds [0] motherboard swap: 15 minutes, 54 seconds [0] [1]
I've broken the "4-minute mile" for the former, but not the "15 minute" mark for the latter. Next time.
Or, maybe a contestant on "Fix That Filer" ala "Name That Tune":
FilerAdmin1: I can fix that filer in 6 minutes. FilerAdmin2: I can fix that filer in 5 minutes. FilerAdmin3: I can fix that filer in 4 minutes. Wink Martindale: Okay, FilerAdmin3....fix that filer!
Okay, time for some breakfast and some serious work now...
[0] This includes a 32 second "penalty" for upping 4 links on a vif. [1] Three PCI cards: NVRAM, FCAL, quad ethernet
Until next time...
The Mathworks, Inc. 508-647-7000 x7792 3 Apple Hill Drive, Natick, MA 01760-2098 508-647-7001 FAX tmerrill@mathworks.com http://www.mathworks.com ---
On Thu, 13 Jul 2000, Todd C. Merrill wrote:
memory swap: 3 minutes, 57 seconds [0] motherboard swap: 15 minutes, 54 seconds [0] [1]
I've broken the "4-minute mile" for the former, but not the "15 minute" mark for the latter. Next time.
But do you loosen the screws before or after the clock starts... :)
-Ronan
On Thu, 13 Jul 2000, Ronan Mullally wrote:
But do you loosen the screws before or after the clock starts... :)
I don't know about Todd but I always have a few loose screws. :)
But do you loosen the screws before or after the clock starts... :)
A better question is - do you actually screw the PCI cards into the slot? You can skip that step and save lotsa time (although you are more vulnerable to a bad connection caused by someone bumping the filer).
I assume power tools are allowed....
Bruce
On Thu, 13 Jul 2000, Bruce Sterling Woodcock wrote:
But do you loosen the screws before or after the clock starts... :)
A better question is - do you actually screw the PCI cards into the slot? You can skip that step and save lotsa time (although you are more vulnerable to a bad connection caused by someone bumping the filer).
I assume power tools are allowed....
The only tools you are allowed are a grounding strap and a Leatherman.
*All* screws *inside* the box are unsecured and secured in the timed interval. <hint,hint> The ones inside your head should definitely be loose, all the time.
No connections outside the box, redundant or otherwise, can be broken until the console port shows an "ok" prompt after a proper shutdown. Depending on what pieces are being changed, you don't want to loose any data potentially on the NVRAM card!
Hmmm...if these Olympics get international recognition, there are going to have to be classes of events. It's no fair to people who have multiple FCAL cards, with cluster failover cards, gig and quad ethernets cards, etc. There can be a 3 PCI card event, etc. ;)
First prize? How about an F8xx? ;) A T-shirt for the gold, silver, and bronze winners at least.
Until next time...
The Mathworks, Inc. 508-647-7000 x7792 3 Apple Hill Drive, Natick, MA 01760-2098 508-647-7001 FAX tmerrill@mathworks.com http://www.mathworks.com ---
1.) the shutdown procedure should be included in the timed event.
2.) No hardware modification before the shutdown procedure is started. Hint, once you hit the enter key, begin removing the screws.
3.) New classes; one man, two man, and team events not to exceed four team members. Kind of like the difference between luge and bobsledding. :)
-gdg
"Todd C. Merrill" wrote:
The only tools you are allowed are a grounding strap and aLeatherman.
*All* screws *inside* the box are unsecured and secured in thetimed interval. <hint,hint> The ones inside your head should definitely be loose, all the time.
No connections outside the box, redundant or otherwise, can bebroken until the console port shows an "ok" prompt after a proper shutdown. Depending on what pieces are being changed, you don't want to loose any data potentially on the NVRAM card!
Hmmm...if these Olympics get international recognition, thereare going to have to be classes of events. It's no fair to people who have multiple FCAL cards, with cluster failover cards, gig and quad ethernets cards, etc. There can be a 3 PCI card event, etc. ;)
First prize? How about an F8xx? ;) A T-shirt for the gold,silver, and bronze winners at least.
Until next time...The Mathworks, Inc. 508-647-7000 x7792 3 Apple Hill Drive, Natick, MA 01760-2098 508-647-7001 FAX tmerrill@mathworks.com http://www.mathworks.com
-- --------------------------------------------------------------- G D Geen mailto:geen@ti.com Texas Instruments Phone : (214)480.7896 System Administrator FAX : (214)480.7676 --------------------------------------------------------------- Life is what happens while you're busy making other plans. -J. Lennon
On Thu, 13 Jul 2000, G D Geen wrote:
3.) New classes; one man, two man, and team events not to exceed four team members. Kind of like the difference between luge and bobsledding. :)
I can see it now...maintenance in the server rooms will start looking like a pit stop at an automobile race. Who's the lucky sap whose sole job it is to squeegie the LCD display? ;)
Until next time...
The Mathworks, Inc. 508-647-7000 x7792 3 Apple Hill Drive, Natick, MA 01760-2098 508-647-7001 FAX tmerrill@mathworks.com http://www.mathworks.com ---
Personally, if I had enough experience replacing motherboards that I could brag about how fast I could replace them, I'd be looking for a new vendor. Then again, maybe he was in a third-party support company or runs hundreds of filers.
Frank
--- Frank Smith fsmith@hoovers.com Systems Administrator Voice: 512-374-4673 Hoover's Online Fax: 512-374-4501
.... perhaps first prize should be a job offer from NetApp? ;)
Cheers, Derek Kelly Genomecorp
On 7/13/00 at 8:24 AM, tmerrill@mathworks.com (Todd C. Merrill) wrote:
;)
Had a humorous thought...we've had to replace a lot of motherboards
and
memory in the last 2 weeks for various reasons, and I've been getting pretty good at it. I was looking back at the logs at the actual downtime for these various tasks. My best times so far have been,
where
"downtime" is measured from the filer messgages:
System shut down with "halt" command.
to
NetApp Release 5.w.xRyPz boot complete.
memory swap: 3 minutes, 57 seconds [0] motherboard swap: 15 minutes, 54 seconds [0] [1]
I've broken the "4-minute mile" for the former, but not the "15
minute"
mark for the latter. Next time.
Or, maybe a contestant on "Fix That Filer" ala "Name That Tune":
FilerAdmin1: I can fix that filer in 6 minutes. FilerAdmin2: I can fix that filer in 5 minutes. FilerAdmin3: I can fix that filer in 4 minutes. Wink Martindale: Okay, FilerAdmin3....fix that filer!
Okay, time for some breakfast and some serious work now...
[0] This includes a 32 second "penalty" for upping 4 links on a vif. [1] Three PCI cards: NVRAM, FCAL, quad ethernet
Until next time...
The Mathworks, Inc. 508-647-7000 x7792 3 Apple Hill Drive, Natick, MA 01760-2098 508-647-7001 FAX tmerrill@mathworks.com http://www.mathworks.com
I can just hear management now. 'You want me to buy you a new filer so you can train for WHAT?'
I'll let someone else tell him what I get if I win. :)
.... perhaps first prize should be a job offer from NetApp? ;)
Cheers, Derek Kelly Genomecorp
On Thu, 13 Jul 2000, Todd C. Merrill wrote:
motherboard swap: 15 minutes, 54 seconds [0] [1]
So my guess is that an untrained tech may take twice that, or about 32 minutes. That's good to know, since we bought an entire filer unit so we can unbolt the dead one from the cabinet, swap cards, bolt in the new one, and go. I'm scheduling a filer drill this quarter, and I'll report back the time. ;-) This will make great justification material to management:
"We spent an extra $15000 to have a complete filer ready, reducing our MTTR from 32 minutes to 9 minutes. This saves us 21 minutes of downtime (at $8543.27 per minute), thus saving the company $179408.67 of potential lost business!"
"Great job, Johnson! Take the rest of the week off!"
-
Brian Tao -
Bruce Sterling Woodcock -
David H. Brierley -
Derek Kelly -
Frank Smith -
G D Geen -
Jeff Bryer -
Ronan Mullally -
Todd C. Merrill