Writes were bouncing around, but no more than 10 Megabytes/sec. Even went the writes went down to 1MB/sec the util was still burried at 99 or 100%.
--- Derek Lai Derek.Lai@onyxco.com wrote:
I don't think there is much difference between rebuilding a data disk vs. parity disk. Unless you are doing lots of writes. Since the writes will also have to update the parity disk, they may be contending with the rebuild. Do you have any idea how much writes were going on?
Derek
-----Original Message----- From: Jerry [mailto:juanino@yahoo.com] Sent: Friday, October 15, 2004 9:02 AM To: Derek Lai Cc: 'toasters@mathworks.com' Subject: RE: parity drive rebuild causing ls hangs
I think there are only 5 72g disks in that raid group. Still, I've done this with data disks many times, and the rebuild at "medium" is not really noticeable. We set it to low during the rebuild, still no effect.
We're not talking about something a performance metric picked up, we're talking about a 30-60 sec on just an ls (small disk i/o presumeably). Something wasn't right.
--- Derek Lai Derek.Lai@onyxco.com wrote:
What is your setting on raid.reconstruct.perf_impact? The default is set
to
medium. You can try to set it to low and the ls performance might be better. But keep in mind the rebuilds might take longer.
How many disk is in your raidgroup? Keep in mind that when the filer is in rebuild mode, the number of I/O skyrockets. If you have 8 disks in the raidgroup and one disk fails, any single I/O
request
for a piece of information store in that raidgroup is going to cause 7X the amount of I/O compared to normal operation. This is not peculiar to NetApp but a normal thing for raid.
One other way to get better performance without having the rebuilds take longer is to reduce the number of disk you have in the raidgroup.
Derek
-----Original Message----- From: Jerry [mailto:juanino@yahoo.com] Sent: Friday, October 15, 2004 6:55 AM To: list toasters Subject: parity drive rebuild causing ls hangs
Anyone ever experience really bad performance when rebuilding a parity disk? We had a parity disk
fail
on our FAS940 and when it was trying to rebuild
the
disk i/o util went to 100% (observed with
sysstat).
Reads and Writes did not appear high, but I don't think rebuild traffic effects those numbers.
During this time, "ls" was taking between 30 and
60
seconds (unacceptable). We thought for sure this couldn't be normal, since we've had disks fail and rebuild many times. The difference this time is
it
was a parity disk, but I don't think that should make a difference other than taking a little longer to rebuild. Sure enough, after the rebuild was
complete
it started working again. Any opinions?
Jerry
Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
__________________________________ Do you Yahoo!? Yahoo! Mail - Helps protect you from nasty viruses. http://promotions.yahoo.com/new_mail
_______________________________ Do you Yahoo!? Declare Yourself - Register online to vote today! http://vote.yahoo.com