I don't think you want to focus on disk utilization during rebuilds. Even
with the low impact setting it is supposed to rebuild as fast as it can,
just give higher priority to regular data traffic.
-----Original Message-----
From: Jerry [mailto:juanino@yahoo.com]
Sent: Friday, October 15, 2004 9:37 AM
To: Derek Lai
Cc: 'toasters@mathworks.com'
Subject: RE: parity drive rebuild causing ls hangs
Writes were bouncing around, but no more than 10
Megabytes/sec. Even went the writes went down to
1MB/sec the util was still burried at 99 or 100%.
I don't think there is much difference between
rebuilding a data disk vs.
parity disk. Unless you are doing lots of writes.
Since the writes will also
have to update the parity disk, they may be
contending with the rebuild. Do
you have any idea how much writes were going on?
Derek
-----Original Message-----
From: Jerry [mailto:juanino@yahoo.com]
Sent: Friday, October 15, 2004 9:02 AM
To: Derek Lai
Cc: 'toasters@mathworks.com'
Subject: RE: parity drive rebuild causing ls hangs
I think there are only 5 72g disks in that raid
group.
Still, I've done this with data disks many times,
and
the rebuild at "medium" is not really noticeable.
We
set it to low during the rebuild, still no effect.
We're not talking about something a performance
metric
picked up, we're talking about a 30-60 sec on just
an
ls (small disk i/o presumeably). Something wasn't
right.
--- Derek Lai Derek.Lai@onyxco.com wrote:
What is your setting on
raid.reconstruct.perf_impact? The default is set
to
medium. You can try to set it to low and the ls
performance might be better.
But keep in mind the rebuilds might take longer.
How many disk is in your raidgroup? Keep in mind
that when the filer is in
rebuild mode, the number of I/O skyrockets. If you
have 8 disks in the
raidgroup and one disk fails, any single I/O
request
for a piece of
information store in that raidgroup is going to
cause 7X the amount of I/O
compared to normal operation. This is not peculiar
to NetApp but a normal
thing for raid.
One other way to get better performance without
having the rebuilds take
longer is to reduce the number of disk you have in
the raidgroup.
Derek
-----Original Message-----
From: Jerry [mailto:juanino@yahoo.com]
Sent: Friday, October 15, 2004 6:55 AM
To: list toasters
Subject: parity drive rebuild causing ls hangs
Anyone ever experience really bad performance when
rebuilding a parity disk? We had a parity disk
fail
on our FAS940 and when it was trying to rebuild
the
disk i/o util went to 100% (observed with
sysstat).
Reads and Writes did not appear high, but I don't
think rebuild traffic effects those numbers.
During this time, "ls" was taking between 30 and
60
seconds (unacceptable). We thought for sure this
couldn't be normal, since we've had disks fail and
rebuild many times. The difference this time is
it
was a parity disk, but I don't think that should
make
a difference other than taking a little longer to
rebuild. Sure enough, after the rebuild was
complete
it started working again. Any opinions?
Jerry
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam
protection around
http://mail.yahoo.com
__________________________________
Do you Yahoo!?
Yahoo! Mail - Helps protect you from nasty viruses.
http://promotions.yahoo.com/new_mail
_______________________________
Do you Yahoo!?
Declare Yourself - Register online to vote today!
http://vote.yahoo.com
Derek Lai