We recently upgraded our clustered F740 pair to 5.2.3 from 5.2.2. This necessitated an upgrade of the firmware from 2.1_a2 to 2.2_a2, as well as a disk firmware upgrade from FB37 to FB59.
On one of our filers, the first disk scrub that ran after the upgrade (6 days after the upgrade) found a bunch on parity inconsistencies:
Sun Oct 3 02:50:39 EDT [viking: consumer]: Scrub found 212 parity inconsistencies Sun Oct 3 02:50:39 EDT [viking: consumer]: Scrub found 0 media errors Sun Oct 3 02:50:39 EDT [viking: consumer]: Disk scrubbing finished...
Apparently, with 5.2.3 the filer now generates autosupport mail on disk scrubs, so a couple days after this scrub, I received email from NetApp that they had noticed the error. (This is two anomalous autosupport emails in a row that NetApp has opened cases on, so I'd like to acknowledge NA on that.)
NA's recommendation was to re-run a disk scrub (which I did, and it completed w/o finding any errors) and then to run wackz (which I haven't done).
Unforunately, I cannot find _any_ documentation on wackz on NOW. I've also searched the toasters archive for wackz and didn't find much. In particular, I'd like to know how much downtime I'm going to be saddled with. NA claims that wackz can process 25 million inodes/hour. Given:
viking> df -i Filesystem iused ifree %iused Mounted on /vol/cim0a/ 1057771 2593665 29% /vol/cim0a/
that means the wackz should complete in < 10 minutes. From what I've heard about wack, I find this hard to believe (I've heard stories of wack running for 8+ hours). Or I'm missing something?
I'm interested in hearing from anyone who has run wackz on a similar configuration and how long it took to complete. This filer has 21 9GB disks. The cim0a volume is composed of 3 raid groups (5+1, 5+1, 4+1). The vol0 volume is composed of a single raid group (1+1).
BTW - I'm still not clear on the difference between wack and wackz. The previous explanation to toasters (wackz runs faster, does things in an optimized fashion) wasn't very enlightening. wack must do something above and beyond wackz (or the other way around), otherwise why would both still be included in DOT?
RFE: It would be nice if the filer could run a wack (read-only) and report inconsistencies so that I could check the file system w/o downtime. If the filer requires an immutable filesystem to run wack, then DOT should allow you to tag a whole volume as read-only.
I'd like to thank Ron Thibault of NA for his assistance so far.
j. -- Jay Soffian jay@cimedia.com UNIX Systems Engineer 404.572.1941 Cox Interactive Media
Hi,
I will try to address a couple of points, although I am sure that the folks at netapp will give you a much more academic answer:
wackz is similar to wack, but it works on zombie files as well as the regular files (or should I say inodes). You might ask what is a zombie file/inode - great question, I can try and say that it is a file that was semi erased from the system - something marked as a "bad block". I expect NA to come up with a more sophisticated answer. The regular wack command does not check these files/inodes (at least in the current Ontap versions available today).
On the system that we ran wackz we had 9345431 inodes and about 600GB of data it took somewhere in the vicinity of 60 minutes using Ontap 5.3.2. This is a great improvement in time related to the change in Ontap verions and not in the change in the wack command.
I have heard that the wackz command will very likely become default in some future undisclosed Ontap release.
Jay Soffian wrote:
We recently upgraded our clustered F740 pair to 5.2.3 from 5.2.2. This necessitated an upgrade of the firmware from 2.1_a2 to 2.2_a2, as well as a disk firmware upgrade from FB37 to FB59.
On one of our filers, the first disk scrub that ran after the upgrade (6 days after the upgrade) found a bunch on parity inconsistencies:
Sun Oct 3 02:50:39 EDT [viking: consumer]: Scrub found 212 parity inconsistencies Sun Oct 3 02:50:39 EDT [viking: consumer]: Scrub found 0 media errors Sun Oct 3 02:50:39 EDT [viking: consumer]: Disk scrubbing finished...
Apparently, with 5.2.3 the filer now generates autosupport mail on disk scrubs, so a couple days after this scrub, I received email from NetApp that they had noticed the error. (This is two anomalous autosupport emails in a row that NetApp has opened cases on, so I'd like to acknowledge NA on that.)
NA's recommendation was to re-run a disk scrub (which I did, and it completed w/o finding any errors) and then to run wackz (which I haven't done).
Unforunately, I cannot find _any_ documentation on wackz on NOW. I've also searched the toasters archive for wackz and didn't find much. In particular, I'd like to know how much downtime I'm going to be saddled with. NA claims that wackz can process 25 million inodes/hour. Given:
viking> df -i Filesystem iused ifree %iused Mounted on /vol/cim0a/ 1057771 2593665 29% /vol/cim0a/
that means the wackz should complete in < 10 minutes. From what I've heard about wack, I find this hard to believe (I've heard stories of wack running for 8+ hours). Or I'm missing something?
I'm interested in hearing from anyone who has run wackz on a similar configuration and how long it took to complete. This filer has 21 9GB disks. The cim0a volume is composed of 3 raid groups (5+1, 5+1, 4+1). The vol0 volume is composed of a single raid group (1+1).
BTW - I'm still not clear on the difference between wack and wackz. The previous explanation to toasters (wackz runs faster, does things in an optimized fashion) wasn't very enlightening. wack must do something above and beyond wackz (or the other way around), otherwise why would both still be included in DOT?
RFE: It would be nice if the filer could run a wack (read-only) and report inconsistencies so that I could check the file system w/o downtime. If the filer requires an immutable filesystem to run wack, then DOT should allow you to tag a whole volume as read-only.
I'd like to thank Ron Thibault of NA for his assistance so far.
j.
Jay Soffian jay@cimedia.com UNIX Systems Engineer 404.572.1941 Cox Interactive Media
wackz is similar to wack, but it works on zombie files as well as the regular files (or should I say inodes). You might ask what is a zombie file/inode - great question, I can try and say that it is a file that was semi erased from the system - something marked as a "bad block". I expect NA to come up with a more sophisticated answer.
E.g.:
All the file system updates for a delete must be written out in a single consistency point[1], so that the entire delete takes place atomically.
This means that whatever CP (consistency point) will write out those changes cannot complete until the delete finishes.
A CP is what pushes file system changes to disk; until a CP completes, the NVRAM cannot be purged of entries for file system operations that made those changes, as, if the CP doesn't complete before a system reboot (so that none of the file system changes are visible in the file system), all of those operations must be replayed on the reboot.
A delete of a very large file can take quite a while; this means that a CP that would write out the file system changes for said delete can take quite a while, which means it could take quite a while to purge file system operations from the NVRAM.
If the NVRAM is full, this means it could take a while before the system is able to perform any file system operations that get logged to NVRAM - which includes just about every file system operation (including reads - they get logged so that file access time updates don't get lost, at least if you haven't disabled access time updating).
Therefore, a large delete can keep a filer from getting any work done for a while, if the NVRAM backs up.
To fix this, we introduced a mechanism that allows a partially-completed delete to be written out in a CP; the "half-dead" file is a "zombie", and an invisible metadata file keeps track of "zombie" files, which will be deleted if there are any such files detected when the system reboots.
[1] For a brief description of the way "consistency points" work, see
http://www.netapp.com/tech_library/3001.html#I43
which links to part of the WAFL section of the "An NFS File Server Appliance" paper by Dave Hitz in our technical library; the technical library is linked to by the teeny "Tech" link next to "Contact Us" at the bottom of our home page. I guess the idea was not to do technical stuff in public, as it'd frighten the horses^H^H^H^H^H^HPointy-Haired Bosses who probably sign the purchase orders for all the big E-word systems we're selling....
"Jay" == Jay Soffian jay@cimedia.com writes:
Jay> On one of our filers, the first disk scrub that ran after the Jay> upgrade (6 days after the upgrade) found a bunch on parity Jay> inconsistencies:
Some additional questions:
Do parity errors during a disk scrub necessarily indicate something is wrong with the filesystem?
Is there any way to verify the health of a filesystem short of running wack? All outward appearances indicate that the filesystem is okay.
NA is now recommending that I run wackz off of 5.3.1D1 ASAP ('tonight or tomorrow night'). For a device we count on for 100% uptime, I find this rather distressing. The filer really needs a way to perform a filesystem health check w/o downtime. NA also needs to publish accurate timing data on wack via NOW. The only document I can find has ancient data.
j. -- Jay Soffian jay@cimedia.com UNIX Systems Engineer 404.572.1941 Cox Interactive Media
-
Guy Harris -
Jay Soffian -
Menachem Kaiser