For a long time we've known backing up our largest volume (3.5T) was
slow. More recently I've been investigating why and it seems like a
problem with only that shelf or possibly aggregate. Basically it is
several times slower than any other shelf/aggregate we have, it
seems bottlenecked whether I am reading/writing from nfs, ndmp,
reallocate scans, etc, that shelf is always slower. I will probably
have a support case opened tomorrow with netapp but I feel like
checking with the list to see what else I can find out on my own.
When doing NDMP backups I get only around 230Mbit/sec as opposed to
800+ on others. The performance drops distinctly on the hour
probably for snapshots (see pic). Details below. 0c.25 seems like
a hot disk but the activity on that aggr also seems too high since
the network bandwidth is fairly small. A 'reallocate measure' on
the two large volumes on aggregate hanksata0 both return a score of
'1'.
I guess my two main questions are, how do I figure out what is
causing the activity on hanksata0 (especially the hot disk which is
sometimes at 100%) and if its not just activity but an actual
problem, how could I further debug the slow performance to find out
what items are at fault?
I used ndmpcopy to copy a fast volume with large files from another
filer to a new volume on hanksata0 and hanksata1. The volume on
hanksata0 is slow but the one on hanksata1 is not. Both of those
aggregates are on the same loop with hanksata1 terminating it.
Sun Feb 28 20:14:20 EST [hank: wafl.scan.start:info]: Starting WAFL
layout measurement on volume scratchtest.
Sun Feb 28 20:19:01 EST [hank: wafl.reallocate.check.value:info]:
Allocation measurement check on
'/vol/scratchtest' is 2.
^^^ almost 5 minutes!
Sun Feb 28 20:13:38 EST [hank: wafl.scan.start:info]: Starting
WAFL layout measurement on volume scratchtest2.
Sun Feb 28 20:14:12 EST [hank: wafl.reallocate.check.value:info]:
Allocation measurement check on
'/vol/scratchtest2' is 1.
^^^ less than 1 min
When I write to scratchtest, you can see the network bandwidth jump
up for a few seconds then it stalls for about twice as long,
presumably so the filer can catch up writing, then it repeats.
Speed averages around 30-40MB/sec if that.
I even tried using the spare sata disk from both of these shelves
to make a new volume, copied scratchtest to it (which took 26
minutes for around 40G), and reads were equally slow as the existing
scratchtest, although I'm not sure if thats because a single disk is
too slow to prove anything, or theres a shelf problem.
hanksata0 6120662048 6041632124 79029924 99%
hanksata0/.snapshot 322140104 14465904 307674200 4%
hanksata1 8162374688 2191140992 5971233696 27%
hanksata1/.snapshot 429598664 39636812 389961852 9%
hanksata0 and 1 are both ds14mk2 AT but hanksata0 has
X268_HGEMI aka X268A-R5 (750m x 14) and hanksata1 has
disks X269_HGEMI aka X269A-R5 (1T x 14). hanksata0 has
been around since we got the filer say around 2 years ago,
hanksata1 was added within the last half year. Both
shelves have always had 11 data disks, 2 parity, 1 spare,
the aggregates were never grown.
volumes on hanksata0 besides root (all created over a year ago):
volume 1 (research):
NO dedupe (too big)
10 million inodes, approx 3.5T, 108G in snapshots
endures random user read/write but usually fairly light traffic.
Populated initially with rsync then opened to user access via NFS.
Sun Feb 28 21:38:11 EST [hank: wafl.reallocate.check.value:info]:
Allocation measurement check on '/vol/research' is 1.
volume 2 (reinstallbackups):
dedupe enabled
6.6 million files, approx 1.6T, 862G in snapshots
volume created over a year ago and has several dozen gigs of windows
PC backups written or read multiple times per week using CIFS but
otherwise COMPLETELY idle. Older data is generally deleted to
snapshots after some weeks and the snapshots expire after a few weeks.
Only accessed via CIFS.
Mon Mar 1 12:15:58 EST [hank: wafl.reallocate.check.value:info]:
Allocation measurement check on '/vol/reinstallbackups' is 1.
hanksata1 only has one volume besides the small test ones I made,
it runs plenty fast.
dedupe enabled
4.3 million files, approx 1.6T, 12G in snapshots
created a few months ago on an otherwise unused new aggregate with
initial rsync,
then daily rsyncs from another fileserver that is not very active
disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs
/hanksata0/plex0/rg0:
0c.16 7 5.69 0.94 1.00 55269 3.22 3.02 2439 1.52 2.71 579 0.00 .... . 0.00 .... .
0c.17 9 6.34 0.94 1.00 74308 3.84 2.86 2228 1.56 2.93 873 0.00 .... . 0.00 .... .
0c.18 63 121.00 118.86 1.01 30249 1.38 3.26 3516 0.76 5.43 2684 0.00 .... . 0.00 .... .
0c.19 60 117.74 116.69 1.00 30546 0.40 3.73 5049 0.65 5.56 2840 0.00 .... . 0.00 .... .
0c.20 60 120.82 119.66 1.02 29156 0.43 5.33 5469 0.72 4.80 3583 0.00 .... . 0.00 .... .
0c.21 60 119.37 118.25 1.02 29654 0.36 4.60 5870 0.76 5.76 3140 0.00 .... . 0.00 .... .
0c.22 62 124.87 123.32 1.02 29423 0.62 5.65 5677 0.94 3.58 2710 0.00 .... . 0.00 .... .
0c.23 62 119.48 118.35 1.03 30494 0.36 4.00 6875 0.76 5.14 3417 0.00 .... . 0.00 .... .
0c.24 61 119.08 117.96 1.02 29981 0.47 6.92 3289 0.65 3.94 2930 0.00 .... . 0.00 .... .
0c.25 93 118.17 116.72 1.03 45454 0.58 4.00 17719 0.87 4.63 11658 0.00 .... . 0.00 .... .
0c.26 61 121.40 120.27 1.04 29271 0.43 7.75 3097 0.69 5.21 2131 0.00 .... . 0.00 .... .
0c.27 59 115.75 114.81 1.03 29820 0.43 5.50 4530 0.51 6.00 3321 0.00 .... . 0.00 .... .
0c.28 63 125.53 124.15 1.01 30302 0.65 6.94 3808 0.72 3.40 5191 0.00 .... . 0.00 .... .
Both sata shelves are on controller 0c attached to two 3040.
Both sata shelves are on controller 0c attached to two 3040.
Raid-DP in 13-disk raid groups so we have 2 parity and one spare
per shelf.
Active-Active single path HA.
Latest firmwares/code as of beginning of the year. 7.3.2.
no VMs, no snapmirror, nothing fancy that I can think of.
wafl scan status only shows 'active bitmap rearrangement' or
'container block reclamation'.
Thanks for thoughts and input!
