No LUNs on the volumes, they are NFS mounts. About 70 VMDK files (50ish VMs). I probably did reduce the volume size at one point, but it would have been before any data was on it. I think I made it 1.25TB when I wanted it 1.2.
Are you talking about FC multipathing? I do have fibre channel hba's in the filer but they are there to take over from it's partner, there are no LUNs on this head at all.
-----Original Message----- From: Uddhav Regmi [mailto:uddhav.regmi@worldnet.att.net] Sent: Friday, June 06, 2008 8:07 AM To: Page, Jeremy; 'Hadrian Baron'; 'Blake Golliher'; 'Blackmor, Chris' Cc: toasters@mathworks.com Subject: RE: How to identify a hot disk
It is more interesting now.... I'm interested to see little bit about " Both aggr are DP with 36 disks each" Is multipathing involved ? So, you confirm that no issues of ownership and mismatch of disks in either path ? and we sure that you created this whole aggregate just one time and you have never changed anything.... And also I hope volume size was not added / reduced in its life cycle.....
If we are all clear on above, I can think of only two things the way vmdk disks behave and a sis
OK, lets dive more here.....how many luns you have inside nfs2 ? and size ?
I have seen some sis issues while making copying / writing - in particular to vmdk files
-----Original Message----- From: Page, Jeremy [mailto:jeremy.page@gilbarco.com] Sent: Friday, June 06, 2008 7:42 AM To: Uddhav Regmi; Hadrian Baron; Blake Golliher; Blackmor, Chris Cc: toasters@mathworks.com Subject: RE: How to identify a hot disk
Very few large files. The ONLY things on this volume are VMs, so the average file size is likely several gig.
The volume has never reached more then 50% full...but I'm getting 45% savings from ASIS, so maybe theats the issue. No errors on the console. 7.2.4.
/vol/nfs3 is the same type of information as /vol/vmware and /vol/nfs2, but obviously I have not run sis on it yet. The problem volume is /vol/nfs2, which is on a different aggr then the others (it's with /shares and /vol0 on the aggr0, the others are on aggr1). Both aggr are DP with 36 disks each. All the disks where added at the same time (before any data was added to the filer).
Filesystem used saved %saved /vol/vmware/ 513GB 492GB 49% /vol/vol0/ 267MB 0MB 0% /vol/shares/ 1770GB 561GB 24% /vol/nfs2/ 268GB 619GB 70% < pretty nice :) /vol/nfs3/ 275GB 0GB 0%
-----Original Message----- From: Uddhav Regmi [mailto:uddhav.regmi@worldnet.att.net] Sent: Thursday, June 05, 2008 9:20 PM To: 'Hadrian Baron'; Page, Jeremy; 'Blake Golliher'; 'Blackmor, Chris' Cc: toasters@mathworks.com Subject: RE: How to identify a hot disk
what kind of files is coming to this volume after de-dup ( sis ) millions of small files, large chunk of lun files or vmdk files ? have you ever deleted some large files thinking it has reached to its max limit ?
I'm still thinking how this may have happened....
Any messages showing at system console about those hot disks ? which ontap version is this
-uddhav
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Hadrian Baron Sent: Thursday, June 05, 2008 8:39 PM To: 'Page, Jeremy'; Blake Golliher; Blackmor, Chris Cc: toasters@mathworks.com Subject: RE: How to identify a hot disk
I searched disk layout ratio on NOW and found a community post stating it could become bad if your volumes reach 100% usage, did that ever happen?
- Hadrian
-----Original Message----- From: Page, Jeremy [mailto:jeremy.page@gilbarco.com] Sent: Thursday, June 05, 2008 5:38 PM To: Blake Golliher; Blackmor, Chris Cc: Hadrian Baron; toasters@mathworks.com Subject: RE: How to identify a hot disk
I don't understand how the fragmentation occurred in the first place, this is a brand new filer, we've not even had any snapshots age out yet. All the disks where added to the aggr at the same time, lots of free space.
The only thing odd is that we did have ASIS run, I wonder if the fragmentation is from all the stuff getting de-duped (it's a bunch of VMs)
-----Original Message----- From: Blake Golliher [mailto:thelastman@gmail.com] Sent: Thursday, June 05, 2008 5:17 PM To: Blackmor, Chris Cc: Hadrian.Baron@vegas.com; Page, Jeremy; toasters@mathworks.com Subject: Re: How to identify a hot disk
It'll depend on how busy your disks are during prime time. Safety first people!
On Thu, Jun 5, 2008 at 1:38 PM, Blackmor, Chris Chris.Blackmor@amd.com wrote:
My understanding of reallocate is that it will put a definite load on
your
filer and shouldn't be run during prime time.
----- Original Message ----- From: owner-toasters@mathworks.com owner-toasters@mathworks.com To: 'Page, Jeremy' jeremy.page@gilbarco.com; toasters@mathworks.com toasters@mathworks.com Sent: Thu Jun 05 10:57:12 2008 Subject: RE: How to identify a hot disk
It's good you are at 50% aggr usage, as you'll need 50% free space in
each
volume you run the reallocate on. I think running the reallocate is
the
best first step as it is fairly un-intrusive and you can run it during
the
day unless you are hammering the filer constantly. When we run it we
use
the parameter -f to force reallocation without caring how well it is
already
laid out. Not sure about your question on A-SIS.
HTH,
- Hadrian
From: owner-toasters@mathworks.com
[mailto:owner-toasters@mathworks.com] On
Behalf Of Page, Jeremy Sent: Thursday, June 05, 2008 7:13 AM To: toasters@mathworks.com Subject: RE: How to identify a hot disk
Thu Jun 5 10:06:02 EDT [gvr-array02: wafl.scan.layout.advise:info]:
WAFL
layout ratio for volume nfs2 is 4.01. A ratio of 1 is optimal. Based
on your
free space, 1.42 is expected.
Would you say I need to do a reallocate? I'm not sure why this is so fragmented, this file system has never been more then 50% full, could
A-SIS
have something to do with it?
Jeremy M. Page____________________
Systems Architect
- email:Jeremy.Page@gilbarco.com - ( phone: 336.547.5399 - 6 fax:
336.547.5163 - ( cell: 336.601.7274
From: Uddhav Regmi [mailto:uddhav.regmi@worldnet.att.net] Sent: Thursday, June 05, 2008 9:38 AM To: Page, Jeremy; toasters@mathworks.com Subject: RE: How to identify a hot disk
hmmm
very interesting....
looks like those are max out
do wafl scan measure layout and see where you stand....
if needed do reallocate.....
I have seen hundreds of cases where it helped a lot
-uddhav
From: owner-toasters@mathworks.com
[mailto:owner-toasters@mathworks.com] On
Behalf Of Page, Jeremy Sent: Thursday, June 05, 2008 8:27 AM To: toasters@mathworks.com Subject: Re: How to identify a hot disk
Not sure why but I have two disks that are maxed out while the rest
are far
lower utilization. What would cause this, the raid groups where
created all
at the same time, there are 10 disks per raid group and 3 groups in
the
aggragate.
/aggr0/plex0/rg0:
1c.16 2 0.94 0.18 1.00 42250 0.49 12.18 1396
0.27
11.83 521 0.00 .... . 0.00 .... .
1c.32 6 2.56 0.49 1.00 104545 1.80 4.18 1323 0.27 11.83 634 0.00 .... . 0.00 .... .
1c.48 98 116.05 114.20 1.62 18481 1.53 4.24 1743
0.31
11.00 2649 0.00 .... . 0.00 .... .
1c.17 41 55.21 54.76 2.17 5697 0.27 20.17 1537
0.18
16.25 600 0.00 .... . 0.00 .... .
1c.33 49 72.07 71.62 1.84 4760 0.27 20.67 871
0.18
16.75 1239 0.00 .... . 0.00 .... .
1c.49 45 61.42 60.97 2.18 4047 0.22 23.00 913
0.22
14.40 931 0.00 .... . 0.00 .... .
1c.64 5 117.40 116.86 1.61 307 0.36 15.25 1336
0.18
18.00 319 0.00 .... . 0.00 .... .
1c.65 4 71.76 71.17 1.83 292 0.27 20.67 1298
0.31
10.43 370 0.00 .... . 0.00 .... .
1c.80 4 49.50 49.05 2.36 333 0.22 22.80 1333
0.22
14.60 712 0.00 .... . 0.00 .... .
1c.81 5 95.14 94.69 1.71 292 0.22 22.80 1325
0.22
14.60 548 0.00 .... . 0.00 .... .
/aggr0/plex0/rg1:
1c.66 1 0.67 0.00 .... . 0.31 19.29 1311
0.36
10.50 238 0.00 .... . 0.00 .... .
1c.83 1 0.67 0.00 .... . 0.31 19.29 1415
0.36
10.50 190 0.00 .... . 0.00 .... .
1c.82 4 48.60 48.24 2.21 320 0.22 22.80 1553
0.13
21.33 234 0.00 .... . 0.00 .... .
1c.19 51 67.89 67.44 1.92 5315 0.22 22.80 1281
0.22
13.20 788 0.00 .... . 0.00 .... .
1c.18 55 72.75 72.34 1.90 4996 0.22 23.00 1122
0.18
16.00 1109 0.00 .... . 0.00 .... .
1c.35 30 36.10 35.52 2.67 3190 0.31 15.86 1802
0.27
11.33 588 0.00 .... . 0.00 .... .
1c.34 41 52.97 52.43 2.04 4207 0.31 16.29 1570
0.22
13.60 750 0.00 .... . 0.00 .... .
1c.51 100 119.82 119.46 1.57 25873 0.22 22.80 1588
0.13
21.33 2313 0.00 .... . 0.00 .... .
1c.50 59 72.03 71.44 1.83 7750 0.27 19.33 1233
0.31
10.86 1289 0.00 .... . 0.00 .... .
1c.67 4 94.60 94.15 1.68 279 0.18 24.50 1071
0.27
12.33 338 0.00 .... . 0.00 .... .
/aggr0/plex0/rg2:
1c.85 1 0.94 0.00 .... . 0.54 12.00 1806
0.40
10.11 538 0.00 .... . 0.00 .... .
1c.84 1 0.94 0.00 .... . 0.54 12.00 1729
0.40
10.11 495 0.00 .... . 0.00 .... .
1c.21 47 67.40 66.90 1.86 4701 0.27 19.17 1452
0.22
14.20 845 0.00 .... . 0.00 .... .
1c.20 56 73.56 73.02 1.79 4866 0.18 25.50 1039 0.36 9.63 870 0.00 .... . 0.00 .... .
1c.37 51 68.79 68.21 1.76 6072 0.27 19.17 1174
0.31
11.43 988 0.00 .... . 0.00 .... .
1c.36 42 50.85 50.18 2.31 3807 0.40 12.78 1852
0.27
13.33 800 0.00 .... . 0.00 .... .
1c.53 59 75.85 75.18 1.86 5024 0.36 14.25 2237
0.31
10.43 493 0.00 .... . 0.00 .... .
1c.52 18 21.27 20.77 3.83 2205 0.27 19.17 1496
0.22
14.20 465 0.00 .... . 0.00 .... .
1c.69 5 71.76 71.00 1.96 296 0.40 12.78 2087 0.36 9.63 610 0.00 .... . 0.00 .... .
1c.68 5 71.58 71.13 1.84 352 0.22 23.60 1314
0.22
14.80 514 0.00 .... . 0.00 .... .
This message (including any attachments) contains confidential and/or proprietary information intended only for the addressee. Any unauthorized disclosure, copying, distribution or reliance on the contents of this information is strictly prohibited and may constitute a violation of law. If you are not the intended recipient, please notify the sender immediately by responding to this e-mail, and delete the message from your system. If you have any questions about this e-mail please notify the sender immediately.
This message (including any attachments) contains confidential and/or proprietary information intended only for the addressee. Any unauthorized disclosure, copying, distribution or reliance on the contents of this information is strictly prohibited and may constitute a violation of law. If you are not the intended recipient, please notify the sender immediately by responding to this e-mail, and delete the message from your system. If you have any questions about this e-mail please notify the sender immediately.
This message (including any attachments) contains confidential and/or proprietary information intended only for the addressee. Any unauthorized disclosure, copying, distribution or reliance on the contents of this information is strictly prohibited and may constitute a violation of law. If you are not the intended recipient, please notify the sender immediately by responding to this e-mail, and delete the message from your system. If you have any questions about this e-mail please notify the sender immediately.
This message (including any attachments) contains confidential and/or proprietary information intended only for the addressee. Any unauthorized disclosure, copying, distribution or reliance on the contents of this information is strictly prohibited and may constitute a violation of law. If you are not the intended recipient, please notify the sender immediately by responding to this e-mail, and delete the message from your system. If you have any questions about this e-mail please notify the sender immediately.
This message (including any attachments) contains confidential and/or proprietary information intended only for the addressee. Any unauthorized disclosure, copying, distribution or reliance on the contents of this information is strictly prohibited and may constitute a violation of law. If you are not the intended recipient, please notify the sender immediately by responding to this e-mail, and delete the message from your system. If you have any questions about this e-mail please notify the sender immediately.
-
Page, Jeremy