toasters December 2011

toasters@lists.teaparty.net

30 participants
13 discussions

64 bit root volume conversion.
by Jeff Cleverley 31 Dec '11

31 Dec '11

Greetings, We have a filer (3140) running 8.0.1 and the group wanted to convert it to a 64 bit root aggregate and vol0. I let them practice on my 8.1 test box and everything went as planned. When they did it on their 8.0.1 box, all went as plan except for the assigning the root volume. They had moved the original vol0 out of the way and renamed the new 64 bit volume to vol0. When they ran the "vol options vol0 root" command it gave the following output: >> Necessary system files are not present on the volume 'vol0'. It is recommended that you copy system files from the current root volume to the new root volume using the steps outlined in System Administration Guide, chapter 'Understanding Root Volumes', section 'Changing Root Volume'. If you do not, then after reboot, the appliance will enter setup and service will be disrupted until all the setup steps (including a reinstall of system software) have been completed. Continue with set root operation (y/n)? >> He answered no. We did file listings and verified all the files are in both places. There were no obvious ndmpcopy errors. The size of space used differs a little bit, but I assume the 32 vs 64 bit aggregate layouts would cause some fluff. I even had him destroy the 64 bit vol0 volume and run everything again. Same results. Does anyone know what this is complaining about? Thanks, Jeff -- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611

2 1

ndmp restores
by Ian Dotson 21 Dec '11

21 Dec '11

We've just started using our first Netapp here. One surprise involves backups (we're using NetVault's NDMP plugin). I'll let our local backup guy explain: "We're setting up our new netapp, have the ndmp plugin in place and can back up ok. But I don't see how to set up backups of netapp filer volumes so that I can do restores in the same way we can do restores from windows file servers. Specifically, the ndmp restore window only shows the files backed up in that specific dump, so if I want to restore all files that existed in a directory up to that date, it appears I have to separately do a restore of the level 0 dump, then all pertinent level 1 dumps, then all pertinent level 2 dumps, etc. With windows file restores, I could restore the whole directory with one job. How do I set up the netapp ndmp to work like that? " The NetVault folks reply that "the NDMP protocol doesn't allow for that. Each restore builds on the previous restore, so you have to do Level 0, Level 1, etc.. It would be the same with any filer that utilizes the NDMP protocol" -- which doesn't seem right at all. So, is this indeed the same for those using NetBackup, etc.? Are you forced to restore each dump level individually? Thanks, Ian Dotson

2 2

Flex_scale licensing question.
by Jeff Cleverley 20 Dec '11

20 Dec '11

Greetings, I'm confused about the flex_scale licensing. We have installed PAMII modules in 2 of our clusters running 7.3.x. We have a license for pamii, but not for flex_scale. To manage the extended cache, it has us set flexscale.xxx options. Since we don't have flex_scale licensed, are these options doing any good at all? I didn't know if flex_scale was tied to the old flashcache modules and pamii now incorporates the same thing for the new PAMII cards. Thanks, Jeff -- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611

3 6

Disk space increasing after snapshots are deleted.
by Jeff Cleverley 18 Dec '11

18 Dec '11

Greetings, We have a 8.0.1 filer that runs as a NearStore for SV backups. The group using this normally kept 120 nightly backups. The aggregates were getting full so they decided to delete all snapshots older than 90 days. This was done with a script using snap delete <vol> <snapshot> with a list of snapshots fed into the command. Some of the lists were not in reverse order so it deleted nightly.91 before nightly.119. I know deleting them in this type of order will work anyway, but it causes the system to do more processing. The problem is the disk space is continuing to grow on all volumes in the aggregates. An aggr show_space of the aggregates shows everything increasing in used space. The volumes have no space reservation. I'm trying to figure out how this mass deletion of snapshots (several hundred) is causing an increase in space usage. Any ideas? Thanks, Jeff -- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611

4 6

Installing & Testing Flash Cache cards
by Fletcher Cocquyt 15 Dec '11

15 Dec '11

We are taking delivery of a couple flash cache cards (512Gb demo units) soon. I plan to install them in our standby 3270 cluster then present several cache enabled volumes for various benchmark us cases (Oracle, VMware, Web etc) Qs: 1 - how easy/hard is the physical installation? Any gotchas to avoid? 2 - can I enable / disable caching on the volumes to help define the benefit flash cache is providing? 3 - how do I best monitor the flash cache metrics (I use cacti to graph "Cache Age" currently) thanks -- Fletcher Cocquyt Principal Engineer Information Resources and Technology (IRT) Stanford University School of Medicine Email: fcocquyt(a)stanford.edu Phone: (650) 724-7485

5 6

flexcache and cold caches
by Peter D. Gray 15 Dec '11

15 Dec '11

Hi toasters, we have just recently "upgraded" our netapp infrastructure and moved from lots of small FC drives, to a much smaller number of SATA drives. We knew this would cause a performance decrease and compensated by installing 512GB flexcache cards. Its a HA 3170A. So far it all kind of works. However, its clear that the caches take a long time to heat up, several hours in fact. During that time, performance is "average" to say the least. So, I guess I have 2 questions for people who are doing similar things: 1) are there any tips on ways to improve the cold start performance (I cannot think of any). 2) if I have a HA failover, I imagine my performance will be even worse since the second heads cache will be completely cold for the data from the failed head and to make things worse, the second head will have to eject data from the cache for its own local volumes. Anybody with experience of this? 3) Any bright ideas other than more spindles? It would be nice if the caches were nvram, to avoid the cold start problem. I really netapp should use SSD's for the cache for exactly this reason. Or am I missing something? Regards, pdg

4 5

Re: 64 bit root volume conversion.
by Jeff Cleverley 15 Dec '11

15 Dec '11

>From what I can see from the audit logs they ran "ndmpcopy /vol/old_vol0 /vol/vol0" after they had renamed the original vol0 to old_vol0. I checked the audit logs on the test system and they did the ndmpcopy from vol0 to new_vol0, then did the rename. Would running the ndmpcopy when the original vol0 had been renamed first affect any of the files or the copy somehow? I wouldn't expect so, but I'll have him try it again in the morning. Jeff > > On Dec 14, 2011, at 2:10 PM, Jeff Cleverley <jeff.cleverley(a)avagotech.com> wrote: > >> Greetings, >> >> We have a filer (3140) running 8.0.1 and the group wanted to convert >> it to a 64 bit root aggregate and vol0. I let them practice on my 8.1 >> test box and everything went as planned. When they did it on their >> 8.0.1 box, all went as plan except for the assigning the root volume. >> >> They had moved the original vol0 out of the way and renamed the new 64 >> bit volume to vol0. When they ran the "vol options vol0 root" command >> it gave the following output: >> >>>> >> Necessary system files are not present on the volume 'vol0'. It is >> recommended that you copy system files from the current root volume to >> the new root volume using the steps outlined in System Administration >> Guide, chapter 'Understanding Root Volumes', section 'Changing Root >> Volume'. If you do not, then after reboot, the appliance will enter >> setup and service will be disrupted until all the setup steps >> (including a reinstall of system software) have been completed. >> Continue with set root operation (y/n)? >>>> >> >> He answered no. We did file listings and verified all the files are >> in both places. There were no obvious ndmpcopy errors. The size of >> space used differs a little bit, but I assume the 32 vs 64 bit >> aggregate layouts would cause some fluff. I even had him destroy the >> 64 bit vol0 volume and run everything again. Same results. >> >> Does anyone know what this is complaining about? >> >> Thanks, >> >> Jeff -- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611

3 7

System Manager for 8.1
by Jeff Cleverley 12 Dec '11

12 Dec '11

Greetings, I've got 8.1RC1 installed on a test filer. FilerView went away with 8.1 7-mode and we now have to use System Manager. I have 2 questions about this: 1. Has anyone found a hack to make FilerView work such as copying over files or directories from an 8.x system? 2. If I only want to set up a centralized configuration that everyone could see, would I have to set up a OnCommand Core package on a server? Right now I have to install the rpm on my linux box and a different setup on my PC. I have to tell it to go discover all of the filers for each installation. This makes supporting this a pain for our support team. Everyone has to install it on their PC and desktops in order to be able to manage the 8.1 filers. It really seems a waste to have to do so many installs, and then tell it to go find all the needed filers. I hope there is a better way :-) I'm only looking for FilerView capabilities to create aggregates/volumes/shares, etc. I'm not needing monitoring or other advanced tools right now. Thanks, Jeff -- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611

11 20

Re: limit i/o for lun or volume
by Alon Zeltser 06 Dec '11

06 Dec '11

I run sysstat perfstat cmpg and other performance tools the problem here is not in the writes but in the reads while the user run his stuff I get 1500 iops most of them read which is kind of awsome for only 5 data disks and 2 parity disks so back to back cp is not the issue here But I will try to set the priority for this volume to low maybe it will help Any other ideas on how to limit io ? Fred Grieco <fredgrieco(a)yahoo.com> wrote: Have you run sysstat on the command line of the filer? (sysstat -x 1) Are you getting a lot of "b" deferred back to back type CPs? My understanding is that the filer starts rejecting new scsi requests when there are back-to-back deferred CPs (because there's no where to put them in memory locally). That's when you get resets and errors on the hosts. Flexshare will limit the IOs to that volume and will reduce the deferred CPs, which may prevent the resets and errors. The host itself may not like that though-- there may be a timeout or some other tuning in iscsi that will help with this. ________________________________ From: Alon Zeltser <alonz(a)emet.co.il> To: Fred Grieco <fredgrieco(a)yahoo.com>; "toasters(a)teaparty.net" <toasters(a)teaparty.net> Sent: Tuesday, December 6, 2011 9:43 AM Subject: Re: limit i/o for lun or volume Thank you for replay but as I wrote im aware of flexshare but I don't think it will help since this system is not impacting other systems and not compeating with other vms on resources Its only hurting itself it doing so much iops and getting such bad latency that after few minutes its doing lun reset and the vm freezes and the user procces exit abnormally I'm trying to limit the iops from this system so it wont load the Netapp so much and get better latency I don't care if it will take much longer to run Thanks again Fred Grieco <fredgrieco(a)yahoo.com> wrote: Do you mean that there is a performance issue with that particular VM, or with other systems on the filer that are caused by I/O on that VM? You should be able to use flex share to set a lower priority for a "cache-hogging" and reduce its impact on the other volumes. On the command line, run "priority on." Then run "priority set volume $volumename level=Low" for the problem volume. This will give less priority to that volume in the write cache, free up the cache for other things, and (hopefully) reduce latency on those other volumes. In your case, you'll need to have that RDM in its own volume. ________________________________ From: Alon Zeltser <alonz(a)emet.co.il> To: toasters(a)teaparty.net Sent: Tuesday, December 6, 2011 6:00 AM Subject: limit i/o for lun or volume Hi all i have a situation of a virtual machine running on esx 4.0 over iscsi this machine (while the user is running his stuff) is using a lot of i/o on a very small netapp controller (2020) with only 5 data disks aggr and getting very bad latency and eventually getting a lun reset errors and the machine freeze for a few minutes i have increased the timeouts from the windows side but it doesn't seems to help this application don't need so much i/o and can be run a 5.4k disk on an old laptop but given the resources of netapp it uses them to the fullest it is important to mention that this application is running over MSSQL db and the i/o problem goes to rdm lun my question: is there a way to limit the i/o of this machine from either netapp side /vmware side / windows side / network side i'm aweare of vmware storage i/o control but this is only supported in vsphere 4.1 and an upgrade is not an options right now i'm also aware of flexshare from the netapp side but i don't think giving low priority to this volume will help in this case is there other way you can think of to limit the i/o goes from vmware virtual machine through iscsi to an rdm lun or his hosting volume? thank you _______________________________________________ Toasters mailing list Toasters(a)teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

1 0

Re: limit i/o for lun or volume
by Fred Grieco 06 Dec '11

06 Dec '11

Have you run sysstat on the command line of the filer? (sysstat -x 1) Are you getting a lot of "b" deferred back to back type CPs? My understanding is that the filer starts rejecting new scsi requests when there are back-to-back deferred CPs (because there's no where to put them in memory locally). That's when you get resets and errors on the hosts. Flexshare will limit the IOs to that volume and will reduce the deferred CPs, which may prevent the resets and errors. The host itself may not like that though-- there may be a timeout or some other tuning in iscsi that will help with this. ________________________________ From: Alon Zeltser <alonz(a)emet.co.il> To: Fred Grieco <fredgrieco(a)yahoo.com>; "toasters(a)teaparty.net" <toasters(a)teaparty.net> Sent: Tuesday, December 6, 2011 9:43 AM Subject: Re: limit i/o for lun or volume Thank you for replay but as I wrote im aware of flexshare but I don't think it will help since this system is not impacting other systems and not compeating with other vms on resources Its only hurting itself it doing so much iops and getting such bad latency that after few minutes its doing lun reset and the vm freezes and the user procces exit abnormally I'm trying to limit the iops from this system so it wont load the Netapp so much and get better latency I don't care if it will take much longer to run Thanks again Fred Grieco <fredgrieco(a)yahoo.com> wrote: Do you mean that there is a performance issue with that particular VM, or with other systems on the filer that are caused by I/O on that VM? You should be able to use flex share to set a lower priority for a "cache-hogging" and reduce its impact on the other volumes. On the command line, run "priority on." Then run "priority set volume $volumename level=Low" for the problem volume. This will give less priority to that volume in the write cache, free up the cache for other things, and (hopefully) reduce latency on those other volumes. In your case, you'll need to have that RDM in its own volume. ________________________________ From: Alon Zeltser <alonz(a)emet.co.il> To: toasters(a)teaparty.net Sent: Tuesday, December 6, 2011 6:00 AM Subject: limit i/o for lun or volume Hi all i have a situation of a virtual machine running on esx 4.0 over iscsi this machine (while the user is running his stuff) is using a lot of i/o on a very small netapp controller (2020) with only 5 data disks aggr and getting very bad latency and eventually getting a lun reset errors and the machine freeze for a few minutes i have increased the timeouts from the windows side but it doesn't seems to help this application don't need so much i/o and can be run a 5.4k disk on an old laptop but given the resources of netapp it uses them to the fullest it is important to mention that this application is running over MSSQL db and the i/o problem goes to rdm lun my question: is there a way to limit the i/o of this machine from either netapp side /vmware side / windows side / network side i'm aweare of vmware storage i/o control but this is only supported in vsphere 4.1 and an upgrade is not an options right now i'm also aware of flexshare from the netapp side but i don't think giving low priority to this volume will help in this case is there other way you can think of to limit the i/o goes from vmware virtual machine through iscsi to an rdm lun or his hosting volume? thank you _______________________________________________ Toasters mailing list Toasters(a)teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters December 2011