toasters November 2008

toasters@lists.teaparty.net

54 participants
58 discussions

snapmirror dest volumes have guarantee error after DOT upgrade
by Stephen C. Losen 17 Nov '08

17 Nov '08

Hello Toasters, I upgraded a FAS6040 from 7.2.4 to 7.3. It has several snapmirror destination volumes and now I am getting these errors in the messages file whenever snapmirror runs: Sun Nov 16 00:15:25 EST [wafl.vol.guarantee.fail:error]: Space for volume lib_content0 is NOT guaranteed vol options lib_content0 shows this: nosnap=off, nosnapdir=off, minra=off, no_atime_update=off, nvfail=off, ignore_inconsistent=off, snapmirrored=on, upgraded_replica, create_ucode=off, convert_ucode=off, maxdirsize=62914, schedsnapname=ordinal, fs_size_fixed=on, compression=off, guarantee=volume(disabled), svo_enable=off, svo_checksum=off, svo_allow_rman=off, svo_reject_errors=off, no_i2p=off, fractional_reserve=0, extent=off, try_first=volume_grow, snapshot_clone_dependency=off The above only happens to snapmirror destination volumes. We also have non-snapmirrored volumes in the same aggregate and they do not show this behavior, so it seems to be related to snapmirror. There is plenty of room in the containing aggregate: df -Ah aggr1 aggr1 10TB 7422GB 3006GB 71% aggr1/.snapshot 0GB 0GB 0GB ---% The total size of the volumes in aggr1 is nowhere near filling it. The filer with the snapmirror source volumes is running 7.2.3. Steve Losen scl(a)virginia.edu phone: 434-924-0640 University of Virginia ITC Unix Support

3 5

snapmirror dest volumes have guarantee error after DOT upgrade
by Stephen C. Losen 17 Nov '08

17 Nov '08

1 0

snapmirror dest volumes have guarantee error after DOT upgrade
by Stephen C. Losen 17 Nov '08

17 Nov '08

1 0

Re: Removing a shelf.
by Ray Van Dolson 14 Nov '08

14 Nov '08

On Fri, Nov 14, 2008 at 02:08:34PM -0800, Jeff Mohler wrote: > None..just when that is done do: > > vol options root vol1new, and reboot. Nice. Too bad about the reboot being necessary, but makes sense. Found some KB's on NOW to read regarding this as well. Learning experience! Thanks all, Ray

1 0

Re: FAS6070 OPS
by tmacmd＠gmail.com 14 Nov '08

14 Nov '08

My gx cluster (2x6070 heads) was yielding 2-3 gigaBYTES per second under load Does that help? I see upwards of 5 gBYTES per second with 4 heads (2x6070 and 2x6080) --tmac ------Original Message------ From: admin(a)berkcom.net Sender: owner-toasters(a)mathworks.com To: NetApp Toasters List Subject: FAS6070 OPS Sent: Nov 14, 2008 3:19 PM Anyone out there running a FAS6070 or FAS6030 hard? I have been trying to find real world NFS OPS numbers on this head and cant seem to. I know all OPS are not created equal but a peak OPS and what type of traffic you are pushing at it would be a great help. Thanks Jack *disclaimer* Berkcom.net does not have any personal or business relationship to berkcom.com or Berkeley Communications. Sent from my Verizon Wireless BlackBerry

1 0

options cifs.max_mpx
by Todd C. Merrill 14 Nov '08

14 Nov '08

Anybody consciously set this to something other than the default value of 50? If you have raised it, why did you do so, and more importantly, did you see any negative impact on the *toaster*? We're currently at 253 in one of our environments, and we're being asked to bump it to the maximum recommended value (1124). I've read the NOW articles on the parameter, so no need to explain what it is or how it works. I'm just looking for experience, good or bad, for raising it to the much larger values. This environment already has our Windows machines beating the crap out of a handful of 6070's, and I'm proceeding cautiously before giving each Windows client EVEN MORE connections to the toasters... Until next time... The MathWorks, Inc. 508-647-7000 x7792 3 Apple Hill Drive, Natick, MA 01760-2098 508-647-7001 FAX tmerrill(a)mathworks.com http://www.mathworks.com --- low the deswizzling to complete. We are running 7.2.4, which doesn't have the apparent deswizzle re-start capabilities in later versions. We stopped all regular snapshots, all snapmirrors, and all snapvaults to the volumes in question, and let the deswizzling run from Friday night through the weekend. By Tue morning, it wasn't finished. :( So, the next weekend, we stopped all snapshot activity *and* deleted all but two snapshots on the affected volumes (we had 30-40 prior). That was barely enough...the deswizzle scans finished by 7 AM the following Tuesday. These weren't extraordinary volumes...just user data (home directories) with volume sizes of about 500 GB with a couple million files. So, we are back to normal operating load levels now. Yea! (I think a knowledge base article on NOW is in order for this consequence of running volume snapmirrors. It took us by surprise, and it took the collective knowledge of this mailing list to nail down the answer.) Until next time... The MathWorks, Inc. 508-647-7000 x7792 3 Apple Hill Drive, Natick, MA 01760-2098 508-647-7001 FAX tmerrill(a)mathworks.com http://www.mathworks.com ---

3 2

RE: Removing a shelf.
by Leeds, Daniel 13 Nov '08

13 Nov '08

you are 100% correct. the ability to hot-remove shelves that contain no data would be an incredible value gain. we are currently migrating into a multistore scenario in which we can do planned migrations of all data from one filer cluster to another and free up the original filer for maintenance tasks without any extended downtime or change to the clients in our NFS environments. this is the next best thing while GX merges back into the current 7.x base down the road. we get planned maintenance windows of one hour or less and the ability to run environments without change on a different set of filer heads. ontap upgrades, hardware swaps, environment migrations much less painful and actually doable at this point. we have tons of 72/144gb drives to swap out before their support expiry in 2011 :) good luck to the OP. -- Daniel Leeds Manager, Storage Operations Edmunds, Inc. 1620 26th Street, Suite 400 South Santa Monica, CA 90404 310-309-4999 desk 310-430-0536 cell -----Original Message----- From: Jeff Mohler [mailto:speedtoys.racing@gmail.com] Sent: Thu 11/13/2008 3:27 PM To: Leeds, Daniel Cc: Nils Vogels; Ray Van Dolson; toasters(a)mathworks.com Subject: Re: Removing a shelf. If youre not so lucky, you can lose visibility to an entire random shelf on the system, forcing the rebuild of X missing drives. I have seen this happen, and after a hard shutdown of all devices to re-map the loops, it all comes back, but you're still stuck rebuilding missing drives from the unsupported pull. On Thu, Nov 13, 2008 at 3:03 PM, Leeds, Daniel <dleeds(a)edmunds.com> wrote: > > yes, i neglected to mention we did a clean shutdown shortly after. this was > coordinated as a way to fix an urgent need which it sounded like the > original poster had in regards to sending a loaner shelf back. > > it did function fine without any issues until an approved downtime window > could be scheduled. > > in most of our environments we have zero ability for any downtime scheduled > or otherwise which is really killing us with a current shelf bug causing > drives to randomly go amber and support recommends power cycling the shelf > to correct it + updated firmware--but thats a whole other story. > > --daniel > > -- > Daniel Leeds > Manager, Storage Operations > Edmunds, Inc. > 1620 26th Street, Suite 400 South > Santa Monica, CA 90404 > > 310-309-4999 desk > 310-430-0536 cell > > > > -----Original Message----- > From: Jeff Mohler [mailto:speedtoys.racing@gmail.com] > Sent: Thu 11/13/2008 3:01 PM > To: Leeds, Daniel > Cc: Nils Vogels; Ray Van Dolson; toasters(a)mathworks.com > Subject: Re: Removing a shelf. > > Daniel: > > I would check with support, IIRC, hot removing a shelf is _really_ > frowned upon, and can cause mapping issues that can lead to > _requiring_ you to power down all nodes and shelves to correct. > > Even in failover mode, shelf mapping is persistent. > > https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb40710 > > > > > > >> Daniel Leeds >> Manager, Storage Operations >> Edmunds, Inc. >> 1620 26th Street, Suite 400 South >> Santa Monica, CA 90404 >> >> 310-309-4999 desk >> 310-430-0536 cell >> >> >> >> -----Original Message----- >> From: owner-toasters(a)mathworks.com on behalf of Nils Vogels >> Sent: Thu 11/13/2008 1:16 PM >> To: Ray Van Dolson >> Cc: toasters(a)mathworks.com >> Subject: Re: Removing a shelf. >> >> Hey Ray, >> >> On Thu, Nov 13, 2008 at 9:31 PM, Ray Van Dolson <rvandolson(a)esri.com> >> wrote: >>> >>> An aggregate spans two shelves, one of which is the shelf that needs to >>> be returned. This aggregate contains a FlexVol currently in use. I'd >>> like to preserve the FlexVol but remove the shelf. >> >> If you can afford the downtime, you could offline the aggregate >> concerned, swap the disks into the shelf that can remain, and online >> the aggregate again. Should be a few minutes. >> >> Another way, which does take a lot longer, is to use "disk replace" to >> copy the used disks from their old position to their new ones. No >> downtime, but the disks get copied one at a time, which can be a few >> hours. >> >> HTH & HAND >> >> -- >> Simple guidelines to happiness: >> Work like you don't need the money, >> Love like your heart has never been broken and >> Dance like no one can see you. >> >> > >

1 0

RE: Removing a shelf.
by Leeds, Daniel 13 Nov '08

13 Nov '08

yes, i neglected to mention we did a clean shutdown shortly after. this was coordinated as a way to fix an urgent need which it sounded like the original poster had in regards to sending a loaner shelf back. it did function fine without any issues until an approved downtime window could be scheduled. in most of our environments we have zero ability for any downtime scheduled or otherwise which is really killing us with a current shelf bug causing drives to randomly go amber and support recommends power cycling the shelf to correct it + updated firmware--but thats a whole other story. --daniel -- Daniel Leeds Manager, Storage Operations Edmunds, Inc. 1620 26th Street, Suite 400 South Santa Monica, CA 90404 310-309-4999 desk 310-430-0536 cell -----Original Message----- From: Jeff Mohler [mailto:speedtoys.racing@gmail.com] Sent: Thu 11/13/2008 3:01 PM To: Leeds, Daniel Cc: Nils Vogels; Ray Van Dolson; toasters(a)mathworks.com Subject: Re: Removing a shelf. Daniel: I would check with support, IIRC, hot removing a shelf is _really_ frowned upon, and can cause mapping issues that can lead to _requiring_ you to power down all nodes and shelves to correct. Even in failover mode, shelf mapping is persistent. https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb40710 > Daniel Leeds > Manager, Storage Operations > Edmunds, Inc. > 1620 26th Street, Suite 400 South > Santa Monica, CA 90404 > > 310-309-4999 desk > 310-430-0536 cell > > > > -----Original Message----- > From: owner-toasters(a)mathworks.com on behalf of Nils Vogels > Sent: Thu 11/13/2008 1:16 PM > To: Ray Van Dolson > Cc: toasters(a)mathworks.com > Subject: Re: Removing a shelf. > > Hey Ray, > > On Thu, Nov 13, 2008 at 9:31 PM, Ray Van Dolson <rvandolson(a)esri.com> wrote: >> >> An aggregate spans two shelves, one of which is the shelf that needs to >> be returned. This aggregate contains a FlexVol currently in use. I'd >> like to preserve the FlexVol but remove the shelf. > > If you can afford the downtime, you could offline the aggregate > concerned, swap the disks into the shelf that can remain, and online > the aggregate again. Should be a few minutes. > > Another way, which does take a lot longer, is to use "disk replace" to > copy the used disks from their old position to their new ones. No > downtime, but the disks get copied one at a time, which can be a few > hours. > > HTH & HAND > > -- > Simple guidelines to happiness: > Work like you don't need the money, > Love like your heart has never been broken and > Dance like no one can see you. > >

1 0

Snapvault slow on one specific volume?
by Page, Jeremy 13 Nov '08

13 Nov '08

I have an aggr with two volumes on it. One of them is a 3.5 TB CIFS/NFS share that is reasonably fast to snapvault and a 1 TB NFS share (ESX VMs) that is exceptionally slow. As in it's been doing it's initial copy for over a week and still has not finished. NDMP backups of this volume are also quite slow, does anyone know why it would be so much slower then the other volume using the same spindles? The filer is not under extreme load, although occasionally it's pretty busy. Here is a "normal" sysstat: CPU Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk ops/s in out read write read write age hit time ty util 12% 1007 2970 8996 15769 9117 0 0 8 93% 49% : 41% 18% 920 2792 6510 11715 6924 0 0 8 99% 84% T 39% 15% 1276 3580 10469 15942 8041 0 0 10 92% 33% T 36% 13% 1487 3416 11347 15632 4907 0 0 11 89% 42% : 43% 17% 1417 3180 9890 14000 9444 0 0 9 98% 79% T 41% 13% 972 3704 9705 15427 9934 0 0 7 92% 46% T 51% 18% 1087 2947 11911 17717 4640 0 0 9 98% 33% T 47% 11% 1204 3358 11219 14090 5159 0 0 7 88% 100% : 50% 12% 1161 2808 9085 12640 5936 0 0 9 90% 33% T 44% 13% 981 4735 11919 16125 7097 0 0 9 92% 45% : 43% 15% 1158 5780 12480 17565 8266 0 0 10 92% 88% T 41% I'm just having difficulty trying to determine why two volumes on the same spindles would be so different in the time it takes to do their initial transfer. Also, the VM's do not seem slower then those hosted on other aggregates (this one is 3 RG of 11 disks each, Ontap 7.2.4 on a 3070A IBM rebranded). This message (including any attachments) contains confidential and/or proprietary information intended only for the addressee. Any unauthorized disclosure, copying, distribution or reliance on the contents of this information is strictly prohibited and may constitute a violation of law. If you are not the intended recipient, please notify the sender immediately by responding to this e-mail, and delete the message from your system. If you have any questions about this e-mail please notify the sender immediately.

7 15

Re: writes through NVRAM *also* in read cache?
by Blackmor, Chris 13 Nov '08

13 Nov '08

Writes are also put into read cache. If you have a highly cache dependent file system on a filer with high writes you are also caching those and will roll what is in cache more. We ended up moving our tools repos to low write filers to alleviate the problems we were experiencing from this. C- ----- Original Message ----- From: owner-toasters(a)mathworks.com <owner-toasters(a)mathworks.com> To: toasters(a)mathworks.com <toasters(a)mathworks.com> Sent: Thu Nov 13 11:09:05 2008 Subject: writes through NVRAM *also* in read cache? While I'm thinking of it... During a conversation with my sales rep., this question came up... (Haven't had a chance to ask our local SE's yet...) If a write happens through NVRAM (which it always does), once the write is acknowledged onto disk and the NVRAM flushed, is that data in read cache *as well*? Or, is the path to disk for a write unique, and when you want to read that data, it has to be read from disk and *then* put into read cache? I'm thinking of our software development environment, where we (of course) do a lot of writing of object files and then linking them into libraries and executables.... Until next time... The MathWorks, Inc. 508-647-7000 x7792 3 Apple Hill Drive, Natick, MA 01760-2098 508-647-7001 FAX tmerrill(a)mathworks.com http://www.mathworks.com ---

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters November 2008