February 2013 - toasters - lists.teaparty.net

Aggregate Disk Busy 100% with volume IOPS low
by Fletcher Cocquyt 01 Apr '13

01 Apr '13

3270 cluster, OnTAP 8.1-7mode We are investigating a SATA aggregate showing repeated 5am disk 100% busy spikes without its volumes showing any corresponding IOPS spike as reported by Netapp Management Console (NMC). The 5am disk busy spikes correlate with very high latency on volumes on a different SAS aggregate. These volumes host VMs which then timeout, some needing reboots. Today when I heard from Netapp support after reviewing my perfstat the engineer reported this is expected since NVRAM buffers are shared btw aggregates. But when I dig further into the NMC stats I see the SATA aggregate disk busy actually corresponds to a DROP in IOPS on the 3 volumes hosted on the SATA aggregate - almost like some internal aggregate operations are starving out the external volume ops. I checked the snapshots (vol and aggr), snap mirror, dedup and none of the usual suspects were running. When I look at the NMC throughput graphs and switch on the legend - it shows a 5am READ blocks/sec spike corresponding perfectly to the disk busy. Where are these AGGR level READ operations coming from that are missing from the constituent volume IOPS, and in fact seem to be starving out volume level IO? I don't see much in the messages log, but will check the rest of the logs for internal type OPS thanks for any insight

13 24

wafl_cp_slovol_warning_1 with big latency spikes
by Fletcher Cocquyt 08 Mar '13

08 Mar '13

Yesterday morning one of the heads on our 3270 experienced large NFS latency spikes causing our VMware hosts and their VMs to log storage timeouts. This latency does not correlate to any external metrics like CPU, network, OPS etc. But in the logs do show CP events on the aggregate hosting the VMs: Jan 14 05:27:56 [n04:wafl.cp.slovol:warning]: aggregate aggr2 is holding up the CP. And the EMS log has CP events logged for the duration of the episode - what can we do to prevent these issues? <wafl_cp_toolong_warning_1 total_ms="117825" total_dbufs="32276" clean="4312" v_ino="3" v_bm="29" a_ino="0" a_bm="3428" flush="1209"/> </LR> <LR d="14Jan2013 05:19:38" n="irt-na04" t="1358169578" id="1335304168/148007" p="4" s="Ok" o="wafl_CP_proc" vf="" type="0" seq="633232" > <wafl_cp_slovol_warning_1 voltype="aggregate" volowner="" volname="aggr2" volident="" nt="35" nb="22045" clean="1346852" v_ino="0" v_bm="113" a_ino="0" a_bm="4" flush="0" rgid="2"/> Netapp support wants me to run perfstats, but the issue is not ongoing - things are idle thanks Fletcher Cocquyt Principal Engineer Information Resources and Technology (IRT) Stanford University School of Medicine Email: fcocquyt(a)stanford.edu Phone: (650) 724-7485

5 4

Re: wafl_cp_slovol_warning_1 with big latency spikes
by Brian Beaulieu 04 Mar '13

04 Mar '13

I had opened a case when this happened and let the TSE know that I'm not the only one having this issue.. asked for it to be escalated and treated as a possible bug. We'll see where it goes. Brian On Thu, Jan 17, 2013 at 4:16 PM, Scott Eno <s.eno(a)icloud.com> wrote: > We're on 8.1.1P1 across the environment. > > 7-mode, of course. > > > On Jan 17, 2013, at 3:35 PM, Brian Beaulieu <brian.beaulieu(a)gmail.com> > wrote: > > Scott, what version of OnTAP are you on? > We're on 8.1.1P1 > > I saw a BURT that sounded related to this but it was apparently fixed by > 8.1.1P1. > > Brian > > > On Thu, Jan 17, 2013 at 2:21 PM, Jeff Mohler <speedtoys.racing(a)gmail.com>wrote: > >> :s is wafl updating special files in the CP process. >> >> >> Going on _that_ long....??? A few seconds of special file updates in a >> CP sure, but that much? >> >> I'd be pretty pushy on getting an answer, id put that in the "its a big >> bug" bucket. That's not normal IO activity in a healthy system. >> >> >> On Thu, Jan 17, 2013 at 8:07 PM, Scott Eno <s.eno(a)me.com> wrote: >> >>> >>> Yes, this is what I see. ":s" and all the other protocols go to "0". >>> >>> There's been some correlation, when this happens, to cleanup of VMware >>> snapshots (not NetApp snaps on the volumes, but VMware snapshots of vm's >>> via vcenter). But it happens other times too. >>> >>> >>> >>> On Jan 17, 2013, at 1:47 PM, Brian Beaulieu <brian.beaulieu(a)gmail.com> >>> wrote: >>> >>> 3rd time is the charm. >>> >>> I've attached my sysstat from the other night when NFS/CIFS hung up... >>> is this what you've seen as well? >>> >>> During that issue, FCP was also slow.. had some MPIO failovers happening >>> on our AIX LPARs. But, AIX handles that just fine and at least has an >>> alternate path through the other filer. NFS isn't so lucky. >>> >>> I have a 3250+1TB PAM sitting on deck.. you'd think that the 3240+512GB >>> PAM would be sufficient for what we do. >>> While I do have SATA in use for VMWare, it's not heavy hitting VMs.. >>> it's the dormant stuff, mostly. >>> I'm moving a lot of it, though, to 6xDS4243x600GB-15k shelves ASAP. >>> >>> I'm drinking the PAM kool-aid too but do have some measurable results >>> primarily on our PeopleSoft DB2 databases. >>> I definitely wouldn't bet on SATA+PAM == FC/SAS performance. >>> >>> Brian >>> <sysstat - Copy.txt>_______________________________________________ >>> >>> Toasters mailing list >>> Toasters(a)teaparty.net >>> http://www.teaparty.net/mailman/listinfo/toasters >>> >>> >>> >>> _______________________________________________ >>> Toasters mailing list >>> Toasters(a)teaparty.net >>> http://www.teaparty.net/mailman/listinfo/toasters >>> >>> >> >> >> -- >> --- >> Gustatus Similis Pullus > > > _______________________________________________ > Toasters mailing list > Toasters(a)teaparty.net > http://www.teaparty.net/mailman/listinfo/toasters > > >

4 3

DS2246 shelves and drives questions.
by Jeff Cleverley 28 Feb '13

28 Feb '13

Greetings, We will be upgrading an existing 6080 cluster to a 6280 cluster. We also have a 6080 cluster we'll be adding disks to also. Both clusters are 7-mode. We'll probably end up at 8.1 on both clusters. My 2 questions stem from the disks and shelves being proposed for the upgrades (ds2246 with 450g SAS). 1. We currently have DS4243 shelves with 600gig SAS. I know I can't mix these in the same stack because of the 3/6gb connection differences. I have a free port on each SAS adapter in our 6080s. Can I put the 6gb shelves on the same adapter as long as they are on their own stack? I'm guessing it may not be a best practice, but we don't have any slots free in the 6080 for more cards. 2. I found a white paper that states the 10K drive performance can be up to 20% slower than the 15K SAS drives for random reads. That seems like a pretty big performance it. It also says mixing the disks in aggregates is not recommended. Other than the smaller footprint and better density, I'm not seeing an advantage over using 600g SAS in the DS4243 shelves. Are there any advantages other than the faster SAS backend to warrant going to these shelves in our environment? Thanks, Jeff -- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611

4 4

clustered ONTAP, junction paths, export policies, NFS mounts ?
by Steve Losen 28 Feb '13

28 Feb '13

Hi Folks, Can anyone point me to some good documentation for the above? I've looked on the Netapp Support site at the man pages and admin guides and they cover the command options, but I haven't found anything explaining how all this stuff works together. I understand that the purpose is to provide a unified namespace, but I would like more information about what is legal/illegal, recommended practices, etc. I there a white paper on this? Steve Losen scl(a)virginia.edu phone: 434-924-0640 University of Virginia ITC Unix Support

5 5

Re: NDMP speed question
by tmac 27 Feb '13

27 Feb '13

For everyone... http://www.symantec.com/business/support/index?page=content&id=TECH51967 Hope it helps. By the way, looks like up to 256 is ok now. --tmac *Tim McCarthy* *Principal Consultant* Clustered ONTAP Clustered ONTAP NCDA ID: XK7R3GEKC1QQ2LVD RHCE5 805007643429572 NCSIE ID: C14QPHE21FR4YWD4 Expires: 08 November 2014 Expires w/release of RHEL7 Expires: 08 November 2014 On Wed, Feb 27, 2013 at 3:35 AM, Jayanathan, David <djayan(a)qualcomm.com>wrote: > Hi Tim,**** > > ** ** > > This mail from you caught my eye because our backup environment heavily > relies on NDMP and we are always looking for ways to improve it. Recently > our backup team has started to change their architecture to do 3-way NDMP > over 10GbE to a master server which has drives attached to it via tape SAN > and has multiplexing/pooling enabled.**** > > ** ** > > Were you able to dig up what specific environment variable you were > thinking of to change the backup blocking factor?**** > > ** ** > > Thanks in advance J**** > > ** ** > > -David**** > > ** ** > > *From:* toasters-bounces(a)teaparty.net [mailto: > toasters-bounces(a)teaparty.net] *On Behalf Of *tmac > *Sent:* Wednesday, February 20, 2013 11:53 AM > *To:* Klise, Steve > > *Cc:* toasters(a)teaparty.net > *Subject:* Re: NDMP speed question**** > > ** ** > > You know, another way to speed up backup is to change the backup blocking > factor.**** > > ** ** > > I think the default is 63 and if you are going over the network, it is > certainly feasible to make it 126.**** > > ** ** > > If I recall, there is an environment variable that can be set/modified to > change the blocking factor.**** > > > **** > > --tmac**** > > ** ** > > *Tim McCarthy***** > > *Principal Consultant***** > > ** ** > > [image: Image removed by sender.] [image: Image removed by sender.] > [image: Image removed by sender.]**** > > ** ** > > Clustered ONTAP > Clustered ONTAP**** > > NCDA ID: XK7R3GEKC1QQ2LVD RHCE5 805007643429572 NCSIE ID: > C14QPHE21FR4YWD4**** > > Expires: 08 November 2014 Expires w/release of RHEL7 > Expires: 08 November 2014**** > > ** ** > > On Wed, Feb 20, 2013 at 1:38 PM, Klise, Steve <klises(a)sutterhealth.org> > wrote:**** > > Just a lame question here, but how many and type of spindles are behind > the volume you are dumping? If you only have a handful of drives, could be > the culprit.**** > > > -----Original Message----- > From: toasters-bounces(a)teaparty.net [mailto:toasters-bounces@teaparty.net] > On Behalf Of Scott Eno > Sent: Wednesday, February 20, 2013 10:36 AM > To: Patrick Giagnocavo > Cc: toasters(a)teaparty.net > Subject: Re: NDMP speed question > > **** > > Likely a slight boost, but I am at the mercy of the network guys. I do > what they tell me. > > > On Feb 20, 2013, at 1:17 PM, Patrick Giagnocavo <xemacs5(a)gmail.com> wrote: > > > Stupid question, but isn't 10Gbe a lot faster with e.g. 9000 byte MTU? > I thought I saw in the original post, a 1500 byte MTU: > > > > > > From original post: > > > > e1a: flags=0x5f4e867<UP,BROADCAST, > > RUNNING,MULTICAST,TCPCKSUM,NOWINS> mtu > > 1500 > > > > if the 1500 byte MTU can be changed to 9000 without interrupting service > (depends on switch and other configuration, most likely), you might see a > speed bump. > > > > Cheers > > > > Patrick > > _______________________________________________ > > Toasters mailing list > > Toasters(a)teaparty.net > > http://www.teaparty.net/mailman/listinfo/toasters > > > _______________________________________________ > Toasters mailing list > Toasters(a)teaparty.net > http://www.teaparty.net/mailman/listinfo/toasters > > _______________________________________________ > Toasters mailing list > Toasters(a)teaparty.net > http://www.teaparty.net/mailman/listinfo/toasters**** > > ** ** >

2 2

absurd numbers in sysstat
by Scott Eno 26 Feb '13

26 Feb '13

Ever come across this? That's a whole lot of reads and writes in one second! All other numbers seem accurate, but those Disk read and kB/s write numbers are absurd. CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 56% 11041 843 0 11926 48613 42164 571755764 5350017024 0 0 1 94% 100% :v 100% 12 0 30 0 0 655 647 66% 5655 943 0 6612 38929 30760 39402 62888 0 0 1 98% 97% Hs 100% 3 0 11 0 0 53 375 52% 4508 419 0 4992 37034 23069 36347 92920 0 0 1 97% 100% :f 100% 19 0 46 0 0 842 800 CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 33% 1716 1830 0 3805 5017 34051 26328 43388 0 0 53s 96% 100% :f 24% 0 0 259 0 0 36 7332 42% 9556 1372 0 11483 10188 18549 21440 23824 0 0 53s 96% 100% :v 35% 7 0 548 0 0 307 8364 40% 3067 1878 0 6167 11743 38513 36296 68 0 0 53s 95% 5% : 47% 3 0 1219 0 0 803 14610 35% 1764 1664 0 4289 16814 38170 48450 0 0 0 57s 94% 0% - 59% 6 0 855 0 0 392 11342 35% 3068 931 0 4669 15851 52406 6530898751 4766869982 0 0 55s 95% 0% - 58% 12 0 658 0 0 17 9101 27% 3008 928 0 4263 14336 54182 60996 0 0 0 55s 94% 0% - 64% 3 0 324 0 0 0 9834 53% 4781 3550 0 8356 37070 56790 57780 15816 0 0 1s 96% 40% : 93% 0 0 25 0 0 135 541 72% 6323 1761 0 8095 42569 22303 75804 100132 0 0 0s 96% 76% Hf 91% 0 0 11 0 0 163 16 56% 11134 3358 0 14525 35197 22982 6598065628 4808667416 0 0 0s 93% 100% :f 73% 20 0 13 0 0 70 0 79% 10695 2746 0 13449 112103 17847 74439 96096 0 0 0s 96% 78% Hq 89% 3 0 5 0 0 17 0 84% 9690 2643 0 12438 91183 26266 82004 162335 0 0 42s 96% 100% Hn 100% 11 0 94 0 0 979 726 77% 10334 5223 0 15659 92746 28877 51385 107409 0 0 42s 96% 99% : 100% 88 0 14 0 0 7 74 CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 56% 7776 5374 0 13736 12413 20623 40795 70765 0 0 1 97% 100% :f 46% 7 0 579 0 0 258 4419 48% 9344 3753 0 13575 12647 50606 37204 35308 0 0 1 97% 61% : 46% 0 0 478 0 0 10 4063 38% 9098 2426 0 12015 11441 29765 17704 0 0 0 1 97% 0% - 74% 0 0 491 0 0 279 4137 48% 10616 2339 0 13448 13008 41186 6618900600 4809364036 0 0 1 97% 0% - 68% 15 0 478 0 0 49 3924 50% 13892 1838 0 15995 12338 50661 41572 24 0 0 1 97% 0% - 76% 3 0 262 0 0 123 2077

2 2

Netapp monitoring - OOIDs in 1.3.6.1.4.1.789.1.5 (filesys).
by Rafał Radecki 22 Feb '13

22 Feb '13

Hi All. I am using NetApp Release 8.0.2P4 7-Mode in active/active mode. I am monitoring it with snmp and Cacti. Yesterday I've performed takeover, sp reboot and giveback. During this process OIDS from 1.3.6.1.4.1.789.1.5 have changed for volumes. I have no list from before the change so I cannot compare but I see in the graphs that volumes are available under different OIDs. Have someone had such problems when restarting netapp/performing takeover giveback? Best regards, Rafal Radecki.

4 3

Adding a vlan but preserving base interface's IP
by Ray Van Dolson 22 Feb '13

22 Feb '13

ONTAP 7.3.6. This is a FAS2050C (IBM N3600 A20) that has a SAS expansion card in each controller. As such, we're limited to only the two onboard GigE ports. e0a and e0b both go to different physical switches and suck in two different networks, so we can't vif 'em together easily. I have a need to introduce a vlan interface to e0a and am wondering if it's possible to do so non-disruptively and preserve access to the e0a via the untagged traffic on the network. Both interfaces are configured with 'nfo'. I've found conflicting information on this.... some say you can mix untagged on your e0a interface and tagged traffic on e0a-vlan## just fine. Others say it's not doable and I'll need to get my network guys to tag all the traffic and move my e0a to a vlan interface. Not sure I'm seeing a way to do this without some downtime... even if I do a cluster failover, I don't see how a failback would work if I've mucked with the interfaces. Thanks, Ray

2 1

NDMP speed question
by Christopher S Eno 21 Feb '13

21 Feb '13

Hi, In testing NDMP dumps over 10GbE, I am seeing it max out at 1.5Gb/s. Some here are thinking that NDMP should be using more of that pipe. Anyone know about, or have numbers of NDMP transfer rates over 10GbE? Any caps or thresholds in OnTap that keep NDMP from gobbling up resources on the controller? I have run tests over the 10GbE connection from controller to controller, 10k drives to 10k drives, sata drives to sata drives, and always 1.5Gb/s is the ceiling. FTP'ing data from server to server across the same 10GbE network shows rates of 2.5Gb/s or higher. My NDMP options: ndmpd.access all ndmpd.authtype challenge ndmpd.connectlog.enabled off ndmpd.data_port_range all ndmpd.enable on ndmpd.fh_node_retry_interval 250 ndmpd.ignore_ctime.enabled off ndmpd.maxversion 4 ndmpd.offset_map.enable on ndmpd.password_length 16 ndmpd.preferred_interface e1a (value might be overwritten in takeover) ndmpd.tcpnodelay.enable on ndmpd.tcpwinsize 65534 IF e1a configuration: e1a: flags=0x5f4e867<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM,NOWINS> mtu 1500 inet x.x.x.x netmask 0xffffff00 broadcast x.x.x.x partner e1a (not in use) ether 00:07:43:08:98:ae (auto-10g_sr-fd-up) flowcontrol none

7 14

Mixed CIFS/NFS Access and issues w/ Locking
by Ray Van Dolson 21 Feb '13

21 Feb '13

We have an ONTAP 8.0.x filer here serving out a volume with an NTFS qtree via both CIFS and NFS. We're encountering the following situation: 1. An OSX user browses to the path via CIFS in Finder and then walks away/moves on to another task leaving the Finder window open on the path. - This generates a mandatory read lock on the folder (verified via lock status): CIFS path=\path(/path) host=10.28.102.30() owner=njarecha state=GRANTED mode=Read-denyN oplock=None 2. An NFS client comes along now (automated build process) and attempts delete the same folder. This appears to succeed from the NFS client's perspective. - We now see the following: CIFS path=\path(/path) host=10.28.102.30() owner=njarecha state=GRANTED mode=Read-denyN oplock=None CIFS path=\path(-) host= owner= state=GWAITING mode=DelOnClose At this point, CIFS access to the path now is completely blocked. No one can navigate into the folder, nor view its ACLs (save likely for the OSX client still holding the read lock). NFS clients can continue to access the contents of the folder (both read/write), but rmdir/rm operations on the folder itself fail with Permission denied. The fix is to (a) close the Finder window or (b) clear the lock via 'lock break' on the filer. I'll note that Windows Explorer doesn't appear to open a read lock when you only browse to a path -- not sure why Finder does. We're looking for options here, but first I'm wondering if the filer's response here isn't ideal. Rather than completing closing off access to CIFS clients, shouldn't #2 above just fail from the get-go rather than generating the DelOnClose queued event (if a mandatory read lock exists on the resource)? Other options we're considering: - Modifying automated processes to avoid the "rmdir" command. - Sticking with only one access protocol (likely CIFS). - Modifying OSX Finder behavior to not generate and hold a CIFS read lock (not sure how easy this would be to do). - Finding some sort of tunable on the ONTAP side to minimize impact (e.g. ignore the read lock and let the NFS rmdir operation to succeed) at the potential elevated risk of reduced data integrity. Anyone have any thoughts/experiences with something similar? Thanks, Ray

3 4

can't shoot SSH commands to a filer
by Randy Rue 20 Feb '13

20 Feb '13

Hello All, I have a 3020 running 7.2.4 that is failing to respond to SSH commands when they're sent one at a time. I can ssh in as root and get a shell via putty or ssh. But if I try to send an "uptime" command I get either "FATAL ERROR: Server unexpectedly closed network connection" (from plink) or "Connection to carbon-b closed by remote host." from ssh. Also, when I'm logged in interactively, I see lots of this blowing across the console: carbon-b> receive signals: rsh protocol failure in circuit setup. 300 received receive signals: rsh protocol failure in circuit setup. 300 received Try again. Try again. receive signals: rsh protocol failure in circuit setup. 300 received receive signals: rsh protocol failure in circuit setup. 300 received receive signals: rsh protocol failure in circuit setup. 300 received receive signals: rsh protocol failure in circuit setup. 300 received receive signals: rsh protocol failure in circuit setup. 300 received Try again. receive signals: rsh protocol failure in circuit setup. 300 received Try again. receive signals: rsh protocol failure in circuit setup. 300 received Try again. receive signals: rsh protocol failure in circuit setup. 300 received Try again. Try again. Try again. receive signals: rsh protocol failure in circuit setup. 300 received Try again. Try again. Try again. Try again. Try again. Try again. Try again. Try again. receive signals: rsh protocol failure in circuit setup. 300 received Try again. receive signals: rsh protocol failure in circuit setup. 300 received Try again. receive signals: rsh protocol failure in circuit setup. 300 received Try again. Try again. receive signals: rsh protocol failure in circuit setup. 300 received Try again. Try again. Try again. Try again. Try again. Try again. receive signals: rsh protocol failure in circuit setup. 300 received Try again. Try again. Try again. Try again. Try again. receive signals: rsh protocol failure in circuit setup. 300 received Try again. Try again. Try again. Try again. Try again. Try again. receive signals: rsh protocol failure in circuit setup. 300 received We have lots of processes (monitoring, alerting, scripted processes) that touch this box via non-interactive SSH, is each of these errors some failed attempt to connect? If I understand the inner workings correctly, pretty much all admin traffic on a filer channels into an internal RSH pipe. Are there different pipes for interactive and non-interactive SSH? I have tried stopping and restarting SSH (options ssh.enable off and on). Also RSH (options rsh.enable off and on). No love. Anyone have any ideas? Randy Seattle

2 2

Migration of Oncommand Host services
by KICKEN Ben (INT/TEI) 15 Feb '13

15 Feb '13

Greetings toasters, I am looking for migration procedures for the Oncommand Host Services, installed on the DFM/OCUM server (which is a VM) to the vCenter server (also a VM), without losing all configured datasets, protection and provisioning policies which Snapvault primary data to secondary storage. I was thinking of installing Host Package 1.2 on vCenter server and authorize it on the OCUM host - which will present all Vmware objects twice in the OC COnsole. Next unregister the OCUM hostservice (dfm hs unregister). Will all abovementioned datasets and policies still be available and active, on the hostservice running on the vCenter Host? Unfortunately I have no test environment for simulating this before doing this in real life; asked this in Communities and created a case at netapp support, but no good suggestions yet. thanks for any input Ben

2 1

Live disk shelf removal.
by Jeff Cleverley 13 Feb '13

13 Feb '13

Greetings, Is there a way to remove DS14 MK4 shelves from a running cluster? I need to look into isolating some drives and removing some shelves and would like to do it without having to halt the cluster. This is a 7.3.5.1 Ontap cluster on FAS6080s. Thanks, Jeff -- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611

6 6

HTTP & CFO
by Ray Van Dolson 13 Feb '13

13 Feb '13

ONTAP 8.0.x with httpd.enable set and virtual hosting configured via httpd.hostprefixes. Our PRD filer in question has two controllers in CFO mode which works fine for NFS/CIFS services. Will the HTTP services fail over properly as well? Tried to test this with the ONTAP 8.0.4 simulator, but you can't test CFO there apparently. Am especially curious about the httpd.hostprefixes settings and if they'll fail over gracefully. Thanks, Ray

1 0

can't touch a CIFS folder?
by Randy Rue 12 Feb '13

12 Feb '13

Hello All, Got a subfolder on a CIFS only vfiler (NTFS qtree) that I can't open, delete or rename, even though I'm connected as a user that's a member of the local administrators for the vfiler. From a windows host if I right-click the folder and choose properties there isn't even a permissions tab. Can't seize ownership. Can't open the folder (denied). >From a cifs-mounted linux host I also strike out but with odd errors. [root@host]# mv _DIVA\ templates delete mv: cannot stat `_DIVA templates': No such file or directory [root@host]# ls _DIVA templates [root@host]# ls -al total 24 drwxrwxrwx 3 root root 0 Feb 11 21:07 . drwxrwxrwx 28 root root 0 Jan 28 12:23 .. ?--------- ? ? ? ? ? _DIVA templates [root@host]# Note that the folder is called "_DIVA templates" I've tried copying and pasting it a few different ways to see if maybe it had funky/non-printable characters in the name but it doesn't appear so. Any ideas? Randy Rue Seattle WA

5 6

6080 heads with 6040 NVRAM cards.
by Jeff Cleverley 12 Feb '13

12 Feb '13

Greetings, I'm thinking about doing something that is not supported and was wondering if anyone had done the same or has more detailed insight. We have a very busy cluster (6040s 7.3.5.1P4). It looks like we are largely maxing out the heads for CPU. We are getting a pair of 6080s and really need to try and do the head swap live (takeover / giveback) if at all possible. The unsupported part I want to do is keep the 6040 NVRAM cards and put them in the 6080s as I swap them. The reason for this is I would not have to change the system ID ownership on all the drives. I know changing the system ID is generally not a big deal by booting each head to maintenance mode and reassigning the old SID to the new SID. In our case it worries me. Last week we were going to move a project to the other head by reassigning the appropriate drives for a couple of aggregates. While trying to reassign these the SAS buses started panic'ing and crashed the controlling filer. The entire cluster was down. The ensuing mess took several hours to clean up. If it crashed while trying to change ownership of a few drives, I'm afraid of what will happen when it tries to reassign all the old SID drives for the new NVRAM card. I was hoping if we could keep the cards, we could swap heads, not change SIDs, and minimize our chance of repeating the crash. I could do the disks one at a time, but I have 796 drives on this cluster and would rather not. Is there a requirement for the hardware to have the bigger memory cards? Since there are more CPUs, I can see where maybe something needs it, I just don't know what. We will probably have a downtime in a couple of months where I can put the correct ones back in. Thanks, Jeff -- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611

5 6

OnTap 8 Cluster install
by René Bormann 11 Feb '13

11 Feb '13

Hello Toasters, in our lab I got a FAS3070 Cluster to play around with. So this friday afternoon I wanted to install OnTap 8.1.2 Cluster over my 7-Mode test-installtion. Data loss is no problem. I tried to install the cluster software over the 7-mode Version with software update several times, but on every reboot. Ontap greets me with "7-Mode". Which point am I missing? Thanks, René

6 5

Merging Snapvault Volumes
by Steven Kreuzer 08 Feb '13

08 Feb '13

I opened up a case with NetApp about this, but I figured I should also post here and see what the community has to say. In a nutshell, I have two separate clusters and a volume with home directories on Filer_A that I was snapvaulting to Filer_B. During the hurricane, I had to shut down Filer_A, break the snapvault relationship on Filer_B to promote the volume to RW and then started to have everyone use Filer_B as their home. Folks have been working on Filer_B for 3 or so months now but I am ready to flip back and make Filer_A the primary once again. What are my options for basically taking all the new data as well as the snapshots that are on Filer_B and syncing them back to Filer_A, without losing any of the snapshots that currently exist on Filer_A? Right now the only option that comes to mind is the use rsync and copy a snapshot from Filer_B to Filer_A, take a snapshot on Filer_A, and repeat that process over and over until they have all been recreated on Filer_A. With that, does anyone have any better suggestions? Thanks in advance

5 6

Nifty ONTAP option nfs.max_num_aux_groups
by Steve Losen 08 Feb '13

08 Feb '13

Hi folks, We tend to use unix style security on shares that are accessed via both NFS and CIFS. Being a university we have a large number of users and groups, and some folks belong to numerous groups. (We use group permissions for shared projects and some folks are members of many unix groups.) We use NFS v3 exclusively and the auxiliary group list is passed by the NFS client in each NFS packet, and it is limited to 16 groups. This is rather inconvenient for some NFS users. Does anyone know if this is a hard NFS v3 limit? For CIFS we authenticate via a Windows domain and the CIFS credentials are mapped to Unix credentials for unix style security. We noticed with "cifs shares -s username" that folks were being limited to 32 unix groups. I have discovered that you can do this: options nfs.max_num_aux_groups 256 (the only legal values are 32 and 256, we have DOT 8.0.1 7-Mode.) and now our CIFS users who belong to over 32 unix groups are getting all their groups. No help for NFS v3 but this will make several of our CIFS users happy. Steve Losen scl(a)virginia.edu phone: 434-924-0640 University of Virginia ITC Unix Support

3 2

NetApp 8.0.2, raiddp, 2 failed disks - performance problem.
by Rafał Radecki 08 Feb '13

08 Feb '13

Hi All. I am new to this forum and quite new to NetApp. I work on FAS3210 with 8.0.2 7-Mode. We have 22 SAS disks in our appliance. The aggregate is: >aggr status Aggr State Status Options aggrX online raid_dp, aggr root degraded redirect 32-bit because we have lost 2 disks 2 days ago (one spare and one data disk, NetApp has sent new disks to us). The problem is that the appliance is performing bad in this situation. NFS latency in file operations changed from for example 0,04ms to 0,4ms. I know that the aggregate is now using parity disks in place of one of the failed disks but is this performance drop common? Are there any known ways/howtos to cut down this problem? Best regards, Rafal Radecki.

3 2

RE: Nifty ONTAP option nfs.max_num_aux_groups
by Touretsky, Gregory 08 Feb '13

08 Feb '13

NFSv3 supports both auth_sys (traditional) and rpcsec_gss (Kerberos) authentication. The latter has no 16 gids limit, but might require multiple changes in your environment to support cron jobs, unattended batch, etc. -----Original Message----- From: Steve Losen [scl(a)virginia.edu<mailto:scl@virginia.edu>] Sent: Friday, February 08, 2013 01:47 PM GMT Standard Time To: toasters(a)teaparty.net Subject: Nifty ONTAP option nfs.max_num_aux_groups Hi folks, We tend to use unix style security on shares that are accessed via both NFS and CIFS. Being a university we have a large number of users and groups, and some folks belong to numerous groups. (We use group permissions for shared projects and some folks are members of many unix groups.) We use NFS v3 exclusively and the auxiliary group list is passed by the NFS client in each NFS packet, and it is limited to 16 groups. This is rather inconvenient for some NFS users. Does anyone know if this is a hard NFS v3 limit? For CIFS we authenticate via a Windows domain and the CIFS credentials are mapped to Unix credentials for unix style security. We noticed with "cifs shares -s username" that folks were being limited to 32 unix groups. I have discovered that you can do this: options nfs.max_num_aux_groups 256 (the only legal values are 32 and 256, we have DOT 8.0.1 7-Mode.) and now our CIFS users who belong to over 32 unix groups are getting all their groups. No help for NFS v3 but this will make several of our CIFS users happy. Steve Losen scl(a)virginia.edu phone: 434-924-0640 University of Virginia ITC Unix Support _______________________________________________ Toasters mailing list Toasters(a)teaparty.net http://www.teaparty.net/mailman/listinfo/toasters --------------------------------------------------------------------- Intel Israel (74) Limited This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Max fiber run length from 3250 to MK14
by Mr Fletcher Cocquyt 08 Feb '13

08 Feb '13

We have rack space constraints. Can our 3250 heads be 6 racks away from our MK14 trays? Thanks, Fletcher

3 2

Minimum PSUs required for DS4243 with 24x SAS
by Alexander Griesser 07 Feb '13

07 Feb '13

Hey there, we were just re-cabling some of the power cords on our DS4243 shelfs with 4 PSUs and 24x 15k 600GB SAS. When I power down one of the 4 available PSUs in the DS4243, I get the following messages in system log and autosupports are generated: Wed Feb 6 19:28:15 CET [barricade:ses.status.psError:CRITICAL]: DS4243 (S/N SHJHU000000055F) shelf 2 on channel 0a power error for Power supply 4: critical status; power supply failed. This module is on the rear side of the shelf, at the bottom right. Wed Feb 6 19:28:15 CET [barricade:ses.status.volError:CRITICAL]: DS4243 (S/N SHJHU000000055F) shelf 2 on channel 0a voltage error for Voltage sensor 8: unrecoverable error. This module is on the rear side of the shelf, at the lower right, on the power supply. Wed Feb 6 19:28:15 CET [barricade:ses.psu.powerReqError:CRITICAL]: Not enough power supplies are present in channel 0a disk shelf 2 to satisfy disk drive and shelf power requirements. Wed Feb 6 19:28:15 CET [barricade:monitor.shelf.fault:CRITICAL]: Fault reported on disk storage shelf attached to channel 0a. Please check fans, power supplies, disks, and temperature sensors. Wed Feb 6 19:28:15 CET [barricade:callhome.shlf.fault:error]: Call home for SHELF_FAULT Wed Feb 6 19:28:15 CET [barricade:monitor.shelf.configError:CRITICAL]: Enclosure services has detected an error in access to shelves or shelf configuration 0a. Wed Feb 6 19:28:15 CET [barricade:callhome.config:warning]: Call home for CONFIGURATION_ERROR Wed Feb 6 19:29:00 CET [barricade:monitor.globalStatus.critical:CRITICAL]: Disk shelf fault. Disk shelf configuration error. Wed Feb 6 19:29:11 CET [barricade:ses.status.psInfo:info]: DS4243 (S/N SHJHU000000055F) shelf 2 on channel 0a power supply information for Power supply 4: normal status. Wed Feb 6 19:29:11 CET [barricade:ses.status.volInfo:info]: DS4243 (S/N SHJHU000000055F) shelf 2 on channel 0a voltage information for Voltage sensor 8: normal status. Wed Feb 6 19:29:11 CET [barricade:monitor.shelf.fault.ok:info]: Fault previously reported on disk storage shelf attached to channel 0a has been corrected. Wed Feb 6 19:29:11 CET [barricade:monitor.shelf.configError.ok:info]: Enclosure services-detected configuration error has been corrected on channel 0a. Wed Feb 6 19:30:00 CET [barricade:monitor.globalStatus.ok:info]: The system's global status is normal. Why exactly does it think that there are not enough power supplies present if I just shut down one of them? I was under the assumption that a full SAS shelf can work with 2 PSUs and a full SATA shelf can work with 1 PSU without problems. Am I wrong? Nothing happend to the filers, no takeover, no errors, etc., but the message in the logs puzzles me. Thanks && bye, Alexander Griesser System-Administrator ANEXIA Internetdienstleistungs GmbH Telefon: +43-463-208501-320 Telefax: +43-463-208501-500 E-Mail: ag(a)anexia.at <mailto:ag@anexia.at> Web: http://www.anexia.at <http://www.anexia.at> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

3 3

NetApp 8.0.2 7-mode - update to 8.1.2.
by Rafał Radecki 06 Feb '13

06 Feb '13

Hi All. I have a FAS3210 appliance with NetApp 8.0.2. I would like to update it to 8.1.2. In http://support.netapp.com/ (My AutoSupport) I see there is a "Upgrade Advisor" section. There is a step-by-step howto. But are there any good documents about updating this software? I would like to make as few mistakes as possible, I will do it for the first time ;) Best regards, Rafal Radecki.

3 3

Authentication Issues w/ SCCM talking to filer via CIFS w/ Computer Account
by Ray Van Dolson 05 Feb '13

05 Feb '13

We're having authentication issues with SCCM pointed at a CIFS share on a filer running ONTAP 8.0.x. SCCM uses a domain computer account to authenticate (our filer is also joined to our domain). We've added the computer account at the share level (as well as "Domain Computers") with full permissions, but continue to get authentication denied errors back from the filer. Speficially: Fri Feb 1 16:28:27 PST [red-str-napc2-p2: auth.trace.authenticateUser.loginTraceIP:info]: AUTH: Login attempt by user red-inf-cm-p01$ of domain DOMAIN from client machine 1.1.1.1 (RED-INF-CM-P01). Fri Feb 1 16:28:27 PST [red-str-napc2-p2: auth.dc.trace.DCConnection.statusMsg:info]: AUTH: TraceDC- attempting authentication with domain controller \\REDDC1. Fri Feb 1 16:28:27 PST [red-str-napc2-p2: auth.trace.authenticateUser.loginRejected:info]: AUTH: Login attempt by user rejected by the domain controller with error 0xc0000199: STATUS_NOLOGON_WORKSTATION_TRUST_ACCOUNT. Fri Feb 1 16:28:28 PST [red-str-napc2-p2: auth.trace.authenticateUser.loginTraceIP:info]: AUTH: Login attempt by user red-inf-cm-p01$ of domain DOMAIN from client machine 1.1.1.1 (RED-INF-CM-P01). This reads like the problem is with AD rejecting the login, but when we point to another CIFS share on a real Windows box we don't get the same problem, so we don't think that's the case. We came across this KB article[1] which seems to disable the use of the computer account for authentication. However, this seems to send *no* authentication information at all (anonymous?) which of course is rejected as well. Help? We'll be reaching out to support as well. Thanks, Ray [1] https://kb.netapp.com/support/index?page=content&id=2013374

3 6

NT_STATUS_NO_MEMORY
by Ray Van Dolson 05 Feb '13

05 Feb '13

Anyone ever encountered this error when attempting to access a CIFS share on an ONTAP 8.0.x system? ("NT_STATUS_NO_MEMORY"). We encountered reports from users of intermittent failures to connect to CIFS shares. From Windows hosts, things would just time out. From Samba clients, the above NT_STATUS_NO_MEMORY error was the result. The "fix" ended up being to restart the CIFS service on the system, but wondering what the true root cause here might be. I'm unsure how accurate the error returned by the Samba client is -- in any case there doesn't appear to be a way to query memory utilization on the NetApp. Thanks, Ray

2 2

VSC 4.1 & Online alignments
by Scott Eno 05 Feb '13

05 Feb '13

Hi, We're trying to get some VM's aligned in our NFS-based VMware environment using the "online" alignment tool packaged in VSC. Since we are NFS, the tool needs a VMFS datastore to use to align the VM. When we run the tool and it moves the VM to the VMFS datastore, the VM no longer shows up as misaligned. However, if we move the VM back to an NFS datastore, it shows up as misaligned again. So, is this a one-way trip? NFS --> VMFS, but that VM can never go back?

4 10

RE: nasty virus
by Klise, Steve 04 Feb '13

04 Feb '13

To fast on the trigger.. I don't want to start up the thread about AV on or not on the filers (I run it on my filer), but we got hit with a nasty variant of this. Not sure why Trend didn't block it, but basically a bunch of my CIF folders were set to read only/hidden.. Not fun. There are some other reasons that need to be mitigated that I wont go into, but this is the virus. http://about-threats.trendmicro.com/Malware.aspx?id=47409&name=WORM_VOBFUS.… To unhide the share, we were mapped to the share location, and ran this.. It unhides the shares lick-ity split. >From an XP box.. Cmd "usebackq delims=#" %i in (`dir *. /ah /s /b`) do attrib -s -h "%i" Just share because I care..

2 1

instantaneous snap restore ?
by Avi Ben Emanuel (Wollman) 04 Feb '13

04 Feb '13

marketing continuously claim that snap restore is instantaneous. yet I have yet to succeeded doing so. "snap restore" of files to a location quite a while and is "copying the files" nothing close to instantaneous. i would be happy to understand what i am missing. the following is a example of a virtual disk that was restored, it took quite a while to "restore"=copy this 80gb file. snap restore -t file -s nightly.0 /vol/vmsata/vmfs1/lms-client/lms-clinet-flat.vmdk Avi -- ------------ Avi Ben Emanuel (Wollman) (אבי בן עמנואל (וולמן Systems Engineer | Unix Microsoft Vmware Jerusalem College of Technology מחלקת תקשוב בית הספר הגבוה לטכנולוגיה - מכון לב POB 16031 Jerusalem 91160 Israel הועד הלאומי 21 ירושלים Email: avi(a)jct.ac.il | Office: +972-2-6751036 | Mobile: +972-52-3809107 ------------

4 3

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters February 2013