toasters February 2007

toasters@lists.teaparty.net

50 participants
33 discussions

weird behaviour on the nfs share plus kernel: nsm_mon_unmon: rpc failed, status=-13
by Linux Admin 15 Feb '07

15 Feb '07

I have user complaining that application fails to write to NFS share The output in syslog Feb 14 17:23:33 host1 kernel: nsm_mon_unmon: rpc failed, status=-13 Feb 14 17:23:33 host1 kernel: lockd: cannot monitor netapp1 Feb 14 17:23:33 host1 kernel: lockd: failed to monitor netapp1 rpcinfo -p netapp1 shows The NSM daemon (statd), RPC program 100024,is not running on the server. 100024 1 tcp 4047 status 100024 1 udp 4047 status The directory has sticky bit set on it or even without they get one 0 size file and nothing else. any pointers on troubleshooting please

2 1

RE: High disk reads but low network out
by Glenn Marsh 14 Feb '07

14 Feb '07

The problem may not be due to a full nisk and/or the lack of contiguous space. It may be due to the type of data on the filer and that your filer may not have enough cache memory. We have cache age issues but we have been having increasing problems with large files (larger than 1GB and some well over 10GB) and the creation of directories with large number of entries (directories with thousands of sub-directories and files). Both of these can cause the cache age to get very low. The real problem occured for us when we added a lot of high end Linux boxes with Gigabit Ethernet connections that really pound on our filers. The only fix is to upgrade to a filer with more cache memory. See Bug ID 3986, there is no fix and if you contact NetApp they will suggest the upgrade. Glenn (yet another one) -----Original Message-----

1 0

ELNK WTF? - Fwd: POST NOT ALLOWED - NETAPP
by Blake Golliher 14 Feb '07

14 Feb '07

Sigh, Earthlink. Please make with the stopping. For the love of God, make it stop! -Blake ---------- Forwarded message ---------- From: Netapp List Owner <netapp-owner(a)lists.corp.earthlink.net> Date: Feb 14, 2007 10:38 AM Subject: POST NOT ALLOWED - NETAPP To: netapp-owner(a)lists.corp.earthlink.net, Blake Golliher <thelastman(a)gmail.com> The message attached could not be sent to "NETAPP" mailing list because of the following error message: -> Domain gmail.com is not allowed to post A copy of this message is being sent to the list owner(s). Thank you, Mordomo Did your number of files dramatically increase? Is there a reconstruction going on? A wafl scan, or an upgrade? Ok that last one might be silly, but you never know.. :) Can you capture a perfstat or at least a few 30 second samples of statit? Perfstat is usually too much, but here's a quick shell script I run to quickly grab what I need. #!/bin/sh # if [ -z $1 ]; then echo " " echo "I need a filer target" echo "An example syntax" echo " get-stats.sh filer01.msg.dcn" echo " " exit 0 fi FILER=$1 # while true do DATAFILE="$FILER`date | awk '{print "_data_" $2 $3 }'`" echo "" >> $DATAFILE date >> $DATAFILE echo "------------------------------" >> $DATAFILE rsh $FILER 'priv set -q diag; statit -b' 2>/dev/null echo "Starting statit sample" >> $DATAFILE rsh $FILER 'priv set -q diag; nfsstat -z' 2>/dev/null echo "Zeroing nfsstat" >> $DATAFILE rsh $FILER 'priv set -q diag; nfs_hist -z' 2>/dev/null echo "Zeroing nfs_hist" >> $DATAFILE rsh $FILER 'priv set -q diag; wafl_susp -z' 2>/dev/null echo "Zeroing wafl_susp" >> $DATAFILE rsh $FILER 'sysstat -xs -c 30 1' >> $DATAFILE # And we wait... rsh $FILER 'priv set -q diag; statit -en' >> $DATAFILE 2>/dev/null rsh $FILER 'priv set -q diag; nfsstat -d' >> $DATAFILE rsh $FILER 'priv set -q diag; nfs_hist' >> $DATAFILE rsh $FILER 'priv set -q diag; wafl_susp -w' >> $DATAFILE echo " ** " >> $DATAFILE done if you don't allow rsh, you can enable passphrase ssh and replace rsh with ssh (I think it's built into 7 for free now..) or just run the commands above in sequence and save them to a text. A few samples of each for about 30 seconds should do it, but only during the problem, not helpful if it's not happening at the moment. I usually run the script for about 5 to 10 minutes, then ctrl+c out of it. Right now I look at it a lot by hand. -Blake On 2/14/07, Paul Letta <letta(a)jlab.org> wrote: > I have been having some performance issues on my F880 the past couple of > days. Specifically, an NDMP backup that normally takes 2 hours to > complete was stuck in the mapping phase for 12 hours before I killed it. > > What I am seeing is a high disk read, but without a corresponding > network out. Here is a sysstat: > > > CPU NFS CIFS HTTP Net kB/s Disk kB/s Tape kB/s Cache > > in out read write read write age > > 62% 1456 6 0 794 16954 46326 0 0 0 4s > > 56% 1418 0 0 597 13689 36506 0 0 0 5s > > 64% 1737 3 0 867 17637 43572 0 0 0 5s > > 53% 1135 18 0 572 14823 42905 1155 0 0 4s > > 73% 2067 20 0 1282 37835 69880 6347 0 0 4s > > 69% 2031 24 0 1267 38868 67632 0 0 0 4s > > 74% 1921 3 0 1027 26644 50881 7778 0 0 5s > > 59% 1459 0 0 774 18843 41313 0 0 0 6s > > 60% 1212 2 0 665 16391 38766 0 0 0 7s > > 54% 1028 19 0 509 11873 33960 0 0 0 7s > > 61% 844 302 0 571 13765 39032 0 0 0 7s > > 72% 1164 598 0 1870 16750 39968 0 0 0 5s > > 71% 948 640 0 8961 9788 29552 0 0 0 5s > > 79% 1297 603 0 24157 5831 26034 0 0 0 4s > > 87% 2130 465 0 23581 9794 31269 0 0 0 2s > > I don't know why the disks are reading so much without it going out the > network. > > This is an F880 running OnTap 7.1. It is somewhat full -- about 92% > (1.2TB out of 1.3TB). > > There isn't any stuck ndmp sessions: > > ndmpd status > ndmpd ON. > No ndmpd sessions active. > > I haven't really looked at what clients may be doing what due to the low > network out numbers- but I am going there next. > > Does anyone have any suggestions on what to look for ? > > Thanks, > > Paul >

1 0

RE: High disk reads but low network out
by Glenn Dekhayser 14 Feb '07

14 Feb '07

Look at your cache age. It's very low, meaning that you are not serving data out of memory, only directly from disk. As the cache age does down, see that your network out numbers go down as well. When your filer gets full, it isn't able to find contiguous space to write data to (thus write 'anywhere'), so it starts breaking things up and you lose temporal locality, which eliminates the benefits of Netapp's read-ahead caching mechanism, thus low cache ages and poor performance. That's probably what's happening. You should think about a wafl_scan reallocate when the system utiliziation goes down, or getting a bunch of data off the box. Glenn (the other one) ************************************************************************ Glenn Marsh Lattice Semiconductor Corporation Senior Unix Systems Administrator 5555 NE Moore Court CAD Engineering Hillsboro, Oregon 97124 glenn.marsh(a)latticesemi.com 503-268-8000 503-268-8136 phone - 503-268-8459 fax ************************************************************************

1 0

Snap Manager for SQL / SnapMirror issue.
by Rod Newcomb 14 Feb '07

14 Feb '07

Help! We have a SQL Cluster, running Server 2003, Snapdrive 3.2r1 and Snapmanager SQL 1.5 On node one, we have an sql instance that runs that Snapshots/Snapmirrors the SQL data over to our offsite location twice a day. Works like a champ. On node two, we have a new instance running all the same software, the backup runs and complets with no issues, but for some reason it will not snapmirror. From the logs, it looks like it doesn't even try. There are no errors. On the node that works, we get the following in the backuplogs: *** REQUEST SNAPMIRROR UPDATE AFTER BACKUP *** Starting SnapMirror update... Querying disk list for SnapMirror update... Requesting SnapMirror update for LUN [G:] on volume [musql2k_1_database] of filer [URBFiler2]... Request for SnapMirror update for LUN [G:] was completed successfully. Requesting SnapMirror update for LUN [K:] on volume [musql2k_1_logs] of filer [URBFiler2]... Request for SnapMirror update for LUN [K:] was completed successfully. Requesting SnapMirror update for LUN [I:] on volume [musql2k_1_snapinfo] of filer [URBFiler2]... Request for SnapMirror update for LUN [I:] was completed successfully. There is no such text on Node 2, even though it's set to Snapmirror: Full database backup Maximum databases of concurrent backup: 255 Naming convention: Generic Run transaction log backup after full database backup: Yes Keep the oldest 3 full backups. Retain up-to-the-minute restore ability for older backups in other management groups: No Leave database attached after DBCC: No Run DBCC CHECKDB with Live Database Before Backup: No Run DBCC CHECKDB with Live Database After Backup: No Backup Management Group: Standard Update SnapMirror after operation: Yes <---------------- Run Command after operation: No I have reinstalled SMSQL, and I can even manually kick off a snapmirror from the node inside of Snapdrive so I know were talking on the right ports. I have double checked the RSH access, and it all seems to be in order. Both the SQL instance has access as well as the Snapdrive service account. Any suggestions? I've been fighting this for five days. Netapp support is trying to help, but we're getting no where fast. Rod Newcomb

4 3

RE: moving data from trad -> flex vols
by George, Andrew 10 Feb '07

10 Feb '07

I usually ndmpcopy the data Edit /etc/cifsconfig_shares.cfg with the new path Restart cifs As I usually take a week or two to migrate a whole volume (I'm rearranging the filesystem while I migrate) I usually wait until I've migrated the whole thing and then rename old volume to something else (or destroy it if I havn't had a user pointing to it for a week) and rename the new vol and check the updates happened correctly in cifsconfig - if I've got doubts after the vol rename I restart cifs to find out how much trouble I'm in (if any) -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of tmac Sent: Saturday, 10 February 2007 4:22 AM To: Stephen C. Losen Cc: Glenn Walker; Glenn Dekhayser; René Bormann; toasters(a)mathworks.com; Fox, Adam Subject: Re: moving data from trad -> flex vols I do something like this: Terminate all file sharing protocols Perform the update/replication/whatever do a global search/replace in the cifsconfig_share.cfg renaming all old vols to new vols. If needed, perform the same function in exports Turn on the protocols. works fine without having to delete and re-add (sometime 10s to a hundred) shares back in --tmac On 2/9/07, Stephen C. Losen <scl(a)sasha.acc.virginia.edu> wrote: > > I've actually used the vol and aggr rename commands frequently, not in the process of migration. My current environment was\has been in disarray before my arrival and I'm trying to go back and standardize and clean up some things. > > > > > > > > That said, in the pretext of migration, perhaps it should be an optional thing to rename the exports/shares? > > > > > > > > Glenn > > You can stop ONTAP from automatically editing /etc/exports with > > options nfs.export.auto-update off > > Then whenever you create, delete, or rename a volume you get a warning > that /etc/exports needs to be edited, but ONTAP leaves it alone. > > I would "exportfs -u" the old stuff before renaming anything. Then > after renaming the volumes so that the new volume has the old name, I > would "exportfs -a" to export the new stuff. > > As for CIFS, I am pretty sure that you need to delete your old shares > before renaming the volumes, rename, and then create the shares again. > > Steve Losen scl(a)virginia.edu phone: 434-924-0640 > > University of Virginia ITC Unix Support > > > "This e-mail and any attachments to it (the "Communication") is, unless otherwise stated, confidential, may contain copyright material and is for the use only of the intended recipient. If you receive the Communication in error, please notify the sender immediately by return e-mail, delete the Communication and the return e-mail, and do not read, copy, retransmit or otherwise deal with it. Any views expressed in the Communication are those of the individual sender only, unless expressly stated to be those of Australia and New Zealand Banking Group Limited ABN 11 005 357 522, or any of its related entities including ANZ National Bank Limited (together "ANZ"). ANZ does not accept liability in connection with the integrity of or errors in the Communication, computer virus, data corruption, interference or delay arising from or in respect of the Communication."

1 0

RE: moving data from trad -> flex vols
by Glenn Walker 10 Feb '07

10 Feb '07

I've actually used the vol and aggr rename commands frequently, not in the process of migration. My current environment was\has been in disarray before my arrival and I'm trying to go back and standardize and clean up some things. That said, in the pretext of migration, perhaps it should be an optional thing to rename the exports/shares? Glenn ________________________________ From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Glenn Dekhayser Sent: Thursday, February 08, 2007 2:48 PM To: René Bormann; toasters(a)mathworks.com Cc: Fox, Adam Subject: RE: moving data from trad -> flex vols Rene: I have seen this behavior and have been simarly frustrated when migrating CIFS volumes from Traditional to Flexvols. In my opinion, Netapp should NOT be renaming CIFS shares when you change volume names. They do this automatically without warning you or logging. I know they're doing it because they think they're being helpful; but really the only time I can forsee changing volume names is if you're doing a migration like you're doing. VERY frustrating indeed. I found myself having to document all the existing shares, and write a script that deletes, then re-creates all the shares and their permissions on the new volume. Hours of work. Netapp, is there a way to disable this "feature" prior to renaming a volume? Glenn (the other one) ________________________________ From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of René Bormann Sent: Thursday, February 08, 2007 12:41 PM To: toasters(a)mathworks.com Cc: René Bormann; Fox, Adam Subject: Re: moving data from trad -> flex vols Hello Toasters, well, the root vol now moved to the new DS14 flex-vol. But this night I tried to move the data (shares / lun's / exports) from the old /vol/vol0 (trad) via "ndmpcopy -l 0 /vol/vol0 /vol/vol1" and some hour later, "ndmpcopy -l 0 /vol/vol0 /vol/vol1". Everthing was copied. All Clients where offline, and than I renamed /vol/vol0 -> /vol/vol0old and /vol/vol1 /vol/vol0. Because I thought than after the reboot all shares and luns and nfs-exports are back on /vol/vol0 but now on the new DS14. Well I failed, I found out that the vol rename also renamed the CIFS Shares Paths and also the LUN paths. So the CIFS clients where still reading from the old FC9 (/vol/vol0old now) and the iSCSI clients where too. Because I did't terminate CIFS and NFS and iSCSI. To be honest, cool renaming feature to prevent stupid germans from losing data ;-) So, I though be smart forget about the LUNs, they are fine, care about the shares. "CIFS terminate" and replace vol0old -> vol0 in the /etc/cifsconfig_share.cfg. "CIFS restart". "Year! I am the master !" But I was too stupid again. Because now my CIFS Clients got the shares but they could not see any data. And I got no idea why! So I "vol rename vol0 vol1" and "vol rename vol0old vol0" And now the CIFS clients connect back to their shares on the old FC9 again. So whats wrong about my proceedings? And the second question to save time, how to I move the iSCSI LUNs to the new volume? Thanks, Rene René Bormann wrote: Thanks to all how answered! Just used my sleepless night here in germany to fix it. Took me half an hour! - René Fox, Adam wrote: If you are using SecureAdmin/Secure FilerView/SSL, you need to restart the secureadmin services when you cross the 7.x line. There was a major code change starting in 7.0 and the service must be manually restarted (a reboot won't do it). Start here for docs: http://now.netapp.com/NOW/knowledge/docs/ontap/rel721/html/ontap/upgrade/co… -- Adam Fox adamfox(a)netapp.com -----Original Message----- From: René Bormann [mailto:bormann@bbtel.de] Sent: Tuesday, February 06, 2007 5:14 PM To: toasters(a)mathworks.com Subject: moving data from trad -> flex vols Hello, we are running an FAS920 with FC9 shelfs with data and new empty DS14 shelfs. On the FC9 we got one volume /vol/vol0 with root and data on it. Last week we upgraded from ONTAP 6.5.6 to 7.2.1 without problems. Now we want to move our data to the DS14 shelfs. We created one aggr1 with lots of drives. After that we created two new flex vols - /vol/root (20GB) and an /vol/vol1 (1TB) for the data. We copied the root data with no clients on the filer with "ndmpcopy -l0 /vol/vol0/etc /vol/root/etc". Worked fine in 7 minutes. Than we change the booting volume with "vol options /vol/root root". Booting from it, no problem -but now by requsting http://filer-ip/na_admin just the first page came and we were not able to start the FilerView Web-App. So so we switch the root vol again with "vol options /vol/vol0 root" and FilerView works fine. In the message file we found that one: Sat Feb 3 09:16:09 CET [httpd.socket.listener.create:error]: HTTPS Initialization failure; could not create listener socket. Nothing else. Can anybody help? Thanks, René -- ----------------------------------------- Telefonkonferenzen zum Festnetzpreis! Einfach kostenlos Konferenzraum anfordern http://www.talkyoo.net ----------------------------------------- René Bormann Geschäftsführer BB Tel GmbH Jungfernstieg 30, 20354 Hamburg Register: Amtsgericht Hamburg, HRB 86074 Geschäftführer: René Bormann Tel: +49 (0) 1805 - 65 777 777 * Fax: +49 (0) 1805 - 65 777 799 * http://www.bbtel.de ----------------------------------------- * (bbtel.de 14ct/min) ** (es entstehen die Kosten, die ihre Telefongesellschaft zu einer Hamburger Festnetz-Rufnummer berechnet.) ----------------------------------------- -- ----------------------------------------- Telefonkonferenzen zum Festnetzpreis! Einfach kostenlos Konferenzraum anfordern http://www.talkyoo.net ----------------------------------------- René Bormann Geschäftsführer BB Tel GmbH Jungfernstieg 30, 20354 Hamburg Register: Amtsgericht Hamburg, HRB 86074 Geschäftführer: René Bormann Tel: +49 (0) 1805 - 65 777 777 * Fax: +49 (0) 1805 - 65 777 799 * http://www.bbtel.de ----------------------------------------- * (bbtel.de 14ct/min) ** (es entstehen die Kosten, die ihre Telefongesellschaft zu einer Hamburger Festnetz-Rufnummer berechnet.) -----------------------------------------

4 3

RE: moving data from trad -> flex vols
by Fox, Adam 08 Feb '07

08 Feb '07

If you are using SecureAdmin/Secure FilerView/SSL, you need to restart the secureadmin services when you cross the 7.x line. There was a major code change starting in 7.0 and the service must be manually restarted (a reboot won't do it). Start here for docs: http://now.netapp.com/NOW/knowledge/docs/ontap/rel721/html/ontap/upgrade/co… -- Adam Fox adamfox(a)netapp.com -----Original Message----- From: René Bormann [mailto:bormann@bbtel.de] Sent: Tuesday, February 06, 2007 5:14 PM To: toasters(a)mathworks.com Subject: moving data from trad -> flex vols Hello, we are running an FAS920 with FC9 shelfs with data and new empty DS14 shelfs. On the FC9 we got one volume /vol/vol0 with root and data on it. Last week we upgraded from ONTAP 6.5.6 to 7.2.1 without problems. Now we want to move our data to the DS14 shelfs. We created one aggr1 with lots of drives. After that we created two new flex vols - /vol/root (20GB) and an /vol/vol1 (1TB) for the data. We copied the root data with no clients on the filer with "ndmpcopy -l0 /vol/vol0/etc /vol/root/etc". Worked fine in 7 minutes. Than we change the booting volume with "vol options /vol/root root". Booting from it, no problem -but now by requsting http://filer-ip/na_admin just the first page came and we were not able to start the FilerView Web-App. So so we switch the root vol again with "vol options /vol/vol0 root" and FilerView works fine. In the message file we found that one: Sat Feb 3 09:16:09 CET [httpd.socket.listener.create:error]: HTTPS Initialization failure; could not create listener socket. Nothing else. Can anybody help? Thanks, René -- ----------------------------------------- Telefonkonferenzen zum Festnetzpreis! Einfach kostenlos Konferenzraum anfordern http://www.talkyoo.net ----------------------------------------- René Bormann Geschäftsführer BB Tel GmbH Jungfernstieg 30, 20354 Hamburg Register: Amtsgericht Hamburg, HRB 86074 Geschäftführer: René Bormann Tel: +49 (0) 1805 - 65 777 777 * Fax: +49 (0) 1805 - 65 777 799 * http://www.bbtel.de ----------------------------------------- * (bbtel.de 14ct/min) ** (es entstehen die Kosten, die ihre Telefongesellschaft zu einer Hamburger Festnetz-Rufnummer berechnet.) -----------------------------------------

5 5

Snapshot question
by Miller, Michael CTR USTRANSCOM J2 07 Feb '07

07 Feb '07

I have a soon to be problem with snapshots coming up. I will be deleting a directory that has 238GB in it and I would really prefer this not to get captured by a snapshot if there is anything I can do about it. I don't want to destroy the exsisting snaps either. Does anyone know of any way to keep this out of the snap? Any thoughts on how to limit the hit? Thanks Mike Miller General Dynamics Information Technology Michael.Miller.ctr(a)ustranscom.mil Michael.Miller(a)gdit.com Phone: 618-229-1185

5 5

moving data from trad -> flex vols
by René Bormann 06 Feb '07

06 Feb '07

Hello, we are running an FAS920 with FC9 shelfs with data and new empty DS14 shelfs. On the FC9 we got one volume /vol/vol0 with root and data on it. Last week we upgraded from ONTAP 6.5.6 to 7.2.1 without problems. Now we want to move our data to the DS14 shelfs. We created one aggr1 with lots of drives. After that we created two new flex vols - /vol/root (20GB) and an /vol/vol1 (1TB) for the data. We copied the root data with no clients on the filer with "ndmpcopy -l0 /vol/vol0/etc /vol/root/etc". Worked fine in 7 minutes. Than we change the booting volume with "vol options /vol/root root". Booting from it, no problem -but now by requsting http://filer-ip/na_admin just the first page came and we were not able to start the FilerView Web-App. So so we switch the root vol again with "vol options /vol/vol0 root" and FilerView works fine. In the message file we found that one: Sat Feb 3 09:16:09 CET [httpd.socket.listener.create:error]: HTTPS Initialization failure; could not create listener socket. Nothing else. Can anybody help? Thanks, René -- ----------------------------------------- Telefonkonferenzen zum Festnetzpreis! Einfach kostenlos Konferenzraum anfordern http://www.talkyoo.net ----------------------------------------- René Bormann Geschäftsführer BB Tel GmbH Jungfernstieg 30, 20354 Hamburg Register: Amtsgericht Hamburg, HRB 86074 Geschäftführer: René Bormann Tel: +49 (0) 1805 - 65 777 777 * Fax: +49 (0) 1805 - 65 777 799 * http://www.bbtel.de ----------------------------------------- * (bbtel.de 14ct/min) ** (es entstehen die Kosten, die ihre Telefongesellschaft zu einer Hamburger Festnetz-Rufnummer berechnet.) -----------------------------------------

2 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters February 2007