toasters

toasters@lists.teaparty.net

2 participants
13522 discussions

Re: Webmail farm
by Jose Celestino 05 Mar '02

05 Mar '02

Thus spake Chris Lamb, on Tue, Mar 05, 2002 at 08:28:35AM -0800: > > > > wafl.maxdirsize 10240 > > > > or about 300000 files. > > Unless your software engineers generally use absurd 40-character > filenames, like ours tend to... :-) > No, our software engineer was DJB (Bernstein) :) And our filenames tend to be big yes, they're maildir: root@mail:/maildirs/user/Maildir/new# ls -1 1011623164.19222.mail4,S\=1415 | wc -c 30 > > No, we have quotas, file size and file number on each user mail account, > > so the maildirs can't grow that much. And the maildirs are hash using > > md5 and some parsing so there's really a "small" number of dirs/files in > > each dir. > > I figured that you must have done something like that, because most folks > realize pretty early on that putting 10's of thousands of files in shallow > (or flat!) directory structures in an application like that just doesn't > scale well at all. But the symptoms you described were fairly similar to > what we were seeing, so I figured I'd toss it out there. > No, more than 30/40 thousand it simply doesn't work. It is too sluggish to be usable! > > > > We had a similar problem at the beggining with 70000+ dirs on one dir > > but the simptoms where not the same. We only noticed a sluggish behavior > > when ls-ing the dir but no "ENOSPC" that I can remember. By the way, > > what does "ENOSPC" really mean? > > ENOSPC is the error code in Unix for "No space left on device". > Ok, related with wafl.maxdirsize , I see. > > Cheers, > > -- Chris > > -- > Chris Lamb, Unix Guy > MeasureCast, Inc. > 503-241-1469 x247 > <skeezics(a)measurecast.com> > -- Jose Celestino <japc(a)co.sapo.pt> SysAdmin::SAPO.pt http://www.sapo.pt --------------------------------------------------------------------- main(){printf("%xu%xk%x!\n",15,12,237);}

1 0

RE: Reset of Archive bit causing very large backups
by Wiebke, Robert (Robert) 05 Mar '02

05 Mar '02

Our solution (until the patch release goes GA) was to go with mtime over ctime (as recommended by NetApp). ndmpd ignore_ctime seems to correct this problem and we have seen our differential backups fall from 80GB per night to under 10GB per night. -Robert -----Original Message----- From: neil lehrer [mailto:nlehrer@ibb.gov] Sent: Monday, March 04, 2002 6:34 PM To: Adrian Phillips Cc: 'Wiebke, Robert (Robert)'; Toaster List Subject: Re: Reset of Archive bit causing very large backups are these backups ndmp? Adrian Phillips wrote: > > Hi Robert; > > We were having the same issue with our backups (Also using Veritas, though it was/is Backup Exec 8.5). The solution is to upgrade to 6.1R1P1D9 (or later). Bug # is 49892. > The other suggestion / solution is through the Veritas s/w, with the suggestion of changing the job from incremental/differential to Daily or Working Set, which would back up files changed in the last x days. > > Cheers, > -Adrian > > ps - our filer is an F760... > > -----Original Message----- > From: Wiebke, Robert (Robert) [mailto:wiebke@lucent.com] > Sent: Tuesday, February 19, 2002 9:08 AM > To: Toaster List > Subject: Reset of Archive bit causing very large backups > > Hello Toasters, > > We are running 5.3.6r1 on our main Filer 740 and we have noticed very large > differential backups lately. A full backup will take up approx. 340GB of > tape, however our differential backups are taking up on average 80GB. Last > week we saw two consecutive differential backups of 240GB and 340GB. > > I've spoken to Veritas (Using Datacenter 3.4 with a W2K master server) and > they tell me the backups themselves are accurate. While investigating this > we noticed that files not accessed in a long time (6+ months) were still > showing their archive bits set. As these files have not been changed in > sometime the archive bit would be expected to not be set. Sure enough, on > all files we looked at, the archive bit was set on all. > > Is this a bug? Is this documented and is a fix available? I've opened a > case with the NOW site, and as of yet I have not heard from them. Would you > expect an 80GB differental backup on a 340GB filer? Even on weekends when > less data was being accessed? > > Any comments appreciated. > Thanks! -- regards +++++++++++++++++++++++++++++++++++++++++++++++++ + Neil Lehrer + + United States International Broadcasting Bureau + System Development Division + + voice 202 619-2524 + fax 202 619-3576 + nlehrer(a)ibb.gov + + " is this crisis an opportunity or just + another grab the fire extinguisher moment?" + +++++++++++++++++++++++++++++++++++++++++++++++++

1 0

ndmp incremental backups
by neil lehrer 05 Mar '02

05 Mar '02

hi, what date does dump use for incrementals? my ndmp incrementals seem on the large side. the etc/dumpdates look like: /vol/vol0/ 0 Sat Mar 2 15:01:26 2002 /vol/vol0/apps 0 Wed Mar 28 12:15:22 2001 /vol/vol0/users 0 Tue Apr 10 11:45:56 2001 /vol/vol0/etc 0 Fri Apr 27 12:44:39 2001 /vol/vol0/all_non_quota_files 0 Mon Apr 30 19:08:44 2001 /vol/vol0/users2 0 Fri May 4 16:03:03 2001 /vol/vol0/ 2 Thu May 3 16:45:15 2001 /vol/vol0/users/a 0 Fri Aug 17 14:38:30 2001 /vol/vol0/ 4 Sun Mar 3 19:01:17 2002 last full was mar2. last incr was mar3. does dump look at the m-date? before i bug my ndmp vendor i want to do some checking myself. i am ontap 6.1.1r2 thanks. -- regards

2 1

RE: clustering filers
by Link, Bill (OTS-EDH) 04 Mar '02

04 Mar '02

We run an 840c cluster with 3TB of storage as part of an AIX HACMP cluster. Prior to that the filer heads were 760's. Although we have had no failures to speak of, during testing we deliberately failed everything from the host rs6000 systems to the filer heads including all network hardware without a problem. (Well, we did have a problem with nfs timeouts during cluster failover because the application couldn't handle the error but we fixed that by making the nfs mounts hard.) Bill Link System Administrator DST Output inc. bill_link(a)billing.com -----Original Message----- From: Jeff Kennedy [mailto:jlkennedy@amcc.com] Sent: Monday, March 04, 2002 7:05 AM To: NDMP List Subject: clustering filers A quick question regarding NetApp filer clutering. How well does it work? Would you spend the money again? Problems you've encountered that make it of questionable value. We are putting more and more applications on the filer that are considered 24X7 when the filer itself is not. Granted, they are extremely reliable but the fact is that they have several single points of failure. I am looking at clustering filers as a way of fixing that. The other possibility is Veritas ServPoint HA on SAN. Thanks. -- ===================== Jeff Kennedy Unix Administrator AMCC jlkennedy(a)amcc.com Notice: This e-mail and any attachments are intended only for the individual or company to which it is addressed and may contain information which is privileged, confidential and prohibited from disclosure or unauthorized use under applicable law. If you are not the intended recipient of this e-mail, you are hereby notified that any use, dissemination, or copying of this e-mail or the information contained in this e-mail is strictly prohibited by the sender. If you have received this transmission in error, please return the material received to the sender and delete all copies from your system. Thank you.

2 1

RE: Reset of Archive bit causing very large backups
by Adrian Phillips 04 Mar '02

04 Mar '02

Hi Robert; We were having the same issue with our backups (Also using Veritas, though it was/is Backup Exec 8.5). The solution is to upgrade to 6.1R1P1D9 (or later). Bug # is 49892. The other suggestion / solution is through the Veritas s/w, with the suggestion of changing the job from incremental/differential to Daily or Working Set, which would back up files changed in the last x days. Cheers, -Adrian ps - our filer is an F760... -----Original Message----- From: Wiebke, Robert (Robert) [mailto:wiebke@lucent.com] Sent: Tuesday, February 19, 2002 9:08 AM To: Toaster List Subject: Reset of Archive bit causing very large backups Hello Toasters, We are running 5.3.6r1 on our main Filer 740 and we have noticed very large differential backups lately. A full backup will take up approx. 340GB of tape, however our differential backups are taking up on average 80GB. Last week we saw two consecutive differential backups of 240GB and 340GB. I've spoken to Veritas (Using Datacenter 3.4 with a W2K master server) and they tell me the backups themselves are accurate. While investigating this we noticed that files not accessed in a long time (6+ months) were still showing their archive bits set. As these files have not been changed in sometime the archive bit would be expected to not be set. Sure enough, on all files we looked at, the archive bit was set on all. Is this a bug? Is this documented and is a fix available? I've opened a case with the NOW site, and as of yet I have not heard from them. Would you expect an 80GB differental backup on a 340GB filer? Even on weekends when less data was being accessed? Any comments appreciated. Thanks!

2 1

Webmail farm
by Jose Celestino 04 Mar '02

04 Mar '02

Hi all. We are currently experiencing some heavy load on a filer serving as storage to a webmail farm: FILER> sysstat 1 [...] 63% 5583 0 0 1047 4996 3524 16 0 0 3 70% 6002 0 0 999 6005 3836 0 0 0 3 65% 5738 0 0 1067 5829 2671 0 0 0 3 68% 5881 0 0 972 6195 3424 16 0 0 3 83% 7174 0 0 1363 7401 5477 0 0 0 3 88% 7951 0 0 1609 8026 3984 0 0 0 3 91% 8041 0 0 1387 8357 7076 16 0 0 3 87% 7732 0 0 1369 8508 4601 0 0 0 3 87% 7258 0 0 1196 7554 6006 681 0 0 3 100% 6290 0 0 1039 6406 8108 5108 0 0 3 95% 6953 0 0 1381 6488 7536 2783 0 0 3 88% 8205 0 0 1427 8375 5456 0 0 0 3 73% 6115 0 0 993 6408 5051 16 0 0 3 79% 7046 0 0 1138 7779 2629 0 0 0 3 83% 6851 0 0 1181 7212 8240 0 0 0 3 86% 7888 0 0 1417 8185 5305 16 0 0 3 79% 7435 0 0 1217 7646 1676 0 0 0 3 50% 4001 0 0 664 4293 2490 0 0 0 3 48% 4253 0 0 711 3939 1564 16 0 0 3 46% 4115 0 0 681 4066 1265 0 0 0 3 [...] The farm consists of 6 frontends, 2xPentiumIII - 800Mhz, 1Gb ram, 100Mb Fast Ethernet, running apache and an altered c-client, doing direct maildir access (no imap, direct filesystem access). The frontends go to about 250 concurrent sessions. There are nearly 1 million (1000000) maildir stored in here. FILER> version NetApp Release 6.0.1R2: Fri Feb 9 01:12:44 PST 2001 FILER> ifconfig -a e0: flags=848043<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.1.103 netmask 0xffffff00 broadcast 192.168.1.255 partner inet 192.168.1.104 (not in use) ether 00:a0:98:00:9f:0a (100tx-fd-up) e2a: flags=8042<BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 00:20:fc:1e:63:d4 (auto-unknown-cfg_down) e2b: flags=8042<BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 00:20:fc:1e:63:d5 (auto-unknown-cfg_down) e2c: flags=8042<BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 00:20:fc:1e:63:d6 (auto-unknown-cfg_down) e2d: flags=8042<BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 00:20:fc:1e:63:d7 (auto-unknown-cfg_down) e7: flags=8042<BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 00:03:47:22:85:5e (auto-1000sx-fd-down) flowcontrol full lo: flags=948049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 4056 inet 127.0.0.1 netmask 0xff000000 broadcast 127.0.0.1 ether 00:00:00:00:00:00 (Shared memory) The filer volume webmail: FILER> df Filesystem kbytes used avail capacity Mounted on /vol/webmail/ 406736736 383407488 23329248 94% /vol/webmail/ /vol/webmail/.snapshot 101684180 0 101684180 0% /vol/webmail/.snapshot is mounted in each of the 6 frontends. NFSstat gives me: FILER> nfsstat Server rpc: TCP: calls badcalls nullrecv badlen xdrcall 0 0 0 0 0 UDP: calls badcalls nullrecv badlen xdrcall 350325996760 0 0 0 Server nfs: calls badcalls 393275655620 Server nfs V2: (25634001711 calls) null getattr setattr root lookup readlink read 0 0% 2872301897 11%41557694 0%0 0% 5182588465 20%124663 0% 16772257357 65% wrcache write create remove rename link symlink 0 0% 604329011 2%7689267 0% 19040366 0%12918825 0%11991858 0%26947 0% mkdir rmdir readdir statfs 167656 0% 827086 0% 108179901 0%718 0% Server nfs V3: (13693563851 calls) null getattr setattr lookup access readlink read 0 0% 5446218 0% 43552013 0%2360996181 17%28976674 0%97560 0% 10689673307 78% write create mkdir symlink mknod remove rmdir 410587211 3%5298393 0% 161919 0% 4 0% 0 0% 15451799 0%82070 0% rename link readdir readdir+ fsstat fsinfo pathconf 12977697 0%9969929 0% 110291020 1%0 0% 928 0% 928 0% 0 0% commit 0 0% The getattr seems way too big and this may point to a bad caching on the frontends. But could this bring the CPU to 100% most of the time? Could this be a wafl issue related with the low available space on the volume? I'll first increase the nfs client cache to try to lower the getattr's. But I fear this won't help much, further optimization, from the bottom up, is needed. Any ideas to help optimize the performance in this scenario? Any ideas are welcome. If you need any further info (I wanted to send a filestats but is taking an eternity...) please ask. TIA. -- Jose Celestino <japc(a)co.sapo.pt> SysAdmin::SAPO.pt http://www.sapo.pt --------------------------------------------------------------------- main(){printf("%xu%xk%x!\n",15,12,237);}

4 6

RE: Webmail farm
by Aldrich David S 04 Mar '02

04 Mar '02

Jose, Try getting your filesystem under 90%. Strange things start happening to the filer when filesystems get above 92-94% full. -----Original Message----- From: Jose Celestino [mailto:japc@co.sapo.pt] Sent: Monday, March 04, 2002 11:41 AM To: toasters(a)mathworks.com Subject: Webmail farm Hi all. We are currently experiencing some heavy load on a filer serving as storage to a webmail farm: FILER> sysstat 1 [...] 63% 5583 0 0 1047 4996 3524 16 0 0 3 70% 6002 0 0 999 6005 3836 0 0 0 3 65% 5738 0 0 1067 5829 2671 0 0 0 3 68% 5881 0 0 972 6195 3424 16 0 0 3 83% 7174 0 0 1363 7401 5477 0 0 0 3 88% 7951 0 0 1609 8026 3984 0 0 0 3 91% 8041 0 0 1387 8357 7076 16 0 0 3 87% 7732 0 0 1369 8508 4601 0 0 0 3 87% 7258 0 0 1196 7554 6006 681 0 0 3 100% 6290 0 0 1039 6406 8108 5108 0 0 3 95% 6953 0 0 1381 6488 7536 2783 0 0 3 88% 8205 0 0 1427 8375 5456 0 0 0 3 73% 6115 0 0 993 6408 5051 16 0 0 3 79% 7046 0 0 1138 7779 2629 0 0 0 3 83% 6851 0 0 1181 7212 8240 0 0 0 3 86% 7888 0 0 1417 8185 5305 16 0 0 3 79% 7435 0 0 1217 7646 1676 0 0 0 3 50% 4001 0 0 664 4293 2490 0 0 0 3 48% 4253 0 0 711 3939 1564 16 0 0 3 46% 4115 0 0 681 4066 1265 0 0 0 3 [...] The farm consists of 6 frontends, 2xPentiumIII - 800Mhz, 1Gb ram, 100Mb Fast Ethernet, running apache and an altered c-client, doing direct maildir access (no imap, direct filesystem access). The frontends go to about 250 concurrent sessions. There are nearly 1 million (1000000) maildir stored in here. FILER> version NetApp Release 6.0.1R2: Fri Feb 9 01:12:44 PST 2001 FILER> ifconfig -a e0: flags=848043<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.1.103 netmask 0xffffff00 broadcast 192.168.1.255 partner inet 192.168.1.104 (not in use) ether 00:a0:98:00:9f:0a (100tx-fd-up) e2a: flags=8042<BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 00:20:fc:1e:63:d4 (auto-unknown-cfg_down) e2b: flags=8042<BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 00:20:fc:1e:63:d5 (auto-unknown-cfg_down) e2c: flags=8042<BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 00:20:fc:1e:63:d6 (auto-unknown-cfg_down) e2d: flags=8042<BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 00:20:fc:1e:63:d7 (auto-unknown-cfg_down) e7: flags=8042<BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 00:03:47:22:85:5e (auto-1000sx-fd-down) flowcontrol full lo: flags=948049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 4056 inet 127.0.0.1 netmask 0xff000000 broadcast 127.0.0.1 ether 00:00:00:00:00:00 (Shared memory) The filer volume webmail: FILER> df Filesystem kbytes used avail capacity Mounted on /vol/webmail/ 406736736 383407488 23329248 94% /vol/webmail/ /vol/webmail/.snapshot 101684180 0 101684180 0% /vol/webmail/.snapshot is mounted in each of the 6 frontends. NFSstat gives me: FILER> nfsstat Server rpc: TCP: calls badcalls nullrecv badlen xdrcall 0 0 0 0 0 UDP: calls badcalls nullrecv badlen xdrcall 350325996760 0 0 0 Server nfs: calls badcalls 393275655620 Server nfs V2: (25634001711 calls) null getattr setattr root lookup readlink read 0 0% 2872301897 11%41557694 0%0 0% 5182588465 20%124663 0% 16772257357 65% wrcache write create remove rename link symlink 0 0% 604329011 2%7689267 0% 19040366 0%12918825 0%11991858 0%26947 0% mkdir rmdir readdir statfs 167656 0% 827086 0% 108179901 0%718 0% Server nfs V3: (13693563851 calls) null getattr setattr lookup access readlink read 0 0% 5446218 0% 43552013 0%2360996181 17%28976674 0%97560 0% 10689673307 78% write create mkdir symlink mknod remove rmdir 410587211 3%5298393 0% 161919 0% 4 0% 0 0% 15451799 0%82070 0% rename link readdir readdir+ fsstat fsinfo pathconf 12977697 0%9969929 0% 110291020 1%0 0% 928 0% 928 0% 0 0% commit 0 0% The getattr seems way too big and this may point to a bad caching on the frontends. But could this bring the CPU to 100% most of the time? Could this be a wafl issue related with the low available space on the volume? I'll first increase the nfs client cache to try to lower the getattr's. But I fear this won't help much, further optimization, from the bottom up, is needed. Any ideas to help optimize the performance in this scenario? Any ideas are welcome. If you need any further info (I wanted to send a filestats but is taking an eternity...) please ask. TIA. -- Jose Celestino <japc(a)co.sapo.pt> SysAdmin::SAPO.pt http://www.sapo.pt --------------------------------------------------------------------- main(){printf("%xu%xk%x!\n",15,12,237);}

1 0

Re: Webmail farm
by Jose Celestino 04 Mar '02

04 Mar '02

Thus spake Probert, Simon - CLM, on Tue, Mar 05, 2002 at 07:12:21AM +1300: > Disk space and Network transport appear to be the most likely. Use Statit > to gather information about volume performance. Consider going to GB > Ethernet. This can be done on copper instead of fiber saving some > implementation costs. > But won't an increase in bandwidth result in a higher CPU load as more packets will get to/from the filer for processing ? > Simon Probert > > Team Leader - EAS > > Mobile 025 484 676 > > Ext. 65303 > > > > -----Original Message----- > From: Jose Celestino [mailto:japc@co.sapo.pt] > Sent: Tuesday, 5 March 2002 6:41 a.m. > To: toasters(a)mathworks.com > Subject: Webmail farm > > > Hi all. > > We are currently experiencing some heavy load on a filer serving as storage > to a webmail farm: [cut] -- Jose Celestino <japc(a)co.sapo.pt> SysAdmin::SAPO.pt http://www.sapo.pt --------------------------------------------------------------------- main(){printf("%xu%xk%x!\n",15,12,237);}

1 0

(no subject)
by Wollermann, Brian 04 Mar '02

04 Mar '02

1 0

clustering filers
by Jeff Kennedy 04 Mar '02

04 Mar '02

A quick question regarding NetApp filer clutering. How well does it work? Would you spend the money again? Problems you've encountered that make it of questionable value. We are putting more and more applications on the filer that are considered 24X7 when the filer itself is not. Granted, they are extremely reliable but the fact is that they have several single points of failure. I am looking at clustering filers as a way of fixing that. The other possibility is Veritas ServPoint HA on SAN. Thanks. -- ===================== Jeff Kennedy Unix Administrator AMCC jlkennedy(a)amcc.com

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters