Hi all.
We are currently experiencing some heavy load on a filer serving as storage to a webmail farm:
FILER> sysstat 1 [...] 63% 5583 0 0 1047 4996 3524 16 0 0 3 70% 6002 0 0 999 6005 3836 0 0 0 3 65% 5738 0 0 1067 5829 2671 0 0 0 3 68% 5881 0 0 972 6195 3424 16 0 0 3 83% 7174 0 0 1363 7401 5477 0 0 0 3 88% 7951 0 0 1609 8026 3984 0 0 0 3 91% 8041 0 0 1387 8357 7076 16 0 0 3 87% 7732 0 0 1369 8508 4601 0 0 0 3 87% 7258 0 0 1196 7554 6006 681 0 0 3 100% 6290 0 0 1039 6406 8108 5108 0 0 3 95% 6953 0 0 1381 6488 7536 2783 0 0 3 88% 8205 0 0 1427 8375 5456 0 0 0 3 73% 6115 0 0 993 6408 5051 16 0 0 3 79% 7046 0 0 1138 7779 2629 0 0 0 3 83% 6851 0 0 1181 7212 8240 0 0 0 3 86% 7888 0 0 1417 8185 5305 16 0 0 3 79% 7435 0 0 1217 7646 1676 0 0 0 3 50% 4001 0 0 664 4293 2490 0 0 0 3 48% 4253 0 0 711 3939 1564 16 0 0 3 46% 4115 0 0 681 4066 1265 0 0 0 3 [...]
The farm consists of 6 frontends, 2xPentiumIII - 800Mhz, 1Gb ram, 100Mb Fast Ethernet, running apache and an altered c-client, doing direct maildir access (no imap, direct filesystem access). The frontends go to about 250 concurrent sessions. There are nearly 1 million (1000000) maildir stored in here.
FILER> version NetApp Release 6.0.1R2: Fri Feb 9 01:12:44 PST 2001
FILER> ifconfig -a e0: flags=848043<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.1.103 netmask 0xffffff00 broadcast 192.168.1.255 partner inet 192.168.1.104 (not in use) ether 00:a0:98:00:9f:0a (100tx-fd-up) e2a: flags=8042<BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 00:20:fc:1e:63:d4 (auto-unknown-cfg_down) e2b: flags=8042<BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 00:20:fc:1e:63:d5 (auto-unknown-cfg_down) e2c: flags=8042<BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 00:20:fc:1e:63:d6 (auto-unknown-cfg_down) e2d: flags=8042<BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 00:20:fc:1e:63:d7 (auto-unknown-cfg_down) e7: flags=8042<BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 00:03:47:22:85:5e (auto-1000sx-fd-down) flowcontrol full lo: flags=948049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 4056 inet 127.0.0.1 netmask 0xff000000 broadcast 127.0.0.1 ether 00:00:00:00:00:00 (Shared memory)
The filer volume webmail:
FILER> df Filesystem kbytes used avail capacity Mounted on /vol/webmail/ 406736736 383407488 23329248 94% /vol/webmail/ /vol/webmail/.snapshot 101684180 0 101684180 0% /vol/webmail/.snapshot
is mounted in each of the 6 frontends.
NFSstat gives me:
FILER> nfsstat
Server rpc: TCP: calls badcalls nullrecv badlen xdrcall 0 0 0 0 0
UDP: calls badcalls nullrecv badlen xdrcall 350325996760 0 0 0
Server nfs: calls badcalls 393275655620
Server nfs V2: (25634001711 calls) null getattr setattr root lookup readlink read 0 0% 2872301897 11%41557694 0%0 0% 5182588465 20%124663 0% 16772257357 65% wrcache write create remove rename link symlink 0 0% 604329011 2%7689267 0% 19040366 0%12918825 0%11991858 0%26947 0% mkdir rmdir readdir statfs 167656 0% 827086 0% 108179901 0%718 0%
Server nfs V3: (13693563851 calls) null getattr setattr lookup access readlink read 0 0% 5446218 0% 43552013 0%2360996181 17%28976674 0%97560 0% 10689673307 78% write create mkdir symlink mknod remove rmdir 410587211 3%5298393 0% 161919 0% 4 0% 0 0% 15451799 0%82070 0% rename link readdir readdir+ fsstat fsinfo pathconf 12977697 0%9969929 0% 110291020 1%0 0% 928 0% 928 0% 0 0% commit 0 0%
The getattr seems way too big and this may point to a bad caching on the frontends. But could this bring the CPU to 100% most of the time? Could this be a wafl issue related with the low available space on the volume?
I'll first increase the nfs client cache to try to lower the getattr's. But I fear this won't help much, further optimization, from the bottom up, is needed.
Any ideas to help optimize the performance in this scenario? Any ideas are welcome.
If you need any further info (I wanted to send a filestats but is taking an eternity...) please ask.
TIA.
Ahh, in the meantime I was able to get the output of filestats, it may be of any help....
FILER> filestats volume webmail snapshot japc VOL=webmail SNAPSHOT=japc INODES=17653504 COUNTED_INODES=9942433 TOTAL_BYTES=373740074688 TOTAL_KB=383386132
FILE SIZE CUMULATIVE COUNT CUMULATIVE TOTAL KB 1K 1410803 3019516 10K 7413094 34777324 100K 9373165 95819528 1M 9888040 272893780 10M 9942389 380408808 100M 9942431 381046720 1G 9942432 381185320 MAX 9942433 383386132
AGE(ATIME) CUMULATIVE COUNT CUMULATIVE TOTAL KB 0 0 0 30D 3675160 168977132 60D 4879663 208385196 90D 6522408 239232636 120D 7459500 268557600 MAX 9942433 383386132
UID COUNT TOTAL KB #64010 9915276 380263412 #0 27157 3122720
GID COUNT TOTAL KB #64010 9891377 380150296 #0 27089 2372920 #1003 67 749800 #65534 23900 113116
Thus spake Jose Celestino, on Mon, Mar 04, 2002 at 05:41:29PM +0000:
Hi all.
We are currently experiencing some heavy load on a filer serving as storage to a webmail farm:
FILER> sysstat 1 [...] 63% 5583 0 0 1047 4996 3524 16 0 0 3 70% 6002 0 0 999 6005 3836 0 0 0 3 65% 5738 0 0 1067 5829 2671 0 0 0 3 68% 5881 0 0 972 6195 3424 16 0 0 3 83% 7174 0 0 1363 7401 5477 0 0 0 3 88% 7951 0 0 1609 8026 3984 0 0 0 3 91% 8041 0 0 1387 8357 7076 16 0 0 3 87% 7732 0 0 1369 8508 4601 0 0 0 3 87% 7258 0 0 1196 7554 6006 681 0 0 3 100% 6290 0 0 1039 6406 8108 5108 0 0 3 95% 6953 0 0 1381 6488 7536 2783 0 0 3 88% 8205 0 0 1427 8375 5456 0 0 0 3 73% 6115 0 0 993 6408 5051 16 0 0 3 79% 7046 0 0 1138 7779 2629 0 0 0 3 83% 6851 0 0 1181 7212 8240 0 0 0 3 86% 7888 0 0 1417 8185 5305 16 0 0 3 79% 7435 0 0 1217 7646 1676 0 0 0 3 50% 4001 0 0 664 4293 2490 0 0 0 3 48% 4253 0 0 711 3939 1564 16 0 0 3 46% 4115 0 0 681 4066 1265 0 0 0 3 [...]
The farm consists of 6 frontends, 2xPentiumIII - 800Mhz, 1Gb ram, 100Mb Fast Ethernet, running apache and an altered c-client, doing direct maildir access (no imap, direct filesystem access). The frontends go to about 250 concurrent sessions. There are nearly 1 million (1000000) maildir stored in here.
FILER> version NetApp Release 6.0.1R2: Fri Feb 9 01:12:44 PST 2001
FILER> ifconfig -a e0: flags=848043<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.1.103 netmask 0xffffff00 broadcast 192.168.1.255 partner inet 192.168.1.104 (not in use) ether 00:a0:98:00:9f:0a (100tx-fd-up) e2a: flags=8042<BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 00:20:fc:1e:63:d4 (auto-unknown-cfg_down) e2b: flags=8042<BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 00:20:fc:1e:63:d5 (auto-unknown-cfg_down) e2c: flags=8042<BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 00:20:fc:1e:63:d6 (auto-unknown-cfg_down) e2d: flags=8042<BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 00:20:fc:1e:63:d7 (auto-unknown-cfg_down) e7: flags=8042<BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 00:03:47:22:85:5e (auto-1000sx-fd-down) flowcontrol full lo: flags=948049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 4056 inet 127.0.0.1 netmask 0xff000000 broadcast 127.0.0.1 ether 00:00:00:00:00:00 (Shared memory)
The filer volume webmail:
FILER> df Filesystem kbytes used avail capacity Mounted on /vol/webmail/ 406736736 383407488 23329248 94% /vol/webmail/ /vol/webmail/.snapshot 101684180 0 101684180 0% /vol/webmail/.snapshot
is mounted in each of the 6 frontends.
NFSstat gives me:
FILER> nfsstat
Server rpc: TCP: calls badcalls nullrecv badlen xdrcall 0 0 0 0 0
UDP: calls badcalls nullrecv badlen xdrcall 350325996760 0 0 0
Server nfs: calls badcalls 393275655620
Server nfs V2: (25634001711 calls) null getattr setattr root lookup readlink read 0 0% 2872301897 11%41557694 0%0 0% 5182588465 20%124663 0% 16772257357 65% wrcache write create remove rename link symlink 0 0% 604329011 2%7689267 0% 19040366 0%12918825 0%11991858 0%26947 0% mkdir rmdir readdir statfs 167656 0% 827086 0% 108179901 0%718 0%
Server nfs V3: (13693563851 calls) null getattr setattr lookup access readlink read 0 0% 5446218 0% 43552013 0%2360996181 17%28976674 0%97560 0% 10689673307 78% write create mkdir symlink mknod remove rmdir 410587211 3%5298393 0% 161919 0% 4 0% 0 0% 15451799 0%82070 0% rename link readdir readdir+ fsstat fsinfo pathconf 12977697 0%9969929 0% 110291020 1%0 0% 928 0% 928 0% 0 0% commit 0 0%
The getattr seems way too big and this may point to a bad caching on the frontends. But could this bring the CPU to 100% most of the time? Could this be a wafl issue related with the low available space on the volume?
I'll first increase the nfs client cache to try to lower the getattr's. But I fear this won't help much, further optimization, from the bottom up, is needed.
Any ideas to help optimize the performance in this scenario? Any ideas are welcome.
If you need any further info (I wanted to send a filestats but is taking an eternity...) please ask.
TIA.
-- Jose Celestino japc@co.sapo.pt SysAdmin::SAPO.pt http://www.sapo.pt
main(){printf("%xu%xk%x!\n",15,12,237);}
On Mon, 4 Mar 2002, Jose Celestino wrote:
Ahh, in the meantime I was able to get the output of filestats, it may be of any help....
[snip]
The getattr seems way too big and this may point to a bad caching on the frontends. But could this bring the CPU to 100% most of the time? Could this be a wafl issue related with the low available space on the volume?
Just out of curiosity, what's your "wafl.maxdirsize" option set to? Is there a chance you've got one directory that's reached its limit? You didn't mention what sort of directory structure your application is using, and with 9.9M files perhaps there's a directory that's full.
I suggest this because it bit us recently and produced _very_ similar symptoms to what you described. We had an application that managed to hit the limit, having put 102,399 files in one directory, and then started looping trying to rename a 102,400th file into it. The result was a load of about 1800+ NFS ops/sec and artificially high cpu usage numbers, plus an /etc/messages file that dutifully logged the thousands upon thousands of "ENOSPC" errors, which our application patiently and persistently ignored. :-) After 24 hours our Cricket NFS ops/sec graphs looked bonkers.
So, it might seem a little wierd, but check your messages log for ENOSPC:
Wed Feb 20 16:05:22 PST [GbE-e7]: Returning ENOSPC for large dir (fsid 26082, inum 2135872)
and see if perhaps you've hit a directory size limit. An application that's trying to be "well behaved" and re-try a failed creat() or rename() could be the source of all those mysterious getattrs. Upping the maxdirsize would alleviate that, as would splitting up any very full directories.
-- Chris
-- Chris Lamb, Unix Guy MeasureCast, Inc. 503-241-1469 x247 skeezics@measurecast.com
On Mon, 2002-03-04 at 09:41, Jose Celestino wrote:
We are currently experiencing some heavy load on a filer serving as storage to a webmail farm:
Which model filer?
The filer volume webmail:
FILER> df Filesystem kbytes used avail capacity Mounted on /vol/webmail/ 406736736 383407488 23329248 94% /vol/webmail/ /vol/webmail/.snapshot 101684180 0 101684180 0% /vol/webmail/.snapshot
Looks like you have the snapshot reserve set to 20% and dont use it. You should recover that space. The more space you give the filer on the volume, the better ;)
The getattr seems way too big and this may point to a bad caching on the frontends. But could this bring the CPU to 100% most of the time? Could this be a wafl issue related with the low available space on the volume?
Are the clients doing v2 or v3 NFS mounts? Old data (from a previous life) suggests that for mail and news applications v2 edges out v3. From the looks of it you have both types of access going on.
Any ideas to help optimize the performance in this scenario? Any ideas are welcome.
It looks (from the 22 second snapshot of sysstat) that the writes are when the filer is getting pushed. Some things to look at as possible improvements (some require more work than others):
- what do the raid groups look like? smaller groups may help the writes - if snapshots are disabled (which makes sese for a mail filer), then recover the snap reserve space. - how much read cache do you have in the filer? Max it out. - it looks like there is a fair ammount of network activity; you might want to enable a vif or upgrade to gige. - what OS do the clients run? some have better nfs performance than others. Perhaps there are newer versions out.
You might also be at the breaking point for the filer (but I think you can get a bit more out of it by making some of the changes listed above).
At what rate do you add mailboxes and grow the data on the filer?
alexei
Thus spake Alexei Rodriguez, on Mon, Mar 04, 2002 at 10:38:22AM -0800:
On Mon, 2002-03-04 at 09:41, Jose Celestino wrote:
We are currently experiencing some heavy load on a filer serving as storage to a webmail farm:
Which model filer?
Sorry, I failed to point that out, F760.
The filer volume webmail:
FILER> df Filesystem kbytes used avail capacity Mounted on /vol/webmail/ 406736736 383407488 23329248 94% /vol/webmail/ /vol/webmail/.snapshot 101684180 0 101684180 0% /vol/webmail/.snapshot
Looks like you have the snapshot reserve set to 20% and dont use it. You should recover that space. The more space you give the filer on the volume, the better ;)
Goddamn! You're right, that really passed me by. 76% free now, that should really relieve the CPU a bit :)
The getattr seems way too big and this may point to a bad caching on the frontends. But could this bring the CPU to 100% most of the time? Could this be a wafl issue related with the low available space on the volume?
Are the clients doing v2 or v3 NFS mounts? Old data (from a previous life) suggests that for mail and news applications v2 edges out v3. From the looks of it you have both types of access going on.
Exactly, I have both, client NFS versions differ slightly. I noticed that also. Should I use only v2 or only v3 ?
Any ideas to help optimize the performance in this scenario? Any ideas are welcome.
It looks (from the 22 second snapshot of sysstat) that the writes are when the filer is getting pushed. Some things to look at as possible improvements (some require more work than others):
- what do the raid groups look like? smaller groups may help the writes
FILER3> vol status -v Volume State Status Options webmail online normal nosnap=on, nosnapdir=off, minra=off, no_atime_update=off,raidsize=14, nvfail=off raid group 0: normal raid group 1: normal
- if snapshots are disabled (which makes sese for a mail filer), then recover the snap reserve space.
Done.
- how much read cache do you have in the filer? Max it out.
I have minra=off, are you talking about this?
- it looks like there is a fair ammount of network activity; you might want to enable a vif or upgrade to gige.
Yes, about 40Mbit in peaks. We are considering trunking 2x100Mb or going to Gb.
- what OS do the clients run? some have better nfs performance than others. Perhaps there are newer versions out.
Linux 2.2.19 only. 2.4.18> possibly soon.
You might also be at the breaking point for the filer (but I think you can get a bit more out of it by making some of the changes listed above).
At what rate do you add mailboxes and grow the data on the filer?
No maildirs are being added to this filer. The Maildirs are being created on another filer, but the data grows about 250/300 Mbytes a day.
I also noticed that we had no_atime_update=off, this seems useless, I have turned it to no_atime_update=on.
alexei
On Mon, Mar 04, 2002 at 06:59:51PM +0000, Jose Celestino wrote:
Linux 2.2.19 only. 2.4.18> possibly soon.
What is your read and write block size?
Linux and NFS have always had issues together in my experience but things may have changed since the new 2.4 kernels.
As I said before, we are pushing more data than you using less ops per second, that could be the difference.
On Mon, Mar 04, 2002 at 05:41:29PM +0000, Jose Celestino wrote:
FILER> ifconfig -a e0: flags=848043<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.1.103 netmask 0xffffff00 broadcast 192.168.1.255 partner inet 192.168.1.104 (not in use) ether 00:a0:98:00:9f:0a (100tx-fd-up) e2a: flags=8042<BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 00:20:fc:1e:63:d4 (auto-unknown-cfg_down) e2b: flags=8042<BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 00:20:fc:1e:63:d5 (auto-unknown-cfg_down) e2c: flags=8042<BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 00:20:fc:1e:63:d6 (auto-unknown-cfg_down) e2d: flags=8042<BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 00:20:fc:1e:63:d7 (auto-unknown-cfg_down) e7: flags=8042<BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 00:03:47:22:85:5e (auto-1000sx-fd-down) flowcontrol full lo: flags=948049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 4056 inet 127.0.0.1 netmask 0xff000000 broadcast 127.0.0.1 ether 00:00:00:00:00:00 (Shared memory)
Hmm...you are just using the built in 100Mbps ethernet?
Why not use the gig interface?
The filer volume webmail:
FILER> df Filesystem kbytes used avail capacity Mounted on /vol/webmail/ 406736736 383407488 23329248 94% /vol/webmail/ /vol/webmail/.snapshot 101684180 0 101684180 0% /vol/webmail/.snapshot
Looks like you aren't using snapshots, why not do a:
snap reserve webmail 1
and reduce the 'snapshot' usage as far as possible since it isn't in use.
As someone else said, reaching above 90% can be hairy, but you aren't using the snapshots, so you shouldn't be hitting '90%' anyway, unless I don't know how snapshots work.
The getattr seems way too big and this may point to a bad caching on the frontends. But could this bring the CPU to 100% most of the time? Could this be a wafl issue related with the low available space on the volume?
getattr is used to see if anything is changing on the server. It is a good thing and perhaps there isn't caching going on with your frontend servers. Read further down for more questions from me.
What kind of filer is this?
Here is what my setup is:
6 POP servers, 3 IMAP servers, 4 combo (IMAP/WEB/POP), F760 as filestorage for Maildir format mailboxes
The 6 POP/3 IMAP servers are Solaris 8 on Sparc.
The 4 combo servers are FreeBSD 4.5-STABLE.
f760-1-mc-mpls> sysstat 2 CPU NFS CIFS HTTP Net kB/s Disk kB/s Tape kB/s Cache in out read write read write age 48% 1268 0 0 2182 4825 9121 3787 0 0 1 41% 1140 0 0 2391 8735 8882 536 0 0 1 34% 1265 0 0 2129 6337 5212 0 0 0 1 35% 1286 0 0 451 10372 7669 0 0 0 1 30% 1318 0 0 484 7207 5592 0 0 0 1 35% 1463 0 0 495 10960 5616 0 0 0 1 43% 1514 0 0 602 13488 6101 260 0 0 1 54% 1567 0 0 586 14221 7018 7672 0 0 1 36% 1427 0 0 551 12698 4454 0 0 0 1 34% 1472 0 0 613 9743 5318 0 0 0 1 29% 1400 0 0 473 7173 5218 0 0 0 1 27% 1449 0 0 368 5183 4944 0 0 0 1 27% 1058 0 0 329 6395 6130 0 0 0 1
f760-1-mc-mpls> nfsstat Server rpc: TCP: calls badcalls nullrecv badlen xdrcall 802611889 6 0 0 6
UDP: calls badcalls nullrecv badlen xdrcall 43142383 16 0 0 16
Server nfs: calls badcalls 845754229 0
Server nfs V2: (43054127 calls) null getattr setattr root lookup readlink read 38 0% 22636988 53%18956 0% 0 0% 15156029 35%149737 0% 547873 1% wrcache write create remove rename link symlink 0 0% 403447 1% 752060 2% 791185 2% 207132 0% 27 0% 1127 0% mkdir rmdir readdir statfs 167242 0% 2499 0% 2219554 5% 233 0%
Server nfs V3: (802700102 calls) null getattr setattr lookup access readlink read 0 0% 194662657 24%2922584 0% 110992483 14%139483257 17%4025240 1% 313928250 39% write create mkdir symlink mknod remove rmdir 4515215 1% 2643548 0% 154071 0% 125 0% 0 0% 4375337 1% 63317 0% rename link readdir readdir+ fsstat fsinfo pathconf 4403724 1% 1988483 0% 2025756 0% 16504930 2%3452 0% 6 0% 7667 0% commit 0 0%
My data rates are as high or higher than yours, but my ops per second is lower for whatever reason. I am not running 1M mailboxes, though, only about 25K (which is a huge difference).
What operating system are your frontends using?
Solaris and FreeBSD both have very good NFS code.
-
Alexei Rodriguez -
Chris Lamb -
Jose Celestino -
Mike Horwath