Does anyone have any tips for tuning a netapp (and/or clients) for nfs access to Maildir email folders? I know people do it, and potentially the load on the server(s) could be less than mbox because you arent shovelling around huge files, but it seems like the netapp is a little slow with opening a larger number of files for reading one by one. By large I mean over a few thousand (three to 130 thousand I have tested with) and slow I mean between 200 and 500 headers read per second from nfs. It is only slow if the filer does not have it cached already, because followup folder opens get several thousand headers per second throughput, similar to mbox. I have not tried other nfs servers, but I have attempted to simulate the load from courier-imapd or dovecot using a shell script to read 2 dozen lines from each of several thousand mail files, and the delay seems to be the netapp. Additionally, if if is a fresh folder open (not cached anywhere), the first 2,000 or so message headers load quickly, it slows down to between 200-500/sec for the next few thousand, and if you let it get above several tens of thousands, it continues to slow even more, down to less than 100 headers per second.
I am somewhat dissapointed that the NOW forums and documentation is pretty much devoid of any mention of Maildir, let alone tuning for it.
I have a FAS940, not heavily loaded, and the clients for this are a small cluster of Dell Poweredge 2650 dual 2ghz xeons, running in pre-production environments of either Debian Linux or FreeBSD 6.x. The filer is running 7.0.3, and the files are stored in a flexvol in an aggregate spanning 3 DS14's full of disks (28x36, 14x72).
On 2/14/06, Adam McDougall mcdouga9@egr.msu.edu wrote:
Does anyone have any tips for tuning a netapp (and/or clients) for nfs access to Maildir email folders? I know people do it, and potentially the load on the server(s) could be less than mbox because you arent shovelling around huge files, but it seems like the netapp is a little slow with opening a larger number of files for reading one by one. By large I mean over a few thousand (three to 130 thousand I have tested with) and slow I mean between 200 and 500 headers read per second from nfs. It is only slow if the filer does not have it cached already, because followup folder opens get several thousand headers per second throughput, similar to mbox. I have not tried other nfs servers, but I have attempted to simulate the load from courier-imapd or dovecot using a shell script to read 2 dozen lines from each of several thousand mail files, and the delay seems to be the netapp. Additionally, if if is a fresh folder open (not cached anywhere), the first 2,000 or so message headers load quickly, it slows down to between 200-500/sec for the next few thousand, and if you let it get above several tens of thousands, it continues to slow even more, down to less than 100 headers per second.
I am somewhat dissapointed that the NOW forums and documentation is pretty much devoid of any mention of Maildir, let alone tuning for it.
I have a FAS940, not heavily loaded, and the clients for this are a small cluster of Dell Poweredge 2650 dual 2ghz xeons, running in pre-production environments of either Debian Linux or FreeBSD 6.x. The filer is running 7.0.3, and the files are stored in a flexvol in an aggregate spanning 3 DS14's full of disks (28x36, 14x72).
In my old life, I had an F760 and 36K mailboxes and 7M files on the filesystem.
In my current life, I have 15K mailboxes and 3.2M files on the filesystem.
I find the NetApp boxes to be more than adequate for handling of Maildir formatted mailboxes using Postfix as the MTA, either virtual or maildrop as final delivery agent, and Courier-IMAP as the end user access to the filesystems.
No special tuning except for turning off atime updates on the volume used for the Maildir storage.
For the client server side, I have used both Solaris and FreeBSD with Solaris winning in performance over NFS vs FreeBSD. I have tested both TCP and UDP mounts, and NFS v2 and V3, and I find that for both of the OSes I have used in production, UDP and NVS v2 was the highest performing.
Best of luck to you.
-- Mike Horwath drechsau@gmail.com
Additionally, if if is a fresh folder open (not cached anywhere), the first 2,000 or so message headers load quickly, it slows down to between 200-500/sec for the next few thousand, and if you let it get above several tens of thousands, it continues to slow even more, down to less than 100 headers per second.
So you have 10000+ files in a single folder?
Unless something has changed in WAFL, there's no general index on the directory (I think there's a small hash, but that's not useful for large lookups), so the linear lookup for each one before it gets cached can get large. Can you ask your local mailer to create a second level of directories to reduce that number a bit?
Once cached, the directory access isn't an issue, but when you're scanning every single file in the directory, I'm guessing it doesn't pre-fill.
In my current life, I have 15K mailboxes and 3.2M files on the filesystem.
I find the NetApp boxes to be more than adequate for handling of Maildir formatted mailboxes using Postfix as the MTA, either virtual or maildrop as final delivery agent, and Courier-IMAP as the end user access to the filesystems.
What's the maximum number of files in a folder? I would think that could be a performance issue during a scan.
Hi Adam,
Your issue probably has something to do with NFS (use TCP), but it's also likely that the problem has a lot to do with Courier, as I've always noticed that courier runs more slowly than qmail when accessing maildirs. Make sure that you are running courier with IMAP_CAPABILITY THREAD=ORDEREDSUBJECT THREAD=REFERENCES... of course this depends on whether the client is sorting or not, but it's best to have IMAP handle this for performance reasons.
BTW, are you using Gig-E?
Is minra on? (maildirs are often very random in terms of I/O)
You may also consider using an IMAP proxy as well if your clients don't support persistent connections (webmail clients.)
Regards, Max
Does anyone have any tips for tuning a netapp (and/or clients) for nfs access to Maildir email folders? I know people do it, and potentially the load on the server(s) could be less than mbox because you arent shovelling around huge files, but it seems like the netapp is a little slow with opening a larger number of files for reading one by one. By large I mean over a few thousand (three to 130 thousand I have tested with) and slow I mean between 200 and 500 headers read per second from nfs. It is only slow if the filer does not have it cached already, because followup folder opens get several thousand headers per second throughput, similar to mbox. I have not tried other nfs servers, but I have attempted to simulate the load from courier-imapd or dovecot using a shell script to read 2 dozen lines from each of several thousand mail files, and the delay seems to be the netapp. Additionally, if if is a fresh folder open (not cached anywhere), the first 2,000 or so message headers load quickly, it slows down to between 200-500/sec for the next few thousand, and if you let it get above several tens of thousands, it continues to slow even more, down to less than 100 headers per second.
I am somewhat dissapointed that the NOW forums and documentation is pretty much devoid of any mention of Maildir, let alone tuning for it.
I have a FAS940, not heavily loaded, and the clients for this are a small cluster of Dell Poweredge 2650 dual 2ghz xeons, running in pre-production environments of either Debian Linux or FreeBSD 6.x. The filer is running 7.0.3, and the files are stored in a flexvol in an aggregate spanning 3 DS14's full of disks (28x36, 14x72).
On Wed, Feb 15, 2006 at 10:20:38AM -0800, Max wrote:
Hi Adam,
Your issue probably has something to do with NFS (use TCP), but it's also likely that the problem has a lot to do with Courier, as I've always noticed that courier runs more slowly than qmail when accessing maildirs. Make sure that you are running courier with IMAP_CAPABILITY THREAD=ORDEREDSUBJECT THREAD=REFERENCES... of course this depends on whether the client is sorting or not, but it's best to have IMAP handle this for performance reasons.
BTW, are you using Gig-E?
Yes
Is minra on? (maildirs are often very random in terms of I/O)
I have tried with and without minra, no noticable difference
You may also consider using an IMAP proxy as well if your clients don't support persistent connections (webmail clients.)
We will be doing this for webmail.
Regards, Max
Does anyone have any tips for tuning a netapp (and/or clients) for nfs access to Maildir email folders? I know people do it, and potentially the load on the server(s) could be less than mbox because you arent shovelling around huge files, but it seems like the netapp is a little slow with opening a larger number of files for reading one by one. By large I mean over a few thousand (three to 130 thousand I have tested with) and slow I mean between 200 and 500 headers read per second from nfs. It is only slow if the filer does not have it cached already, because followup folder opens get several thousand headers per second throughput, similar to mbox. I have not tried other nfs servers, but I have attempted to simulate the load from courier-imapd or dovecot using a shell script to read 2 dozen lines from each of several thousand mail files, and the delay seems to be the netapp. Additionally, if if is a fresh folder open (not cached anywhere), the first 2,000 or so message headers load quickly, it slows down to between 200-500/sec for the next few thousand, and if you let it get above several tens of thousands, it continues to slow even more, down to less than 100 headers per second.
I am somewhat dissapointed that the NOW forums and documentation is pretty much devoid of any mention of Maildir, let alone tuning for it.
I have a FAS940, not heavily loaded, and the clients for this are a small cluster of Dell Poweredge 2650 dual 2ghz xeons, running in pre-production environments of either Debian Linux or FreeBSD 6.x. The filer is running 7.0.3, and the files are stored in a flexvol in an aggregate spanning 3 DS14's full of disks (28x36, 14x72).
-
Adam McDougall -
Darren Dunham -
Max -
Mike Horwath