Hey folks, I'm looking for information that would help us alleviate a bottleneck with mailboxes and home directories on toaster(s).
Yah, we still run sendmail and our inboxes are in the old Unix style format (one file per user, multiple messages per file). I heard some of you mention Maildir and I also came across information on mbx format.
It doesn't look like I would have to do much to switch over to mbx vs. Maildir.
Q: Will mbx buy me performance on the toaster ? Q: Is Maildir a better choice ?
Please include any other suggestions.
Thanks, George
------------------------------------------------------------------------------- George Kahler e-mail: george@yorku.ca Systems Administrator humans: (416) 736-5257 x.22699 Computing Services, York University machines: (416) 736-5830 Ontario, Canada, M3J-1P3
On Thu, 14 Oct 1999, George Kahler wrote:
Yah, we still run sendmail and our inboxes are in the old Unix style format (one file per user, multiple messages per file). I heard some of you mention Maildir and I also came across information on mbx format.
We will use some kind of "maildir" format with one message per file. The reasoning behind it is that many messages will fit within one to several blocks and thus only a couple NFS reads or writes. Each time a mesages is deleted we unlink a file. This conserves diskspace as only the deleted message will be left in any snapshots rather than a copy of the entire old mailbox. Also, unlinking is a relatively inexpensive operation in contrast to reading and writing the entire spool file via the network. Less data to move around means faster performace or the same performance for more people.
Tom
Please include any other suggestions.
ok,
assuming the toaster is the bottleneck, you may also want to turn off atime updates, make sure the toaster is homed on each network on which it has clients, would adding more nvram benefit help? are you running 100tx-fd, or better? for that matter, is network, cpu, or disk i/o the real bottleneck?
-s
I must be missing it -- where is some info on maildir ? I see it mentioned on the qmail page, but I can't find the source and the homepage...
On Fri, 15 Oct 1999 armijo@cs.unm.edu wrote:
Please include any other suggestions.
ok,
assuming the toaster is the bottleneck, you may also want to turn off atime updates, make sure the toaster is homed on each network on which it has clients, would adding more nvram benefit help? are you running 100tx-fd, or better? for that matter, is network, cpu, or disk i/o the real bottleneck?
-s
-- Cue the music, fade to black, no such thing as no payback. -PWEI
[ armijo@cs.unm.edu ]
----------- Jay Orr Systems Administrator Fujitsu Nexion Inc. St. Louis, MO
1999-10-15-16:16:03 Jay Orr:
I must be missing it -- where is some info on maildir ? I see it mentioned on the qmail page, but I can't find the source and the homepage...
URL:http://www.qmail.org/qmail-manual-html/man5/maildir.html URL:ftp://koobera.math.uic.edu/www/proto/maildir.html URL:ftp://ftp.cs.hmc.edu/pub/me/mutt/contrib/maildir.c
For a quick overview, a Maildir is a directory with 3 subdirectories, named "tmp", "new", and "cur" (in the order messages flow through them).
dirname/tmp/ dirname/new/ dirname/cur/
When delivering a new message to a maildir, the writer opens a file under tmp/, with a guaranteed-unique name, typically built up from the time, the pid of the writer, and the hostname. This only counts on the inability to wrap pids within one second, and so far that seems safe. If a single writer (with a single pid) wants to write multiple messages (which it might easily do within a second) it includes a sequence number in the filename. But you can build filenames however you want as long as they're universally unique.
So once the writer has completed writing the message into tmp/, flushed it and closed it and so on, it links it into new/, where it appears for the benefit of mail readers. When a mail reader wants to indicate a change in its status (typically "it's been read, it's not new any more") it renames it over into cur/, and appends some flags to the filename. The contents normally never get rewritten.
If you read djb's docs, he specifies in exquisite detail the precise syscalls you should use, in what order, to write portable code that interacts with maildirs in perfect safety. That makes it seem a lot more complex. For many purposes the far cruder approach works fine, e.g. a local delivery agent
#!/bin/sh -e cd $HOME/Maildir fn=`date +%s`.$$.`hostname` cat >tmp/$fn mv tmp/$fn new/
Yup, it ain't perfect, but it's close enough to be robust in a lot of settings:-).
-Bennett
1999-10-14-16:35:59 George Kahler:
Q: Will mbx buy me performance on the toaster ? Q: Is Maildir a better choice ?
I don't know mbx specifically, but as far as I know it doesn't share Maildir's unique strength for this application (at least, I've never heard people cite mbx as an alternative when the claim of Maildir's uniqueness was proposed:-).
Maildir requires no locking for robust and reliable concurrent access from multiple simultaneous readers and writers.
NFS is by design and intent a stateless protocol, which has advantages. But locking is by nature a stateful operation. Sun tried, and failed, to retrofit locking onto NFS for years and years. They've shipped broken or unreliable implementations of attempts at locking protocols for over a decade now.
Sometimes, if you get lucky, you might find a specific pair of implementations, client and server, that work for one specific application, at least usually. But don't expect it, and if you should get lucky don't expect the luck to last; the next os release on client or server will probably restore the normal, non-working status.
Hence the prudent way to design network-distributed applications using NFS is to ensure that they never, ever require locking of any sort.
Use Maildir, it's your Friend.
-Bennett
-
armijo＠cs.unm.edu -
Bennett Todd -
George Kahler -
Jay Orr -
tkaczma＠gryf.net