Day 59 of 60: Final thoughts
This system will be going back to Sun soon, while I wait to find out whether or not they’ve decided to grant me the system. In the meantime, here are some final thoughts on the last 59 days.
Day 38 of 60: Multiple queues, one queue runner
Today I’m looking at the results that I’ve obtained from the latest round of tests. These tests used sendmail -q to deliver 30,000 messages to a different zone. There were 10 runs to each test, and the different tests collected data on timings for 1, 5, 10, 20, 30, and 40 queue directories.
Day 37 of 60: Instrumenting queue processing time
Previously I’ve written about variables that may affect how rapidly Sendmail can process the mail queue. I’ve now started working to gather data on exactly how much influence these variables have.
Day 32 of 60: Complete instrumentation of queue creation
Or: “How do I use DTrace with programs that fork?”
With some help from the dtrace-discuss[1] mailing list I’ve now written a couple of D scripts that can trace what Sendmail is doing between probe points. There’s a writeup, and sample output, below the fold.
[1] Note — the forum archive doesn’t seem to link to the discussion yet. When it does I’ll update this link to point to the discussion. The subject was “Using pid provider when process forks”.
Day 31 of 60: Queues and connections
Back on day 28 I looked at the effect of multiple queue directories with concurrent senders.
These results showed that there was considerable benefit with 10 senders and 10 queue directories. The benefit going to 20 queue directories with 10 senders was negligible.
At the time I wondered whether this was a general rule — i.e., is anything more than 10 queue directories overkill? Or is there a correlation between the number of queue directories compared to the number of simultaneous sending systems.
Day 30 of 60: What are the single queue directory bottlenecks? (pt 2)
Having established that there’s a significant increase in the amount of taken by the fdsync() and open() system calls when Sendmail creates queue entries with a single queue directory I’ve set about tracking down what that bottleneck is.
Day 29 of 60: What are the single queue directory bottlenecks?
Earlier posts have shown that using a single queue directory imposes a significant bottleneck when processing concurrent connections with Sendmail. Yesterday I posed some questions, and today I’ve started work on answering the first one.
The first question was:
What is responsible for the dramatic slow down in the single-queue case (test 4)?
Day 28 of 60: Instrumenting Sendmail queue file creation (pt 4)
Yesterday I looked at the effect of multiple queue directories when processing messages over a single connection.
Today I’ve been looking at how multiple queue directories can help when processing concurrent connections.
The methodology was identical to the previous tests. The only change was to the smtp-source(1) command line. The previous tests were run with -s 1, indicating one concurrent connection. These tests were run with -s 10, to force 10 concurrent connections.
Day 27 of 60: Instrumenting Sendmail queue file creation (pt 3)
I’ve commited the first sets of results to the repository in the aptly named results/ directory.
To refresh your memory, the question I intended to answer was:
does the number of queue directories (on a single disk) make a significant impact on the time taken to create new entries in the queue?
They’re quite surprising.
Day 27 of 60: Instrumenting Sendmail queue file creation (pt 2)
It’s time to run an instrumented Sendmail, throw some messages at it, and see how it performs. Specifically, does the number of queue directories (on a single disk) make a significant impact on the time taken to create new entries in the queue?
Day 10 of 60: First probes added to Sendmail
Following Monday’s info dump about queues, I’ve spent some time over the last few days reading the DTrace documentation in detail. In particular, the Solaris Dynamic Tracing Guide. This is the DTrace handbook, with a great deal of information about how to use DTrace.
It also contains the information about how to add custom DTrace probes to user applications. I was a bit surprised when I first read that, as it’s only a couple of pages long.
It turns out that adding DTrace probes really is that simple…