http://jc.ngo.org.uk/blog It'll all be over in 60 days Mon, 10 Sep 2007 03:01:09 +0000 http://backend.userland.com/rss092 en I've taken to carrying around a RawWorkflow WhiBal card when I've been out and about practicing my photography, and had a few people ask me what it's for and how it helps. One of the nice things about how we see is that our brains are adept at adjusting our perception ... http://jc.ngo.org.uk/blog/2007/09/10/whibal-easy-white-balance-correction/ I ran in to Smylers' Talking at Conferences: A Beginners' Guide the other day. It's a very useful distillation of things you should consider if you're thinking of giving a talk at a technical conference, such as YAPC Europe 2007, where the talk was delivered. One recommendation in particular struck ... http://jc.ngo.org.uk/blog/2007/09/07/smylers-beginners-guide-to-talking-at-conferences/ If you're a frequent (or even semi-frequent) traveller then you may be interested in a website that I've been using for the past few months. TripIt exists to make it easy to manage one or more travel itineraries, and to share those itineraries with other people (who can be passive viewers, ... http://jc.ngo.org.uk/blog/2007/09/04/tripit/ I note that British Airways e-receipt e-mails are probably going astray for a lot of people. I've had to book a few flights with BA recently. Up until a couple of weeks ago their acknowledgment e-mails came through fine. And then I stopped receiving them. Taking the time ... http://jc.ngo.org.uk/blog/2007/08/21/spam-false-positives-from-british-airways/ It's been a bit quiet around here for the last couple of months. A quick brain dump. In February I left my job helping to run the mail systems at Citigroup, having sorted myself out with a contracting role doing software development in Perl (with some Autosys and Murex mangling ... http://jc.ngo.org.uk/blog/2007/08/02/off-to-switzerland/ If you publish SPF records, send mail to Korean ISPs, and use SPF mechnisms other than ip4:, you may face a problem. Apparently (and this is second-hand, so treat it with some caution), the Korean Information Security Agency (KISA) is producing an RBL of domains to blacklist. This is complemented ... http://jc.ngo.org.uk/blog/2007/02/28/issues-with-spf-and-korean-isps/ I'm a long time (x)emacs user. I use vi for small day-to-day stuff (vi TODO, anyone?), but as far as possible I do any development work with emacs, and have done for the past 12 years. So, a few days ago I was pleased to discover a new mode that ... http://jc.ngo.org.uk/blog/2007/02/27/m-x-ecb-activate/ Following on from yesterday's discussion of new features in Sendmail 8.14.0, today I'm writing about Sendmail's GreetPause feature, and some additional logging for it that's been added in Sendmail 8.14.0. Sendmail has supported pausing before displaying the greeting as an anti-spam measure since 8.13.0. To use it, add a line like: FEATURE(`greet_pause', `10000') to ... http://jc.ngo.org.uk/blog/2007/02/10/sendmail-8140-logging-the-greetpause-firing-time/ Sendmail 8.14.0 was recently released, and it includes a small handful of patches that I sent in. The documentation explains what these options do, but doesn't explain why you might want to use them. So I thought I'd do that in a series of entries here. First, the new ... http://jc.ngo.org.uk/blog/2007/02/08/sendmail-8140-heloname/ Well that Kubuntu experimentation didn't last long. After my last post I proceeded to install VMWare Server in Kubuntu. That went relatively easily, and I prepared a Windows XP VMWare image. The image creation was also relatively smooth, although it took 2 or 3 times longer than a normal Windows install. ... http://jc.ngo.org.uk/blog/2007/02/01/more-adventures-with-vmware/ The Ultra 40 that I won arrived on Wednesday, but I've been too busy since then to do much with it until now. I have spent a little bit of time carrying out some research. My initial plan was to triple boot this server. First as a FreeBSD machine ... http://jc.ngo.org.uk/blog/2007/01/28/getting-started-with-kubuntu/ Last Thursday (the 18th) I attended two open source events in London. The first was the third Subversion UK User Group meeting, and the second was Google's second Open Source Jam. This was almost a disaster for me. I was coming from docklands, and the DLR was suspended due ... http://jc.ngo.org.uk/blog/2007/01/20/subversion-uk-user-group-google-open-source-jam-writeup/ I'm very pleased to say that I've just been granted with a partial commit bit for the Subversion source code. The developers have had enough of me popping up on the mailing list talking about issues with the Perl bindings and posting patches, and have given me the necessary ... http://jc.ngo.org.uk/blog/2007/01/18/subversion-commit-access/ I've given Planet Subversion a facelift, and decided to splash the cash on a domain for it too -- planet-subversion.com. I've also recently added (but neglected to mention here) a feed from Mark Phippard, who has been writing some very interesting articles about the Subclipse project. http://jc.ngo.org.uk/blog/2007/01/17/planet-subversion-updates/ I write and maintain quite a lot of Perl software. Much of it is open source and available on CPAN. Most of it consists of libraries designed to be used by other applications, rather than applications in their own right. One of Perl's strengths is that "There's more than ... http://jc.ngo.org.uk/blog/2007/01/15/perl-parameter-validation-and-error-handling/ One of the common pieces of FUD used to argue against the use of open source software is "There's no vendor support. If it breaks, who do we turn to?". There are a number of answers to this particular concern. First, there's the community of developers that develop the software, and ... http://jc.ngo.org.uk/blog/2007/01/13/open-source-support-and-openlogic/ After a slightly longer development process than I would have liked (I had several hard disks choose inopportune moments to go south -- no data loss thanks to backups, but I took the opportunity to shuffle some hardware around), SVN::Web 0.50 has been released, and should now be available on ... http://jc.ngo.org.uk/blog/2007/01/12/svnweb-050-released/ If you're in London on Thursday 18th January and you've got an interest in Subversion then you might want to come along to the next Subversion UK User Group meeting. It's at 3pm, there are directions and sign up instructions at that link. I'll be there, giving a tour of ... http://jc.ngo.org.uk/blog/2007/01/12/subversion-uk-user-group-google-open-source-jam-18th-january-2007/ A great deal of spam is sent by "botnets". These are (typically) Windows PCs that have been compromised in some manner, and are now illicitly controlled by a third party. This third party uses the network of thousands of PCs that they have compromised to: Send spam Host phishing sites Carry ... http://jc.ngo.org.uk/blog/2007/01/09/alerting-users-that-their-pcs-are-compromised/ I've just discovered that I've been an unwitting participant in an identity theft. But not, perhaps, in the way that you might imagine. As already chronicled, some of my writing recently made it in to The Guardian. As is the way of these things The Guardian like to pay their writers, ... http://jc.ngo.org.uk/blog/2006/12/14/identity-theft/ One of the sites I make a point of reading regularly is use.perl, and in particular, the user journals / blogs. They don't take too long to read, and there's normally a couple of posts a day that teach me something I didn't know about Perl, or that highlight ... http://jc.ngo.org.uk/blog/2006/12/05/useperl-journals-and-full-text-feeds/ You're probably aware of Google's code hosting service. They use Subversion as their revision control system, so if you want to contribute to a project hosted there you really need a Subversion client. I was reading the FAQ for the hosting service the other day and a particular entry struck ... http://jc.ngo.org.uk/blog/2006/11/29/svnweb-and-google-code-hosting/ I've been experimenting with Plagger, a tool for plugging together chains of filters, pumping RSS/ATOM feeds in one end, and getting transformed output at the other end. This doesn't have to be as simple as chaining a few XSLT transformations together, as Plagger filters can carry out additional actions (such as ... http://jc.ngo.org.uk/blog/2006/11/27/planet-subversion/ Two spam related pieces of information today. The first concerns what happens if you're hosted at an ISP with an anti-spam policy, an itchy-trigger finger, and a support desk that is devoid of clue. It appears as though the fine folk over at The Weekly had their infrastructure on a shared server ... http://jc.ngo.org.uk/blog/2006/11/24/trigger-happy-hosting-spam-the-guardian/ Charles Arthur's wondering why spam came through his CAPTCHA system, and concludes that people are probably being paid to sit there and fill out CAPTCHAs. There are a couple of other possibilities. The first is that the CAPTCHA system he's using might be compromised. Some OCR systems can be ... http://jc.ngo.org.uk/blog/2006/11/21/captcha-farming/ I was pinged on IRC earlier today by someone who was having an e-mail discussion with Charles Arthur of the Guardian, in response to this article on Six steps to stopping spam. Since I spend a lot of my day job doing anti-spam engineering for a large organisation, Robbie ... http://jc.ngo.org.uk/blog/2006/11/17/thoughts-on-stopping-spam/ ... or so I thought. After I sent in my submission to Sun's Try and Buy contest, I didn't hear anything back from them for some time. So I thought it wouldn't hurt to drop them a quick note and check that they received it. After all, it's ... http://jc.ngo.org.uk/blog/2006/10/27/i-didnt-win/ This system will be going back to Sun soon, while I wait to find out whether or not they've decided to grant me the system. In the meantime, here are some final thoughts on the last 59 days. DTrace DTrace is remarkably useful. The ability to trivially trace what an ... http://jc.ngo.org.uk/blog/2006/09/06/day-59-of-60-final-thoughts/ Yesterday's tests show that using gcc on both FreeBSD and Solaris yields a marked improvement in the time taken to compile Perl. However, despite the big difference in compile times, the run-times of Perl's test suite aren't dramatically affected. The worst performer, Perl running on Solaris, compiled with Sun's cc ... http://jc.ngo.org.uk/blog/2006/09/06/day-59-of-60-developer-benchmarks-pt-4/ Yesterday I looked at performance compiling Sendmail on Solaris and FreeBSD using Sun's compiler (on Solaris) and gcc (on both systems). In the tests gcc come out handily ahead, with gcc on FreeBSD being 16% faster than gcc on Solaris with low optimisation options, and 12% faster than gcc on Solaris ... http://jc.ngo.org.uk/blog/2006/09/05/day-58-of-60-developer-benchmarks-pt-3/ As I explained on day 55, I've been comparing GCC and the Sun Studio compiler on Solaris, to GCC running on FreeBSD to see if there are any significant differences in the time taken to compile the applications, and if there is, whether that difference is reflected in the time ... http://jc.ngo.org.uk/blog/2006/09/04/day-57-of-60-developer-benchmarks-pt-2/ Fetching and installing FreeBSD was relatively painless. I downloaded the ISO image for FreeBSD 6.1 (AMD64 build), and wrote it to a blank CD. When I reinstalled Solaris some weeks back I made sure to leave some space on the disk for FreeBSD. So then it was a case ... http://jc.ngo.org.uk/blog/2006/09/02/day-55-of-60-installing-freebsd-on-a-sun-ultra-40/ The last week has been quite busy with work that's not related to this project. Mindful that the 60 day time limit is almost up, and aware that I've not done any actual benchmarking of this workstation -- vis a vis "How does Solaris on this hardware compare against ... http://jc.ngo.org.uk/blog/2006/09/02/day-55-of-60-developer-benchmarks-pt-1/ I've been looking at different queue sort strategies to see what their overhead is. Since all the messages are going to be delivered to a single host these results aren't necessarily going to be indicative of what you would see on a production server. However, they should serve ... http://jc.ngo.org.uk/blog/2006/08/24/day-46-of-60-queue-sort-strategies/ It's definitely a bug. Specifically, in the default case, and contrary to the documentation, sendmail does not run one queue runner for every queue directory. It runs precisely one. I brought this up on the Sendmail mailing list, sendmail-2006@support.sendmail.org. The most recent message in that discussion (at the ... http://jc.ngo.org.uk/blog/2006/08/21/day-43-of-60-multiple-queues-multiple-queue-runners-pt-3/ That's odd. You may recall that my most recent tests have involved sending 30,000 messages, split over 1, 5, 10, 20, 30, and 40 queue directories respectively. The Sendmail Operations Guide says: 2.3.1. Queue Groups and Queue Directories There are one or more mail queues. Each mail queue belongs to a queue group. ... http://jc.ngo.org.uk/blog/2006/08/19/day-41-of-60-multiple-queues-multiple-queue-runners-pt-2/ I've started to get data about the effect of multiple queues with multiple queue runners. As before I'm using 1, 5, 10, 20, 30, and 40 queue directories, and I'm instrumenting with queue-run-duration.d. This time I'm starting queue runners with the command sendmail -q30s. This will cause Sendmail to ... http://jc.ngo.org.uk/blog/2006/08/16/day-38-of-60-multiple-queues-multiple-queue-runners-pt-1/ Today I'm looking at the results that I've obtained from the latest round of tests. These tests used sendmail -q to deliver 30,000 messages to a different zone. There were 10 runs to each test, and the different tests collected data on timings for 1, 5, 10, 20, ... http://jc.ngo.org.uk/blog/2006/08/16/day-38-of-60-multiple-queues-one-queue-runner/ Previously I've written about variables that may affect how rapidly Sendmail can process the mail queue. I've now started working to gather data on exactly how much influence these variables have. Basic methodology The relay-zone is configured to accept messages from the internal-zone, and relay them. The internal-zone runs smtp-source(1) to generate ... http://jc.ngo.org.uk/blog/2006/08/15/day-37-of-60-instrumenting-queue-processing-time/ Note: If you're not familiar with sendmail queues, the sendmail queue primer I wrote might be useful. There are two aspects of mail queue management to consider with Sendmail. The first is the process that puts messages in the queue. I've looked at that in some detail already, and ... http://jc.ngo.org.uk/blog/2006/08/11/strategies-for-processing-the-queue/ Or: "How do I use DTrace with programs that fork?" With some help from the dtrace-discuss[1] mailing list I've now written a couple of D scripts that can trace what Sendmail is doing between probe points. There's a writeup, and sample output, below the fold. [1] Note -- the forum archive ... http://jc.ngo.org.uk/blog/2006/08/10/day-32-of-60-complete-instrumentation-of-queue-creation/ Back on day 28 I looked at the effect of multiple queue directories with concurrent senders. These results showed that there was considerable benefit with 10 senders and 10 queue directories. The benefit going to 20 queue directories with 10 senders was negligible. At the time I wondered whether this was ... http://jc.ngo.org.uk/blog/2006/08/09/day-31-of-60-queues-and-connections/ Having established that there's a significant increase in the amount of taken by the fdsync() and open() system calls when Sendmail creates queue entries with a single queue directory I've set about tracking down what that bottleneck is. To confirm that these system calls were the ones most dramatically affected by ... http://jc.ngo.org.uk/blog/2006/08/08/day-30-of-60-what-are-the-single-queue-directory-bottlenecks-pt-2/ Earlier posts have shown that using a single queue directory imposes a significant bottleneck when processing concurrent connections with Sendmail. Yesterday I posed some questions, and today I've started work on answering the first one. The first question was: What is responsible for the dramatic slow down in the single-queue case ... http://jc.ngo.org.uk/blog/2006/08/07/day-29-of-60-what-are-the-single-queue-directory-bottlenecks/ Yesterday I looked at the effect of multiple queue directories when processing messages over a single connection. Today I've been looking at how multiple queue directories can help when processing concurrent connections. The methodology was identical to the previous tests. The only change was to the smtp-source(1) command line. The ... http://jc.ngo.org.uk/blog/2006/08/06/day-28-of-60-instrumenting-sendmail-queue-file-creation-pt-4/ I've commited the first sets of results to the repository in the aptly named results/ directory. To refresh your memory, the question I intended to answer was: does the number of queue directories (on a single disk) make a significant impact on the time taken to create new entries in the queue? They're ... http://jc.ngo.org.uk/blog/2006/08/05/day-27-of-60-instrumenting-sendmail-queue-file-creation-pt-3/ It's time to run an instrumented Sendmail, throw some messages at it, and see how it performs. Specifically, does the number of queue directories (on a single disk) make a significant impact on the time taken to create new entries in the queue? To do this, I need to instrument ... http://jc.ngo.org.uk/blog/2006/08/05/day-27-of-60-instrumenting-sendmail-queue-file-creation-pt-2/ I've (finally) got Sendmail built, zones configured, DTrace working for functions declared static, and a mechanism for creating test SMTP sessions. So it's time to start putting this together, instrumenting Sendmail, and seeing whether or not I can use this to prove (or disprove) some common advice given when configuring Sendmail. First, ... http://jc.ngo.org.uk/blog/2006/08/04/day-26-of-60-instrumenting-sendmail-queue-file-creation-pt-1/ It's the school holidays, and my two children have had friends staying over this past week, which meant that there hasn't been much opportunity to work on this project, and even less opportunity to write about it. So these next few posts are going to be something of a ... http://jc.ngo.org.uk/blog/2006/08/04/day-26-of-60-smtp-source/ I've now got Sendmail built and installed, and adjusted the SMF so that it uses my local version of Sendmail (with DTrace probes) in favour of the system version. There were four chunks to this work. Configuration build infrastructure Sendmail configuration files (/etc/mail/sendmail.cf) are normally built by building a simpler configuration file (sendmail.mc) ... http://jc.ngo.org.uk/blog/2006/07/29/day-20-of-60-running-sendmail-in-the-zones-pt-2/ Now that Sendmail is building and correctly installing in to a custom directory it's time to start looking at how I get my version of Sendmail used instead of the version that's supplied with Solaris. For that, I need to delve in to the Solaris Service Management Facility (SMF). At the risk ... http://jc.ngo.org.uk/blog/2006/07/28/day-19-of-60-running-sendmail-in-the-zones-pt-1/ The issues with M4 have been resolved. A colleague, Andre Lucas, took up the challenge and worked out a fix which he describes in detail. And ministat's now looking much better. It's grown some useful new options, a lot of documentation, and can now (optionally) generate plots ... http://jc.ngo.org.uk/blog/2006/07/28/day-19-of-60-m4-issues-resolved-ministat-updates/ I've spent some of today porting some useful statistics reporting software from C to Perl. ministat reads in two or more files of data and uses the Student's t test to determine if there is any statistical difference between the means of the datasets. This is especially useful when comparing ... http://jc.ngo.org.uk/blog/2006/07/26/day-17-of-60-ministat/ Which is a problem because the Sendmail build system is written in it. Here's the problem I'm trying to solve. As mentioned previously, Sendmail supports a DESTDIR variable, which you can set on the make command line, to specify a prefix to be added to the all the installation paths. The problem is ... http://jc.ngo.org.uk/blog/2006/07/25/day-16-of-60-i-strongly-dislike-m4/ I've just had to carry out a complete reinstall of the OS, which was uncommonly tedious. If you recall, I'd mentioned wanting to upgrade to Solaris 11b40 a few days ago, to get some of the latest and greatest DTrace functionality. Having downloaded and burned the CD ISO images I was ready ... http://jc.ngo.org.uk/blog/2006/07/25/day-16-of-60-reinstall/ I've spent some time today getting Sendmail+DTrace to install properly. This wasn't quite as straightforward as it could be, requiring a little build infrastructure hacking. The general stanza to build and install Sendmail after extracing the source code is: % sh Build % make install This will overwrite your existing Sendmail installation. That's not ... http://jc.ngo.org.uk/blog/2006/07/24/day-15-of-60-installing-sendmail/ I've been a bit busy with other work over the past few days, and haven't made quite as much progress as I'd like. There are a few things that have moved forward though. First, I now have all three zones running with my home directory mounted read-only. This was trivial to ... http://jc.ngo.org.uk/blog/2006/07/23/day-14-of-60-minor-updates/ Following Monday's info dump about queues, I've spent some time over the last few days reading the DTrace documentation in detail. In particular, the Solaris Dynamic Tracing Guide. This is the DTrace handbook, with a great deal of information about how to use DTrace. It also contains the information ... http://jc.ngo.org.uk/blog/2006/07/19/day-10-of-60-first-probes-added-to-sendmail/ The time has come to start adding DTrace functionality to Sendmail. Of course, there's no point in just diving in and adding code left, right, and centre, so over the last couple of days I've been thinking about what I should be instrumenting first. One of the issues that I ... http://jc.ngo.org.uk/blog/2006/07/17/day-8-of-60-sendmail-queues/ I'm just starting to get my feet wet with DTrace. Does anyone know of a decent Emacs mode for editing .d files? http://jc.ngo.org.uk/blog/2006/07/14/day-5-of-60-dtrace-mode-for-emacs/ I've spent a bit of time today preparing some zones that I'll be using for testing my changes to Sendmail. I think I'm going to need three zones. I could probably get away with two, but for the moment three makes things a little simpler. All three zones are ... http://jc.ngo.org.uk/blog/2006/07/14/day-5-of-60-building-test-zones/ It's not much of a milestone, but spent five minutes getting Sendmail to build on Solaris 10 with the Sun Studio 11 compiler. First, make sure that /opt/SUNWspro/bin and /usr/ccs/bin are in your $PATH (for the compiler, linker, and related tools). Then you need this site.config.m4. The configuration that ships with ... http://jc.ngo.org.uk/blog/2006/07/13/day-4-of-60-sendmail-first-build/ One of the new features that Solaris 10 has that I'm interested in is zones. A zone is lightweight virtualisation environment. Unlike VMWare, or Xen, the whole environment is not virtualised. You still have one running OS kernel which arbitrates access to the hardware, for example. ... http://jc.ngo.org.uk/blog/2006/07/13/day-4-of-60-the-learning-zone/ This is a plug for one of the handiest network programs I've used in a long time. Synergy. In essence it's a software KVM. Only without the monitor part. You designate one machine to be the Synergy server. This is the machine that has the keyboard and ... http://jc.ngo.org.uk/blog/2006/07/11/day-2-of-60-synergy/ I wrote earlier about getting pkgsrc builds up and working. Unfortunately, I wrote too soon. Although Perl built successfully, attempts to build other applications that I'm going to need failed in interesting ways. No Subversion, no xemacs, no SVK. After fiddling with the build infrastructure for a little while and ... http://jc.ngo.org.uk/blog/2006/07/11/day-2-of-60-blastwave/ Now that I've started to get a development environment that I feel comfortable with I've imported the latest release of Sendmail in to my Subversion repository. This is publicly accessible, so you can follow along at home if you've got a Subversion client installed. I've imported Sendmail 8.13.7, plus two ... http://jc.ngo.org.uk/blog/2006/07/11/day-2-of-60-importing-and-branching-sendmail/ I've just configured the desktop to prompt me for my SSH credentials once, instead of on every connection, using ssh-agent and an X11 SSH password requestor. This is bread-and-butter stuff that should be easy, made a little more complex by documentation not being accessible on the Sun site. Since I couldn't ... http://jc.ngo.org.uk/blog/2006/07/11/day-2-of-60-ssh-agent-authentication/ In the years that I've been using Solaris it's support for third party packages has, to my mind, always let it down. The open source community writes and releases software at a phenomenal rate, and systems like FreeBSD and Linux have developed a number of interesting ways to make ... http://jc.ngo.org.uk/blog/2006/07/10/fun-and-games-with-pkgsrc/ I've spent a bit of time poking around the workstation and Solaris 10. I feel like a tourist in a Western European city. Everything's pretty familiar -- the roadsigns all look the same (although the font might be a bit different, and the speeds are marked in kph not mph) ... http://jc.ngo.org.uk/blog/2006/07/10/day-1-of-60-tourist/ After a few phone calls along the way to find out where it had got to, the workstation arrived today a little after midday. I was greeted with two large boxes, one containing the machine itself, the other containing a far smaller box with the keyboard and mouse. As a ... http://jc.ngo.org.uk/blog/2006/07/10/day-1-of-60-first-boot/ No, I haven't given up already. Although the box was supposed to arrive on the 5th there's been no sign of it so far. I did try using the tracking number that Sun sent me at mysun.sun.com. Sadly that just gives me: Thank you for your interest in Order Status. ... http://jc.ngo.org.uk/blog/2006/07/07/enoserver/ It started when I read a number of posts at Jonathan Schwartz's blog (in order: here, here, here, here, and here). Jonathan is Sun's CEO (although he wasn't at the time he started this series). The essence of it is that Sun are so stoked about their new hardware that: So... ... http://jc.ngo.org.uk/blog/2006/07/04/raison-detre/