One of the nice things about how we see is that our brains are adept at adjusting our perception of what we see according to the lighting conditions. If we know that something is white we perceive it to be white, and can generally ignore the actual lighting conditions. For example, if you put on a pair of tinted sunglasses for a few minutes everything appears to take on the tint of the glasses. And then your brain adjusts, and everything appears to be the normal colour again.

Cameras don’t do this.

For example, shoot a photo in daylight and everything has a blueish tint. Shoot under indoor lighting and everything looks different. Shoot with a flash and things change again. Most cameras do a reasonable job of adapting to this. Either with a White Balance setting of “auto”, or by allowing you to specify the lighting conditions.

However, “reasonable” does not equal perfect. Which is where the WhiBal card comes in.

The WhiBal is an example of a “grey card”. It’s a very specific, known shade of grey, and many post-processing tools, such as Photoshop or Lightroom, have a feature whereby you can select a part of the image that you know to be neutral, and it can automatically correct the image’s white balance. All you need to do is find a part of the image that’s supposed to be neutral.

That’s not always easy. So what many applications support is the ability to make an exposure change in one image, and then synchronise that change to one or more other images.

A worked example may make this easier to follow. Yesterday I was at Big Basin Redwoods State Park. Each time I walked to a different location where the light had changed (because of the varying amount of tree canopy cover) I took a photo of the WhiBal card, and then carried on shooting whatever caught my eye.

After returning, and importing the images in to Lightroom, one of the first tasks is to correct the white balance. Here’s one of the photos of the WhiBal card before adjustment.

Correcting the white balance for this picture is easy. Just press “w”, which brings up an eye-dropper, and click anywhere in the grey face of the card. That colour corrects the image to this.

Synchronising this colour correction is easy in Lightroom. Select the first photo (the one that contains the WhiBal card). It’s important to select this one first, as Lightroom uses it as the master from which the settings are copied. Then select all the other photos taken under the same conditions. Make sure you’re in the Develop module, and hit the Synchronise button in the bottom right of the interface.

A dialog pops up asking which information you want to sync. Make sure that only White Balance is selected.

And that takes this:

to this:

This isn’t a perfect adjustment yet, but it’s a good start. One of the things this would benefit from is adjusting the white and black point in the image — a feature that Photoshop supports but Lightroom doesn’t. As you can see from the shot of the card, it also includes black and white reference targets which makes these sorts of adjustments very easy too.

Grey cards aren’t new inventions. But the WhiBal is a very handy one to carry around, especially in the credit-card sized form factor.

Share This

One recommendation in particular struck me, in re ordering the material in a presentation effectively.

But there is one metric which can be applied to a talk safe in the knowledge that following it would have taken exactly the same preparation time: would the talk have been better had exactly the same content been presented but in the reverse order.

I’ll be keeping that in mind next time I have to present something.

Share This

TripIt exists to make it easy to manage one or more travel itineraries, and to share those itineraries with other people (who can be passive viewers, or active collaborators).

Signing up is easy, requiring the minimum of personal information. Once you’ve registered and activated your account you can start adding to your trip itineraries. It is possible to do this the tedious, manual way, by typing in the details of you trip — flights, car hire, accommodation bookings, and so on.

However, TripIt has an ace up its sleeve.

By forwarding the confirmation e-mails that you receive from the airline, travel agent, car rental company, and so on, to a special address @tripit.com from the e-mail address you signed up with, TripIt’s systems will parse out the relevant details and import them as itinerary items in to your schedule. You can then amend them as necessary. This makes adding details a breeze.

As well as details for flights, accommodation, and car rental you can also add (Google) maps to your itinerary, either as simple maps, or as directions, and notes.

The collaboration features are simple but effective. Enter the e-mail address of someone you want to share the trip details with, and choose whether you want them to be able to view the itinerary, or to be able to add items to it too. They receive an e-mail with a custom URL allowing them to view/edit the trip details as necessary. These invitations can be managed by the original trip owner, and rescinded as necessary. Collaborators can add notes and comments to the itinerary, amend existing events, and add new ones.

For me this feature alone makes TripIt worthwhile. I no longer need to leave my mother detailed information about my travels around the world (she’s a mother, of course she worries). I can just invite her to view the trip details.

TripIt’s able to include other pertinent information in your itinerary that it gleans from the Internet. For example, for flights where it can determine the aircraft type it includes links to the relevant pages on SeatGuru, direct online check-in links, options to check the flight status, a brief display of the weather forecast for the destination, and so forth.

Other useful features include a “TripThis” bookmarklet which can be used to grab URLs of interesting pages and automatically add them to the itinerary, a “Travel Guide” for your destination, containing information culled from sites such as Wikipedia and Flickr, and growing support for “Friends” — not a feature I’ve used, but they describe as:

… which enables you to connect with other TripIt users. Once connected, you can view an online calendar of each other’s travel plans so you can keep track of traveling friends or business colleagues. This is the first of many tools that will make TripIt even more useful to organizing your travel…and organizing your friends too.

and TripSearch — again, not something I’ve used yet, but it aims to make it easier to find and book flights, hotels, and so forth.

TripIt still has a few issues.

• It’s still quite US-centric. For example, when it imports flight details and displays them on the itinerary it includes the appropriate contact number for the airline. But this is the always the US contact number, rather than the local one.
• There’s no support for timezones yet — although this isn’t as big a problem as I’d expected.
• While there’s a calendar view of the trip available via the website it’s not possible to export this (say as iCal format) in to other applications (at least not that I’ve seen).

However, feedback that I’ve submitted to the developers has been responded to promptly, and I’ve seen features that I’ve suggested appear on the site quite rapidly, so it’s entirely possible that these issues will be addressed soon.

Despite these few shortcomings TripIt has proved to be very useful, both in allowing me to easily organise my travel details, and in making it very easy to share these with other people. It also makes it simple to print out a detailed itinerary that contains all the relevant information, which is much nicer than carrying around print outs of half a dozen different confirmation e-mails which all have their information in different formats.

TripIt’s service is free, and they are still in invitation-only beta mode. I have two invites left, and the first two people to request an invite in the comments can have them — just leave your e-mail address.

Update: shortly after posting this I received an e-mail from TripIt with another 30 invites. So don’t be shy in asking if you think it might be useful.

Share This

I’ve had to book a few flights with BA recently. Up until a couple of weeks ago their acknowledgment e-mails came through fine. And then I stopped receiving them. Taking the time to delve in to the mail logs yesterday I noticed this:

Aug 20 07:47:45 jc sm-mta[15347]: l7KElEmk015347: ruleset=check_mail, 
  arg1=website +LHS=RHS@bounce.baplc.com, 
  relay=ceba-mgw04.baplc.com [163.166.43.64], 
  reject=553 5.1.8 website +LHS=RHS@bounce.baplc.com... 
  Domain of sender address website+LHS=RHS@bounce.baplc.com does not exist

(I’ve redacted the left and right hand side of the actual e-mail address it was being sent to)

If that’s just so much gibberish to you, it says that BA are sending e-mails with a return path of …@bounce.baplc.com. Working through the logs shows that they’ve been doing this for some time.

But at some point in the last few weeks, someone at BA has removed the bounce.baplc.com entry from their DNS. So my, and countless other systems around the world, will begin rejecting messages.

This rejection is quite correct. Since bounce.baplc.com doesn’t exist, my system (and any other system with the same configuration) will have nowhere to send any bounces that might occur. And sending messages from domains that do not exist is also an exceedingly common spammer tactic.

I’ve used the “Report problems with our site” feature to report this to BA, but I don’t have high hopes of anyone listening.

Share This

In February I left my job helping to run the mail systems at Citigroup, having sorted myself out with a contracting role doing software development in Perl (with some Autosys and Murex mangling on the side) for the nice people at Brevan Howard. Quite definitely some of the nicest people I’ve had the pleasure of working with in the financial industry. If you’re a London-based Perl developer looking for work and the opportunity for a contract with them comes up, jump at it.

Along the way I finally got around to sorting myself out with a Flickr account, and you can see my photos at http://www.flickr.com/photos/nikclayton/. Not something I’ve written much about, but that may change in the future.

Anyway, that proved to be temporary. No, they didn’t fire me. Instead, in two days time I jump on a plane bound for Zurich, to start working as a site reliability engineer for Google Switzerland.

I was over there a couple of weeks ago as part of a preview-cum-orientation trip, which coincided with the once-every-three-years Züri Fäscht (Zurich Festival), so I took the opportunity to snap a few shots of fireworks.

Anyway, I hope to write more next week about the process, useful sites for people undergoing a similar move, and so on. That, and getting back to contributing to projects like Subversion — my free time has been practically non-existent for the past 4 months or so, and that’s something that I’ve let slip.

Share This

Apparently (and this is second-hand, so treat it with some caution), the Korean Information Security Agency (KISA) is producing an RBL of domains to blacklist. This is complemented by a whitelist, called WhiteDomain, seeded by using published SPF records.

There’s a webpage for the KISA RBL, with both English and Korean versions. It looks like there’s more content on the Korean version than on the English version, but most of it’s text on images which Google can’t translate, and my Korean’s not up to much.

Many Korean ISPs are using this RBL and whitelist

However, the process for seeding WhiteDomain is only correctly handling SPF records that use the ip4: mechanism. Other mechanisms (like a:, ptr:, and mx:) are being ignored. So if your SPF record doesn’t use the ip4: mechanism (and there are many legitimate reasons for not doing so) you are going to find your e-mail blacklisted by many Korean ISPs.

This is broken for a couple of reasons. First, the SPF specification is quite clear about what a conforming SPF implementation must do:

4. The check_host() Function

The check_host() function fetches SPF records, parses them, and interprets them to determine whether a particular host is or is not permitted to send mail with a given identity. Mail receivers that perform this check MUST correctly evaluate the check_host() function as described here.

An implemenation that claims to process SPF, but that does not follow parts of the specification is broken, pure and simple.

Second, there’s nothing preventing spammers from publishing their own SPF records, authorizing their mail relay. If WhiteDomain is being seeded purely by SPF (and I don’t know for certain that it is, so take this with an appropriately sized piece of salt) it’s likely to become less and less useful over time.

Share This

So, a few days ago I was pleased to discover a new mode that I’d not previously known about.

ecb, the Emacs Code Browser is a minor mode that makes browsing a collection of source code easier. Activate it (assuming it’s installed) with M-x ecb-activate. It defaults to creating four new buffers within the current window. These buffers are displayed next to your primary editing buffer.

The first shows a directory tree view of the filesystem, centred at whichever directory contains the currently open file. The second shows a list of all the files in the currently selected directory. The third shows a list of all the items of interest in the current file (I’ll come back to that in a moment). The fourth shows a list of the currently open files.

The third pane, which shows items of interest, is particularly useful. Here’s a screenshot, where I’ve got a number of .c files open for editing.

Click to enlarge

That third pane (mostly green in the thumbnail) is a list of all the functions (along with their type signature) in the file. Selecting one of those functions jumps to its definition in the file, and makes navigating large source files much easier.

The list of interesting items is not limited to just function definitions. In this screenshot I’m in the middle of editing a Perl module (.pm).

Click to enlarge

As well as containing functions, the list now contains information about documentation in the file (in POD format), and package declarations. Again, this making zipping around the file much easier than repeatedly seaching for “sub foo {“.

I’m still scratching the surface of what ecb can do though.

Share This

Sendmail has supported pausing before displaying the greeting as an anti-spam measure since 8.13.0.

To use it, add a line like:

FEATURE(`greet_pause', `10000')

to your sendmail.mc file, and rebuild the sendmail.cf file. The delay is specified in milliseconds, so that example will cause Sendmail to delay for 10 seconds before displaying the greeting banner.

What that feature was introduced, Sendmail would log connections that fell foul of this check with this line:

rejecting commands from host [ip-addr] due to pre-greeting traffic

This is helpful in determining how many times the check fired (and therefore how many connections it’s stopping). The hostname and IP address information can also be mined from the logs and used to update any blacklists or firewall rules you’re maintaining — if the site is going to trigger this block then you may as well prevent from connecting, and save some connection slots.

However, it doesn’t tell give you any idea of what would happen if you were to increase or decrease the greeting pause time. Suppose you have the delay set to 10 seconds. It may be the case that all the violators are trying to send data within the first second. In which case a 10 second delay is going to penalise all connections for an unnecessary 9 seconds.

On the other hand, it might be the case that blocks because of a 10 second greet pause delay are spread out over the entire 10 seconds. In which case there isn’t much opportunity to reduce the delay, and it could be argued that increasing the delay makes sense.

To be able to answer questions like this I sent in a patch, and from Sendmail 8.14.0 (or 8.13.5 if you enabled the _FFR_LOG_GREET_PAUSE feature) Sendmail now reports:

rejecting commands from host [ip-addr] due to pre-greeting traffic after num seconds

This is sufficient information to produce a histogram that shows the frequency of the delay at the time that connections are rejected because they triggered this check. Here’s an example of what such a report might look like.

# == 1 connection
 Delay   Count          Histogram
     0  (    12)        ############
     1  (     9)        #########
     2  (    14)        ##############
     3  (     9)        #########
     4  (    11)        ###########
     5  (    27)        ###########################
     6  (     8)        ########
     7  (    13)        #############
     8  (     6)        ######
     9  (    11)        ###########

In this example it’s clear that the pre-greeting checks are firing evenly across the range of delays — with the exception at 5 seconds. There’s a definite spike there. That could be inidicative of a particular popular piece of spam-ware that’s configured to only wait 5 seconds before sending mail.

Here’s the Perl script that produced that report. Happy spam hunting.

#!/usr/bin/perl
 
use strict;
use warnings;
 
use List::Util qw(max);
 
# Read maillog from stdin, produce histogram showing greetpause
# distribution
 
my %pauses    = ();
my $max_count = 0;
 
while(<>) {
    next unless $_ =~ /pre-greeting traffic after (\d+) seconds/;
    $pauses{$1}++;
    $max_count = max($pauses{$1}, $max_count);
}
 
my $scale = int($max_count / 40) + 1;
 
print "# == $scale connection", $scale == 1 ? '' : 's', "n";
 
print " Delay\t Count\t\tHistogramn";
foreach my $delay (sort keys %pauses) {
    printf "%6d\t(%6d)\t%s\n", $delay, $pauses{$delay},
        '#' x ($pauses{$delay} / $scale);
}

Share This

First, the new HeloName option.

When Sendmail needs to deliver mail to a different server, it connects to the remote server, waits for it to send the greeting banner, and then sends:

HELO the.host.name

(or EHLO the.host.name, but I’m keeping this simple). Sendmail uses the local, fully qualified hostname of the host when it does this. So if you host is called foo.example.com Sendmail will send HELO foo.example.com.

What happens next depends on how the remote system is configured. It is very typical for the remote site to do a DNS lookup on the provided hostname, to make sure that it exists. This is a common anti-spam measure. If the given hostname doesn’t exist then the rest of the connection will be rejected.

Further, it’s also common to do the reverse lookup. So if foo.example.com resolves to the IP address 1.2.3.4, the remote site will then look up 4.3.2.1.in-addr.arpa, and verify that that resolves back to foo.example.com. Again, if it doesn’t, the connection is refused. This is a less common anti-spam technique, but it’s still found in regular use at hundreds of thousands of sites around the world.

You’ve probably worked out how this can cause problems. It’s common, especially in corporate environments, for hosts to have multiple hostnames, especially if they are internet facing hosts. There’ll be one name by which the host is known internally (and is only published in the internal, corporate DNS), and one name (and associated IP address) by which the host is known externally.

This can lead to situations where the mail server connects to the remote server, and sends:

HELO my.internal.name

my.internal.name is not published in the public DNS, so the remote server rejects the rest of the connection.

It may be even more complicated than that — for example, the internet facing mail relay might be behind a firewall that carries out Network Address Translation (NAT) — in which case, even if the server is configured to use the correct hostname in the HELO step, that hostname might not resolve back to the right IP address, and again, the connection may be refused.

Prior to the HeloName option being added to Sendmail you could use the confDOMAIN_NAME configuration option to specify the full domain name to use.

This works, but has other side-effects. In particular, the value of confDOMAIN_NAME appears in Received: headers, and if you set it obscures the actual hostname, which can make analysis of messages through the Received: headers more difficult.

This is the situation I ran in to at $work. We have many Internet facing mail servers relaying mail to the outside world. Each server knows its own internal hostname, but has no knowledge of the hostname it presents to the outside world, nor does it know the IP address that the packets it sends will finally have (all the servers are behind NAT devices).

I considered configuring each server with this information, but decided that that would be a bad idea. It would have meant tightly coupling our configuration information with configuration information maintained by an entirely separate group (the Networks Team). Should they decide to renumber a NAT’d IP address they would need to ensure that we were informed, and that the change was co-ordinated.

That’s extra work, extra complexity, and extra opportunity for things to go wrong.

Each server’s IP address is already in a large A record, mail.example.com. I considered adjusting confDOMAIN_NAME and related settings so that each outbound mail relay considered itself to be mail.example.com, but rejected that idea on the previously mentioned issues relating to Received: lines and the extra complexity it introduces when debugging.

So, plan C. Specifically, I wrote a small patch for Sendmail that introduces a new configuration option, HeloName. If set, Sendmail will use the value of this option when it sends HELO/EHLO, instead of the fully qualified hostname (or confDOMAIN_NAME setting).

Our Sendmail configuration files now contain this directive:

define(`confHELO_NAME', `mail.example.com')

This is sufficient for all our outbound mail relays to HELO with the same name. This name contains all their IP addresses. And the Received: headers still contain the correct internal hostnames, to make troubleshooting easier.

If we’d been using a closed source MTA this wouldn’t have been possible.

Share This

After my last post I proceeded to install VMWare Server in Kubuntu. That went relatively easily, and I prepared a Windows XP VMWare image.

The image creation was also relatively smooth, although it took 2 or 3 times longer than a normal Windows install. For some reason anything involving disk access (so, most of the installation) seemed to stall at various points.

Anyway, with Windows installed I then went on to install Steam and Half-Life 2 from the installation DVD. Again, there’s nothing special to write home about that process, and Steam went and downloaded the many megabytes of updates that have accumulated since the last time I played Half-Life 2 (2+ years ago).

By this point my hopes had been raised pretty high. Sadly, they were to be dashed. Although Steam was apparently running successfully, attempting to launch any of the games would turn the (Windows) cursor in to an hourglass for a few moments. There’d be a bit of disk thrashing. And then the hourglass returned to a pointer. Other than that there was no sign of any activity to indicate that the game had started (or might be about to start).

So I looked at alternative solutions. I found this forum post about Windows games on Ubuntu that had some pointers. I tried Wine. However, although the installation seemed to go well, and I could use it to run simple Windows apps (wordpad.exe, et al), it didn’t do anything when asked to run the Half-Life 2 installer — it just returned almost immediately to the command line with no apparent error messages.

I briefly considered Cedega. But that’s pay-ware, and their online compatability database suggests that there are still some problems running Half-Life 2.

So, plan B.

I’ve ditched Kubuntu, and instead installed Windows as the primary OS on the machine. I’ve got VMWare Server for Windows installed now, and I’m installing FreeBSD 6.2 and Solaris 10 virtual machines so that I can pick up the DTrace work from where I left off.

The Windows installation was something of a palavar. My original Windows XP CD is old. So old it predates Service Pack 1, never mind Service Pack 2. Amongst other things this means that the installer doesn’t recognise disks large than 128GB. Since the Ultra 40 has a 250GB disk this would have required me to partition the disk, something I’m loathe to do.

After first installing Windows I experimented with the Disk Management Tool to see if I could grow the partition (after installing SP2). However, that failed, with an error explaining that this wasn’t possible because the disk had originally been partitioned by an older version of XP.

So I did some digging on the web and turned up Autostreamer.
In a nutshell, this takes your existing, pre-SP2 install disc, a copy of SP2, and creates a new XP install disc with SP2 installed. Booting from this I was able to create a partition that spanned the entire disk, and install XP SP2.

So in the prior 48 hours I’d managed to install XP 3 times — once in to VMWare, once from a pre-SP1 image, and finally from an SP2 image. Frankly, that’s enough for my lifetime.

Oh, and Half-Life 2 is now installed. So I’ve had a happy few hours spent blasting Combine too. Now if only they’d hurry up and release Team Fortress 2.

Share This

I have spent a little bit of time carrying out some research. My initial plan was to triple boot this server. First as a FreeBSD machine — it’s my preferred Unix flavour, my other Unix boxes run FreeBSD, and it gives me the opportunity to carry out FreeBSD testing on a 64 bit platform.

Then with Solaris 10. I want to carry on the work that I’ve already started with DTrace, and get some more experience with other Solaris technologies.

Finally, as a Windows XP box. Two reasons for this. First, Photoshop. I take quite a lot of pictures, and, not currently having a Dual Core Mac to hand, Photoshop on XP is my preferred image editing system. I also have something of a workflow built around it. The other reason is games. I don’t get much opportunity, but when it arises I can quite happily while away the time playing Half Life 2.

That’s not quite what happened.

I was happy with the triple boot idea as I already have a primary desktop (a somewhat venerable 2×1GHz PIII), so the hassle of triple booting wasn’t going to be too much of a problem. Unfortunately, about 2 weeks ago this desktop decided to give up the ghost, with three hard disks in it failing in fairly short order.

A colleague at work has been talking about VMWare for some time, and has it running on his (Linux) laptop. I’d seen it in action a few times, so a new plan started to form.

Specifically, I’m going to run Linux as the primary OS on this machine. I’ve heard good things about Ubuntu (in terms of being relatively easy to install and get going — I’ve reached an age where dicking about with system installers is no longer considered a productive use of my time). I’m going to use the Kubuntu distribution, because, as I’ve mentioned previously, I don’t get on with GNOME, preferring KDE if I’ve got the choice.

On top of that Linux distribution I’m going to install the free VMWare Server. And in to that I’m going to install FreeBSD, Solaris, and Windows distributions. This will be something of a learning experience, as I last used Linux over a decade ago. I imagine a few things have changed. I’m also interested to see just how much of a performance hit VMWare imposes (especially with my gaming head on).

Hopefully this will make it easy for me to try out other things that have been happening in the FreeBSD space (like PC-BSD).

So I’m sat here writing away with a Kubuntu 6.10 CD prepared and ready to go. I’m just going to boot for first time, see what happens, verify that everything comes up as it should, and then get on with the Kubuntu install.

13.58: Plug everything in and turn it on. The fans make a God-awful noise when they start up, which subsides after 30 seconds or so.

The boot process seems stuck acquiring DHCP lease, which is odd, as everything else on this network has no problems with DHCP. Network cable firmly plugged in to socket marked “0″, perhaps it should be in “1″.

14.04: Possibly not DHCP, as the error messages that have now come up talk about PXE booting, and it shows “Operating System not found”. Why is it PXE booting, instead of booting off the hard disk? Solaris is supposed to be pre-installed.

14.06: Try swapping the network cable to port 1 and reboot. That has some effect — the boot now fails much faster, and displays the “Operating System not found” message again. I wonder if the hard disk is not properly seated.

14.08: Try swapping network cable back. The side of the box has quick release catches, which means that opening it up, pulling out the hot-swappable disk, and reinserting it takes seconds. Try rebooting again.

14.14: Same PXE problems. Time to reboot and check the BIOS. This is interesting, the BIOS boot menu shows that the SATA drive is excluded from the boot order for some reason, with the PXE boot first on the list. So I’ve shuffled that around and now the DVD is checked first, followed by the hard disk, with no PXE boot. Saved and rebooted…

14.16: That’s better. The GRUB boot menu comes up, and Solaris starts. “SunOS Release 5.10 Version Generic_118855-15 64-bit” according to the start up splash screen. That’s enough of that, time to start with Kubuntu.

14.17: I reboot with the Kubuntu CD in the drive.

That’s nice — I can select the keymap straight away. Last time I did this with Solaris that setting wasn’t carried through to the final X configuration, I’m interested in seeing if it works with Kubuntu. I choose the “United Kingdom” mapping.

I choose the “Install” option, and a progress dialog comes up. The text is “Loading Linux Kernel”. This rapidly gets to 95% and then halts.

14.25: Been stuck at 95% for 7 minutes now. That’s a bit excessive. I’m going to reboot and try again. It’s just possible that the CD is damaged in some way. In which case I’ll try once more, and if it fails again, boot in to Solaris and burn the Kubuntu ISO image from there.

14.27: Noted a “Check CD for defects” option on the Kubuntu boot menu, which is a nice touch. I’m running that now. There’s a very dark blue note at the bottom of the screen that this “may take some time”.

14.34: Progress meter on the CD check is now at just over 75%.

14.36: The check has now completed. Took me a moment to realise, as the “Check has now completed, no defects found” message is displayed in that very dark blue text on a black background, which is not very legible.

This time around the kernel loads successfully. CD grinds away for a bit. Ah ha, a blue desktop with a mouse pointer has appeared, shortly followed by a list of items that are being ticked off one by one. This looks a lot like a normal KDE startup to me, albeit in 800×600. I double click the installer application, and walk through the installation wizard.

Bonus points for carrying over my keyboard layout preferences. £, “, and @ are all exactly where I expect to find them.

Disk partitioning is pretty simple. At the moment I’m going for one big partition (with separate swap).

14.54: I appear to be doomed to face progress meters that get stuck. The “Creating ext3 file system for / in partition #1 of SCSI1…” progress bar hasn’t moved for the last 15 minutes, which is unnerving. Is this expected with ext3? I know that FreeBSD’s newfs() would complete on this in a matter of seconds.

What’s especially galling is that although I appear to be in KDE and the mouse moves, none of the menus are currently responding. So I can’t, for example, fire up an xterm and do some digging in to exactly what’s going on at the moment. I shall leave this for another five minutes and then if there’s no progress reboot and try again…

15.06: Rebooted, and gone through the installation process again. Looks as though the earlier install might not have hung, as the partition editor is now showing some of the disk as occupied. I guess the lesson here is to leave it running. So that’s what I’m going to do now.

15.20: Progress dialog has now sprung in to life again. Partitioning/formatting appears to have completed and files are now being installed.

15.40: Everything appears to be installed, so, as prompted, I reboot.

The first problem is that the display is 1024×768. That’s all very well, but the LCD monitor is capable of 1600×1200, so I’m peering at a very small display surrounded by lots of black pixels. I’ve not had this problem with this monitor in X on FreeBSD before.

Adept has popped up and told me that there are 99 packages that need updating, so I’ve let it go off and update those. While that’s happening I’m going to poke around and see what’s wrong with the screen.

KDE / Kubuntu / X doesn’t appear to have anything special for the graphics card and monitor. The monitor, an Iiyama AU4831D isn’t listed in the monitor database, and had to go grovelling through /var/log/Xorg.0.log to determine that it was a Quadro FX 1500.

I’ll try and update to the newest Nvidia drivers and see if that has any effect…

… nope, nothing.

Ah ha. Some judicious searching has turned up this post on the OpenSolaris forums.

To save you the trouble, or should it disappear, if you have an Iiyama AU4831D and you want to get the full resolution you need to add:

    Option "ModeValidation" "NoDFPNativeResolutionCheck"

to /etc/X11/xorg.conf, otherwise the driver gets the wrong idea about the maximum resolution supported by the monitor. Adding that and restarting the X server gets everything going in glorious 1600×1200 again.

What with the updating, configuring KDE to behave the way I like, and futzing around trying to sort out the monitor it’s now 17.00 and I’ve got a few errands to run. So that will probably be all I’ll have time to write today. More updates tomorrow.

Share This

This was almost a disaster for me. I was coming from docklands, and the DLR was suspended due to the storms that hit Europe. What should have been a 20 minute journey to Bank station turned in to a 50 minute nightmare. I was due to give the first presentation, and had visions of causing problems for everyone else. And visions of fainting. Mental note — next time do not give blood 2 hours before delivering a talk. Or perhaps, better not to deliver a presentation 2 hours after giving blood.

Anyway, I just made it in time. The transport disruption caused by the weather meant that there weren’t as many people as the last get together (about 12 I think) but this also meant that it was also possible to have more inclusive discussions about points raised by the presentations.

I think mine went well. There’s a PDF, slide-only version of “Subversion and Perl, a tour through CPAN“. It ended up a tiny bit rushed — 24 slides in 25 minutes. I should probably allow 90 seconds per slide for those sorts of slides next time. It’s a difficult balance to strike when writing them. It’s all very well using the Takahashi Method (big text, very few words, clkao did that to great effect at the previous meeting), but they’re not necessarily going to be very useful to the audience afterwards without extensive notes.

Duane Griffin commented to me that the information about SVN::Notify had saved him some work, so I know at least one person found it useful :-)

The next presentation was from C. Michael Pilato, going through some of what’s up-and-coming in Subversion over the next few minor releases (and 2.0). No real surprises for me there, since I follow the dev mailing list. I was disappointed to discover that this was a conference call. When I saw his name on the schedule I thought CollabNet might have flown him over to do some customer consulting. I was looking forward to shaking his hand and saying “Thanks for the commit bit”.

Andrew Heald followed this with a “lessons learned” discussion / presentation about his experiences migrating a relatively large group of developers (around 80 or so) from CVS to Subversion for Catlin. I hope the slides will be online soon.

The last presentation and demonstration was from Guido Haarmans, talking about the recently introduced Open CollabNet.

I’ve had a poke around Open CollabNet previously, and I confess that I’m not quite getting “it”. This might well be because it’s not aimed at me. There’s a fair amount of content on there, and it includes a number of web based forums. But I find that that just makes things harder to find — if you’re looking for Subversion info do you go to Open CollabNet or subversion.tigris.org? And what do the web forums do that the users and dev mailing lists don’t?

And it doesn’t help that there are no updated feeds of the content available. So it becomes yet another website that I have to remember to visit on a periodic basis (and I can’t roll it up in to Planet Subversion).

On the flip-side, at work the next day I needed a site to point someone completely new to Subversion at, and they found Open CollabNet quite helpful. So, as I say, I could very well just not be in the target audience.

And then it was on to the Google Open Source Jam.

I travelled down with Duane Griffin and Dave Hodgkinson, who’d also been at the Subversion meeting. During the journey Duane and I chatted about tools for extracting statistics from Subversion repositories.

The Google event was organised in to two main sections. First, once we’d arrived, chatted a little bit, anyone that wanted to had the opportunity to give a lightning talk (5 or so minutes) about anything interesting that they were working on. A few things caught my eye — mostly in the “That looks interesting” category, rather than the “I’ll be able to make a positive contribution to that project” category. But I’ve got plenty of open source stuff on my plate at the moment, so that’s no bad thing.

Simon Stewart introduced WebDriver, a tool to make it easy for developers to carry automated tests of webapps, even if they use Ajax and similar technologies. Since it’s hosted at Google I was able show Simon the SVN::Web front end to the WebDriver repo, which he seemed impressed by.

Stathis Sideris showed us ditaa. This is very geeky — it takes ASCII art diagrams like this:

    +--------+   +-------+    +-------+
    |        | --+ ditaa +--> |       |
    |  Text  |   +-------+    |diagram|
    |Document|   |!magic!|    |       |
    |     {d}|   |       |    |       |
    +---+----+   +-------+    +-------+
        :                         ^
        |       Lots of work      |
        +-------------------------+

and converts them to something like this;

The other presentation that caught my eye was from Benoit Xhenseval, talking about, amongst other things, StatSVN. This is a Java application that gathers and reports on statistics about Subversion repositories.

As I mentioned earlier, I’d been talking about this sort of thing with Duane on the way over, and it’s this sort of serendipitous meeting that events like this help facilitate.

We talked about it some more. After a bit of digging in to how it works I suggested they consider splitting the code in to two parts — a back end that is responsible for gathering and storing the data, and a front end that can query the data and generate reports (web based, e-mail, and so on). That would make it possible to integrate it with a number of other tools. Obviously, my interest is in bringing something like this SVN::Web.

Some other Perl mongers were in attendance and I took the opportunity to catch up with some of them. Nick Clark detailed some issues he was having with P4 data export that make migration to Subversion difficult for Perl — if I understand correctly it’s about not being able to get at the P4 metadata that details which changesets have been integrated (which revisions have been merged, in Subversion parlance).

I was also able to jump in part way through an overheard conversation about scraping websites and sending to WAP devices. I gave an impromptu intro to Plagger and the wealth of ways it might be able to help with this.

Next time I should remember to take a camera. Stuart Yeates was snapping away, and has a set of photos from the Google Open Source Jam event on Flickr.

Thanks to the organisers, in particular Joe Walnes — providing space, projects, wireless, food, and drinks works well. Maybe not conducive to actually getting patches committed there and then (perhaps that’s just me), due to eclectic nature of projects. But it’s a great communication enabler.

Share This

As is traditional, my first commit was to the list of Subversion committers. This ensures that the ACLs are set up correctly and so on.

I’ll be working on the perl-bindings-improvements branch, making sure that they’re up to date with respect to the rest of Subversion, and that they behave as “Perlishly” as possible for anyone else who needs to interoperate with Subversion repositories using Perl.

Share This

Share This

One of Perl’s strengths is that “There’s more than one way to do it” (TMTOWTDI, often pronounced tim-toady). This makes for a language that is flexible, and allows you to write a solution to a problem that matches the problem, rather than trying to shoehorn the problem in to the language in order to solve it.

This is also one of Perl’s weaknesses. There can be many ways to solve a particular problem — some of them are not optimal.

One of those, that I’ve rolled countless home-grown solutions for over the years, is that of parameter validation and handling. When writing code (especially library code that’s probably going to be called by code other than your own) it pays to be defensive. Catch and report errors as soon as possible and you can save yourself (and quite possibly someone else) a boatload of debugging time.

Perl’s parameter passing is very flexible, adhering to the TMTOWTDI principle. Which also means there are lots of ways to carry out parameter validation. Here’s the approach that I’ve settled on. I hope you find it useful too.

I’m lazy. I want to hand as much of the work off to someone else’s code as possible. To that end I make use of two other Perl modules. Params::Validate and Exception::Class. Together, these two can do all the heavy lifting of detecting errors in the parameter list and reporting them in a coherent manner to the caller.

Consider a hypothetical module, called Example::Module. This module provides one function, which accepts a hashref, the keys of which specify parameter names, the values of which are the parameter values.

Here’s how I’d start that module.

  1 package Example::Module
  2
  3 use strict;
  4 use warnings
  5 
  6 use Params::Validate qw(:all);
  7 use Exception::Class(
  8     'Example::Module::X::Args'
  9         => { alias => 'throw_args', },
 10 );
 11
 12 Params::Validate::validation_options(
 13     on_fail => sub { throw_args(error => shift) },
 14 );

The first four lines declare the module and enable’s Perl’s strictness checks and related warnings. These are more or less boilerplate, and should be a standard part of any code written in Perl (with, perhaps, the exception of one liners written on the command line).

Line 6 loads Params::Validate, and imports the functions and constants that it defines in to the caller’s (Example::Module) namespace.

Lines 7 through 10 are more interesting, doing a number of things.

The Exception::Class module is loaded. It’s import() routine is called, and told to create a new package, Example::Module::X::Args. I use this convention frequently now — exceptions are created in namespaces based on the main namespace, with an appended “::X”, followed by the type of exception. In this case these exceptions relate to problems with the arguments passed to functions. The alias directive creates a new function, called throw_args(). If called, this function will throw an exception of the type Example::Module::X::Args, passing along any parameters it’s given.

Lines 12 through 13 make use of this. Now that throw_args() has been created, Params::Validate::validation_options() is called. This provides an interface in to the inner workings of Params::Validate. This stanza configures Params::Validate to call the newly created throw_args() function is parameter validation fails. The error => shift code ensures that any error message created by Params::Validate is included in the exception.

That’s all that you need to do in terms of set up. Now you just need to make sure that any function that requires parameter validation uses the Params::Validate functions appropriately.

Suppose we have one function in this module, frob(), that takes a hash ref of arguments:

  1 sub frob {
  2     my $args = validate(@_, {
  3         foo = 1,
  4     });
  5     
  6     return 1;
  7 }

That’s enough to declare that foo is mandatory parameter. Now if you try and call that function without passing in that parameter your application dies with an appropriate message.

Example::Module::frob();
 
# Dies, printing "Mandatory parameter 'foo' missing in call to Example::Module::frob"
 
Example::Module::frob(foo => 1); # works

Exception::Class has other tricks up its sleeve. For example, instead of dying with a single error message you can have it generate a stack trace any time an exception is raised. By inserting this line before the call to validation_options:

Example::Module::X::Args->Trace(1)

this tracing facility is enabled.

Exception::Class can be used to easily throw other errors as well — errors that it might make more sense to trap. Consider an application that retrieves and processes data from a database. Your Exception::Class import might look like this:

use Exception::Class(
    'Example::Module::X::Args'
        => { alias => 'throw_args', },
    'Example::Module::X::DB'
        => { alias => 'throw_db', },
);

Now, if your library needs to indicate that a database error has occured it can:

throw_db("Text of the error message");

and these errors will be distinguishable from errors due to incorrect arguments. More importantly, these errors are distinguished by the class in which they are defined, rather than the text of the error message — distinguishing errors programmatically by the text of error messages is fragile, and prone to problems when messages are changed, either because the text has been reworded, or it has been translated.

You can take this further and produce class hierarchies of exceptions as necessary.

use Exception::Class(
    'Example::Module::X::Args'
        => { alias => 'throw_args', },
    'Example::Module::X::DB',
    'Example::Module::X::DB::Connect'
        => { alias => 'throw_db_connect', },
    'Example::Module::X::DB::Prepare'
        => { alias => 'throw_db_prepare', },
    'Example::Module::X::DB::Query'
        => { alias => 'throw_db_query', },
);

This allows for distinguation of exceptions caused by argument parsing, database connections, query preparation, and query execution.

Edited to add:

You would do that like so:

# Use the 'isa' option to explicitly specify a class hierarchy
use Exception::Class(
    'Example::Module::X::Args'
        => { alias => 'throw_args', },
    'Example::Module::X::DB',
    'Example::Module::X::DB::Connect'
        => { alias => 'throw_db_connect',
             isa   => 'Example::Module::X::DB', },
    'Example::Module::X::DB::Prepare'
        => { alias => 'throw_db_prepare',
             isa   => 'Example::Module::X::DB', },
    'Example::Module::X::DB::Query'
        => { alias => 'throw_db_query',
             isa   => 'Example::Module::X::DB', },
);
 
# Carry out some work.  Trap any exceptions, and handle them
# at the end
eval {
    my $db = connect_to_db();   # might throw ::X::DB::Connect
    my $sql = get_sql_from_phrasebook(); # might call die()
    my $q = prepare_query($db); # might throw ::X::DB::Prepare
    my @results = query_db($q); # might throw ::X::DB::Query
    do_something_with(@results);
}
 
# Handle any exceptions.  If a ::DB::Query exception occurs then
# print one set of diagnostics.  If any other ::DB exception occurs
# print another set of diagnostics.  If any other exception occurs
# (e.g., ::Args, or the code died for some other reason) this is a
# "can't happen" situation, and the best thing to do is to die here
# with debugging information.
my $e;
if($e = Example::Module::X::DB::Query->caught()) {
    print "There was an error querying the database\n";
} elsif($e = Exception::Class::X::DB->caught()) {
    print "There was an error working with the database\n";
} elsif($e = Exception::Class->caught()) {
    print "Something unexpected failed.  The error was:\n";
 
    # rethrow the error
    ref $e ? $e->rethrow() : die $e;
}

This has hoisted the error checking code out of the way of the code that does the actual work (which is in the eval block). This makes the functionality of this code clearer and easier to understand. The error handling logic is also in one place — so if this sort of error handling is common throughout the application it can be abstracted away in to a separate function.

End of edits

Not every application is going to need this sort of exception hierarchy (SVN::Web, for example, has a single exception type, because any error basically means do the same thing; prepare an error message and show it to the user), but it’s nice to know it’s there.

Here’s an example that you can cut/paste and play around with. It shows a second function, frob2(), which uses a few more of the Params::Validate features. It also has various commented lines, which you can uncomment as necessary, to experiment with the effects of the parameter validation and the exceptions that are thrown.

#!/usr/bin/perl
 
use strict;
use warnings;
 
package Example::Module;
 
use strict;
use warnings;
 
use Params::Validate qw(:all);
use Exception::Class(
    'Example::Module::X::Args',
        => { alias => 'throw_args', },
);
 
# Change this to ->Trace(1) to enable stack traces for argument errors
Example::Module::X::Args->Trace(0);
 
Params::Validate::validation_options(
    on_fail => sub { throw_args(error => shift) }
);
    
# Show basic parameter validation and exception throwing.  The 'foo'
# key is mandatory, all other keys are forbidden.
sub frob {
    my $args = validate(@_, {
        foo => 1,
    });
 
    return $args->{foo};
}
 
sub frob2 {
    my $args = validate(@_, {
        foo => {
            type => SCALAR,  
            default => 'The default',
        },
    });

    return $args->{foo};
}

package main;

# Uncomment to see missed parameter handling
#print Example::Module::frob(), "\n";

# Uncomment to see extra parameter handling
#print Example::Module::frob(for => 1, bar => 1), "\n";
 
# Uncomment to see what happens when the params are correct
#print Example::Module::frob({foo => 'bar'}), "\n";
 
# Uncomment to see parameters with defaults
#print Example::Module::frob2(), "\n";
#print Example::Module::frob2({foo => 'bar'}), "\n";
 
# Here's an example of exception handling.  This section will never
# run if any of the previous calls (esp. to frob()) failed, because
# they were not wrapped in eval { }, and will have caused the program
# to abort.  This error will not cause the program to abort, because
# it's wrapped in eval { }.
my $e;
eval {
    print "Just before eval'd call to frob()\n";
    # Uncomment the next line to see how other errors are handled
    #die "This is not an argument error\n";
    print Example::Module::frob(), "\n";
    print "Just after eval'd call to frob()\n"; # won't print
};
 
if($e = Example::Module::X::Args->caught()) {
    print "An argument error occured, but we're continuing\n";
} elsif($e = Exception::Class->caught()) {
    print "Something unexpected failed.  The error was:\n";
 
    # rethrow the error
    ref $e ? $e->rethrow() : die $e;
}
 
print "Made it to the end\n";

Share This

There are a number of answers to this particular concern.

First, there’s the community of developers that develop the software, and the wider community of other users. It’s something of a cliche, but I’ve always received better support from a user group mailing list than I’ve ever received from a commercial vendor. In fact, one of the first questions I ask of any vendor I have to deal with is “How can I send a message to/ask a question of your other customers?”

Second, if you’re large enough, you do the support in-house. This is part of my role at $dayjob, where the open source version of Sendmail is used extensively. If it breaks, or we need a feature adding, or a bug fixing, it’s down to me (and my team) to do it. We might then talk to the community (and feed our changes back, see the release notes for 8.14.0b4 for example) but internally we’re the support group. Recipients of this message are not always comfortable hearing it, but so far I’ve been successful in hammering away at this point.

Third, you can choose to pay for third party support. There are a number of organisations that do (or did) this. In some cases it’s a large part of their reason for being — for example, much of RedHat’s success came from being a vendor that large organisations (who didn’t or couldn’t take point 2 on board) could sign a support deal with. Then there’s companies like CollabNet, who provide (amongst other things) commercial support for Subversion, or Sendmail Inc. who do the same for Sendmail. All of these organisations can (and in many cases have) also build on the open source software and provide their own proprietry benefits, tools, applications, documentation, and so on.

OpenLogic are another organisation providing third party support, training, and consulting, but with something of a twist. They have actively engaged with (some of) the open source community, and formed what they call the “OpenLogic Expert Community”. As their site says:

OpenLogic and the OpenLogic Expert Community bridge that gap by providing a single avenue for obtaining expert support on over 200 open source projects. OpenLogic’s internal technical support team acts as the frontline of support, answering customer questions and investigating incidents. For more complex issues, OpenLogic turns to the OpenLogic Expert Community. These open source experts work in partnership with OpenLogic to troubleshoot and resolve customer problems. Through the Expert Community, OpenLogic is connecting enterprises users with committed, passionate and expert developers dedicated to the success of open source.

This piqued my interest at the beginning of 2006, and I signed up as a member to help out. I’ve been a bit busy since then, and haven’t had an opportunity to do much, but I was fortunate enough that over the Christmas period some queries came in that were in my areas of expertise.

The result’s a deposit in to my Paypal account, and a shiny Xbox 360 on its way to me.

As a bonus, there’s a regularly updated, informative (and informed) OpenLogic blog.

Share This

There are quite a few significant changes in this version…

Chief amongst them is that SVN::Web now supports accessing remote repositories. Your SVN::Web instance no longer needs to access the same filesystem as the repository. This makes various interesting tricks possible too.

This entailed switching to use the SVN::Client API when accessing repositories. This made it possible to support “svn blame” style functionality, which is another feature that people have been asking for.

The mechanism for passing log messages through various filters (e.g., to turn e-mail addresses in the message in to hyperlinks) has also changed, to be more general, and support different filters for different repositories.

The complete changelist is available, as are the updating instructions for migrating to the new version.

Share This

I’ll be there, giving a tour of the Subversion related applications and tools that are available in the Perl space.

And after that it’s the first London Google Open Source Jam of 2007 — an opportunity to catch up with other open source developers, and find out more about interesting projects that people are working on.

Share This

1. Send spam
2. Host phishing sites
3. Carry out denial of service attacks

There are many DNS based Black Lists (DNSBLs) operated by numerous groups that aim to list the IP addresses of systems that have been compromised in this way. This allows mail server operators to configure their systems to query these DNSBLs and reject messages that are being sent from a compromised system.

The problem with this approach is that it’s not visible to the owner of the compromised system. They might notice that it’s behaving a little slower, or that their network connection doesn’t seem as fast as it was, but they’re not going to know why, because there’s no easy mechanism to alert them.

In a perfect world, Internet Service Providers would monitor these DNSBLs, notice when IP addresses of their customers appear on them, and terminate (or provide limited) service to that customer, along with appropriate assistance to help them clean their system.

In practice this rarely happens.

It occured to me that one way to make the fact that their PC has been compromised more visible to end users would be by enlisting the help of companies that host or provide online game environments.

For example, consider Second Life, Eve, or World of Warcraft. All huge, multi-player games/environments, to which millions of people connect every day.

If the companies that host these environments were to check the IP addresses of connecting systems against these DNSBLs, they could provide a warning to the player that it’s highly likely that their PC has been compromised, and that they should make sure their anti-virus is up to date, and so on.

Further, suppose you’ve got a PC and an XBox or Nintendo Wii at home¹. Both of those game systems support online play. And through a networking quirk that I don’t need to go in to here (NAT), it’s highly likely that the PC and the games console(s) are going to appear online with the same IP address.

So if the PC appears on a DNSBL the Wii or XBox is going to appear on that DNSBL too. This provides an opportunity for Microsoft and Nintendo to check the IP address, and again, place a warning in the “dashboard” (XBox dashboard, Nintendo Wi-Fi Connection) that their systems display to the user as they go online.

This could significantly raise the awareness of owners/operators of compromised PCs.

¹ Guess what I got for Christmas :-)

Share This

But not, perhaps, in the way that you might imagine.

As already chronicled, some of my writing recently made it in to The Guardian. As is the way of these things The Guardian like to pay their writers, so I sent off my details to their billing department and waited for the money to come rolling in (as you do).

It turns out that, by an odd coincidence, I’m not the only Nik Clayton to write for The Guardian. I’m not even the first. This other Nick Clayton (note the extra “c”) has written a number of columns for them, and they’re also about technology matters.

This much became apparent when I received an e-mail from The Guardian’s billing department today confirming that they had dispatched payment for two articles that Nick had written to me. This e-mail contained Nick’s name and address details, and the payment details (amounts) for the articles he’s written. But it also contains my bank details (account number and sort code). The money hasn’t been deposited in to my account yet, but I imagine it soon will be.

A bit of Googling turned up Nick’s site, and a bit more Googling turned up a phone number, so I’ve called him, and had the slightly surreal experience of:

Good evening. Could I speak to Nick Clayton?

Speaking

Hi. It’s Nik Clayton here…

Now I know how Dave Gorman must feel.

I’ve tried calling The Guardian’s billing department but the number given in the e-mail redirects to voice mail at the moment, so I’ll be in touch with them again tomorrow morning.

There are at least four risks here.

First, The Guardian’s billing department will apparently change the sort code, bank account, and e-mail address details that they hold for writers on the basis of a single unauthenticated e-mail. My message to them was:

Charles Arthur asked me to send my payment details for

http://technology.guardian.co.uk/online/insideit/story/0,,1954392,00.html

to you.

Sort code is XX XX XX, the account number is XXXXXXXX.

Please let me know if there are any problems.

Second, when they pay their writers they send out an e-mail that contains, in clear, the writer’s name, reference number, full address, sort code, bank account number, and the values of the payments. This may well be enough to carry out a social engineering attack.

Third, this could easily have gone the other way, and my bank account details could have been forwarded to Nick Clayton. Had he been nefarious I imagine that (given that we share the same name) these could have been used to carry out a very effective identity theft.

Fourth, had I not been quite so honest I could probably have got away with this for some time — at the very least, continuing to earn interest on the money that The Guardian have paid.

Hmm. I wonder if The Guardian would like to use this as the basis for an article…

Share This

But there’s a problem.

The feeds for the use.perl journals don’t include the full text of the entry. This is a pain — most entries aren’t too long, and I’d much rather read them all in my aggregator of choice, rather than have to click on lots of different links to see each entry in full.

Plagger to the rescue. It can extract the links from the list of new journals, follow the links to find the individual journal entries, and then extract the content from those pages to build a new feed that contains the full text of the entries.

To do this with Plagger requires two config files.

First, config.yaml. You can put this anywhere you like.

plugins:
  - module: Subscription::Config
    config:
      feed:
        - url: http://use.perl.org/search.pl?op=journals
          meta:
            follow_link: /journal/\d+
  - module: CustomFeed::Simple
  - module: Publish::Feed
    config:
      dir: /home/nik/public_html/
      format: RSS
      filename: use.perl.journals.rss
  - module: Filter::EntryFullText

This instructs Plagger to build a feed by starting at the journal page and following every link that matches the regexp /journal/\d+. This feed is published at use.perl.journals.rss, and takes a trip through Plagger::Plugin::Filter::EntryFullText to extract the content for each entry.

Now you need a config file for Plagger::Plugin::Filter::EntryFullText. This needs to be in the correct sub-directory of your Plagger assets path. On my system that’s /usr/local/share/Plagger/assets/plugins/Filter-EntryFullText. If you take a look in that directory (or its equivalent on your system) you’ll see tens of small configuration files.

There’s one configuration file for every site that Filter::EntryFullText knows about.

I created a new file in this directory called use.perl.journal.yaml, with the following contents.

handle: use\.perl\.org/.*?/journal/\d+
extract_xpath:
  title: //div[@id="journalslashdot"]/div[@class="title"]/h3/text()
  body: //div[@class="intro"]
  author: /html/head/link[@rel="author"]/@title
  day: //div[@id="journalslashdot"]/div[@class="journaldate"]/text()
  time: //div[@id="journalslashdot"]/div[@class="details"]/text()
extract_after_hook: |
  my %months = (January  => 1,  February => 2,  March     => 3, 
                April    => 4,  May      => 5,  June      => 6, 
                July     => 7,  August   => 8,  September => 9, 
                October  => 10, November => 11, December  => 12);
  my($m, $d, $y) = $data->{day} =~ /([a-z]+)\s+(\d+),s(\d{4})/i;
  $m = $months{$m};
  my($h, $M) = $data->{time} =~ /(\d+):(\d+)/;
  $h = 0 if $h == 12;
  $h += 12 if $data->{time} =~ /PM/;
  $data->{date} = "$y-$m-$d $h:$M";

There are a few things going on here.

The handle directive contains a regexp — any pages that match this regexp will have their content extracted using this configuration. Because it’s a regexp metacharactes, like “.” need to be escaped.

The extract_xpath section contains directives that tell the plugin how to retrieve certain key bits of data from the page using XPath expressions. title, body, and author are all special keys that correspond to sections of the generated RSS feed.

There’s a problem with the use.perl journal pages because they don’t contain the date and time the entry was published in a trivially parseable format — the information’s human readable, and split over a couple of different elements.

So here the day and time of the journal entry are extracted in to two separate variables. But these need to be post-processed to get a timestamp for the feed.

This is where extract_after_hook comes in. This is a block of Perl code that’s run after the data has been extracted. In this code $data is a hash ref, with keys that match those used under extract_xpath section.

By this point, $data->{day} looks something like Tuesday December 05, 2006, and $data->{time} might be 08:36 AM. That month name needs to be converted to a number, hence the %months declaration, and the time needs to be normalised. So the hour and minute value is extracted. With use.perl the hour “12″ might be midnight or midday, and all afternoon hours are indicated with “PM” in the string. So the hour is reset to 0 if it’s 12, and then 12 is added to all times in the afternoon.

Finally, these values are brought together and assigned to $data->{date}, which is another special key for Plagger.

Here’s the full RSS feed for use.perl journals. Feel free to use this in your aggregator, it is currently set to auto-update every 30 minutes.

Share This

I was reading the FAQ for the hosting service the other day and a particular entry struck me.

Do you have support for ViewVC or similar repository-browsing tools?

This feature isn’t implemented yet, but you can still browse the latest version of your project’s code by entering the repository URL into a web browser.

Since SVN::Web now supports (well, it does on my development site, which will be released as 0.50 real soon now) accessing remote repositories, it should be pretty easy to have it talk to the repositories hosted at Google.

A minor wrinkle is that SVN::Web expects to be configured with a list of available repositories. But that’s only a small code change to solve. So I’ve done that, and put together a proof-of-concept SVN::Web installation.

The practical upshot of which is that if you have a project at Google Code Hosting, you can take this URL: http://jc.ngo.org.uk:81/svnweb/, append the name of your project, and get a decent web interface on to your repository.

For example, the Reconstructor project (chosen at random) gets this URL: http://jc.ngo.org.uk:81/svnweb/reconstructor.

This is very definitely a proof of concept, and may well disappear if it gets completely hammered, and/or starts sucking up all my hosted bandwidth.

Share This

This doesn’t have to be as simple as chaining a few XSLT transformations together, as Plagger filters can carry out additional actions (such as e-mailing the results to you, calling on the power of Perl modules to create summaries, and so on).

As a learning exercise, I’ve built Planet Subversion (edit I’ve updated the URL to point to the official domain). This takes feeds from a number of different sources and builds a "Planet" site from them. And, of course, with Plagger being open source, it’s easy to contribute any fixes back to the author.

Please let me know if you use Subversion and can recommend any other feeds to add.

If you’d like to produce your own aggregation site using Plagger, here’s the config file that I’m using for Planet Subversion.

plugins:
  - module: Subscription::Config
    config:
      feed:
        - http://svn-dev.blogspot.com/atom.xml
        - http://asdf.blogs.com/asdf/index.rdf
        - http://blog.red-bean.com/sussman/?feed=rss2
        - http://www.subversionary.org/blog/martintomes
        - http://tortoisesvn.net/blog/feed
        - http://www.davidglasser.net/point/feed/
        - http://arildf.spaces.live.com
 
  - module: Bundle::Planet
    config:
      duration: 7 days
      title: Planet Subversion
      description: Everything Subversion
      dir: [set this to your preferred output directory]
      url: http://www.planet-subversion.com/
      theme: sixapart-std
      stylesheet: http://bulknews.typepad.com/blog/styles.css

Share This