More on Atom aggregator XML namespace conformance tests

In my last entry, I asked people to mail me more results. I’ve received feedback from a half-dozen people in mail plus the comments on an entry at Sam Ruby’s, who picked up the story. Herewith the results so far, and a lesson learned.

New results

After accounting for the fact that Thunderbird is actually failing because of an unrelated bug triggered by my test feed, the tally is now:

• Broken:

  • Blogbridge 2.7
  • Bloglines
  • Feedreader 2.90
  • FeedDemon 2.0.0.11 beta1
  • GreatNews 1.0.0.354
  • Firefox 1.5
  • NewsFire 1.2 (v45)
  • RSSOwl 1.2 2005-11-06
  • Safari 2.0.3

• Conformant:

  • FeedTools
  • Google Reader
  • JetBrains Omea
  • Liferea
  • NetNewsWire
  • NewzCrawler
  • RSSBandit
  • SharpReader
  • Shrook
  • Snarfer
  • Thunderbird
  • Universal Feed Parser

Many thanks in particular to James Holderness, who tested roughly a dozen readers.

I have been informed that BottomFeeder parses other feeds structured similarly to the test case correctly, but I can’t do any useful testing myself, because it crashes on me when I click the subscription list. As such, I have removed it from the list for the time being; however, it most certainly has a bug or two that may or may not be related to namespaces.

On the centuries-old art of writing test cases

Thunderbird failed on the test case in my initial test because of an unrelated bug: the feed contains no feed-level permalink, and that broke it; once this is rectified, the test case succeeds correctly. I updated the test case to avoid triggering Thunderbird’s bug.

There is a lesson in this: when you write a test case that checks whether someone else’s software supports a particular feature, you actually need to write two test cases which differ by exactly the one thing you’re testing. Then, you need to check that the baseline without the feature or failure under test works properly, and only then can you draw meaningful conclusions from the result of the feature/failure test case.

Of course, this is simply called the “Scientific method;” and I should have known better.

Given this hindsight, I wrote a test case baseline. The absolutely only thing that differs between the two is the arrangement of namespaces. If you want to test an aggregator, check that the baseline works, first. Meaningful conclusions about the failure of the actual test case can only be drawn iff the baseline works.

Removing a bottleneck

Sensibly, Sam calls this fodder for the Atom Conformance Tests on the Atom wiki. So I’ll be giving this information a new home there. Anyone who cares can then contribute directly.