Benchmarks, love them or hate them, they are here to stay. Though rarely (never?) perfect, they can be helpful and more often than not, misleading. For now though, I want to comment on just one aspect of benchmarks: dates. As a general rule, most people expect benchmarks to change over time because technology generally advances and gets better as time goes on. For my rant on this subject I’m going to pick on the SQLite Database Speed Comparison because it is such a great example of why dates are important to a benchmark (and I am still seeing it referenced, as recently as 31 Dec 2004).
For starters, there is no indication as to when this benchmark was performed. Was it December 2004? October 2003? May 2001? It doesn’t say. Now presumably this benchmark would (should?) carry more weight if it was fairly recent and less weight the older it gets. The only date I could find on the page was the last modified line at the bottom, 2004/10/10. This might lead one to believe that this benchmark was done in the very recent past. There are clues that lead me to believe this is not the case though, specifically the version numbers for the various database systems tested. While I haven’t tracked the SQLite releases very well, I’m reasonably familiar with when PostgreSQL and MySQL releases have happened. So I looked around to find out definitively when each version of the tested software was released. Fortunately each one has a changelog available, with dates:
SQLite 2.7.6: 25 Jan 2003
PostgreSQL 7.1.3: 15 Aug 2001
MySQL 3.23.41: 11 Aug 2001
The most obvious issue here is the huge range in dates. While tested versions of PostgreSQL and MySQL were released in the same month, the SQLite version is 17 months newer than either of the other software versions. At this point I would hazard a guess that this benchmark was probably done not long after January 2003, given the dates above. On this merit alone I pretty much ignore the numbers generated by this benchmark because of the huge gap in development time. We’re talking nearly a year and a half advantage given to SQLite over PostgreSQL and MySQL. Who would possibly think that this would provide any sort of meaningful comparison for someone looking into database benchmarks?
Just by the way of additional information, by January of 2003, MySQL had released version 4.0.10. The MySQL 4.0.x branch was given the ‘Production’ tag two months later in March. Comparing it to 3.23.55 might have also been reasonable, it also came out in January 2003. PostgreSQL released 7.3.5 on 4 February 2003, so they could have used 7.3.1 from December of 2002.
This benchmark is now some where around two years old, making it almost useless for anyone looking for meaningful information today. Of course with out a published date there is now way of knowing this for sure. Unless you did a little bit of homework and looked at when the release of each software version was you could be lead to believe that this test was done three weeks ago. Another important date feature is indicating when the tested version of each software was released. Normally I might not include that as a requirement because it should be reasonable to assume that the most recent production level code available was the one being tested. In this case however that is clearly not the case, so that fact needs to be disclosed.
I’m not picking on SQLite as a product, just using their published benchmark as an example of how excluding date information is a fatal flaw in their document. Notice I did not discuss any of the results or numbers, because in this case once I found out the pertinent dates I’m able to throw out pretty much the entire benchmark because of its age.