The world of benchmarks is fraught with peril, those who enter are likely to get flamed, not matter what the results are. I had already prepared myself for the worst when I saw the title of Using MySQL to benchmark OS performance on NewsForge the other day. After reading through it, and part 2, Comparing MySQL performance, I was pleasantly surprised. At the end of the article I got the feeling that Tony Bourke had made an honest attempt at testing MySQL 4.0.22 on the following operating systems:
- FreeBSD 4.11
- FreeBSD 5.3
- NetBSD 2.0
- Linux 2.6
- Linux 2.4
- Solaris 10 x86 (build 69)
- OpenBSD 3.6
While there are things that I’d recommend doing differently, it certainly seems like Tony did a good job to trying to make this as balanced as possible. Perhaps my biggest beef with his methods was the decision to run all of the tests locally, instead over the network. To his credit he does a good job explaining why he ended up not doing so, but that doesn’t change the fact that for those building apps (web or otherwise), don’t usually run that application on the same system that is running MySQL.
The results of the test still feel a little bit odd. I can’t really hold this against Tony though, I’m sure he was working on a deadline and if you put off publishing forever then why bother doing it in the first place. That said, I suspect that there is more that could be done if more time and resources were available. Some of the other obvious possibilities include using MySQL built for that OS (rpm’s, BSD ports, etc), looking at additional file system tweaks and differences (does Linux still default to async fs mounts?) and trying different versions of MySQL (4.1 just went into production, but 5.x betas have been around for awhile too).