Joyent/TextDrive’s Really, Really, Really Bad Netperf Benchmarks

I came across a Scales With Rails presentation at http://www.scalewithrails.com/downloads/ScaleWithRails-April2006.pdf that focused on scaling Ruby on Rails on Solaris instead of FreeBSD. I’m assuming that they were using FreeBSD at least in part because Joyent had purchased TextDrive which uses FreeBSD. So I’m just cruising through this PDF to see what sort of things they mention and then things got very strange on page 40 of the PDF.

Before I get into the details of page 40 go take a look at page 39. In this section they are talking about the limitations of Ethernet, specifically the gigabit variety. Page 39 provides the simple math, 1Gbps = 125MB/sec and 100Mbps = 12.5 MB/sec. So with that in mind take a look at page 40, where you find results of two netperf runs, one from FreeBSD to FreeBSD and another from Solaris to Solaris, both using gigabit ethernet. These results look to be quotes from Netperf benchmarks of FreeBSD versus Solaris. I won’t comment on the Solaris numbers as I don’t have much experience with Solaris and I don’t have any Solaris systems to run similar tests on. I do however have some FreeBSD systems to run these tests on and I have to say that I find their FreeBSD results to be a complete joke. I’m almost tempted to call them a lie. I’m specifically interested in the FreeBSD to FreeBSD over a switch results, which they show as:

pacific# /usr/local/netperf/netperf -fM -H private.comox.textdrive.com -tTCP_STREAM – -m1472
TCP STREAM TEST to private.comox.textdrive.com
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    MBytes/sec

65536  32768   1472    10.02       4.86

So out of the theoretical 125MB/sec (which you’ll never be able to reach, but that is another story) they were only able to get 4.86MB/sec. Only 3.888% of the absolute max? I was immediately suspect of these numbers, so installed netperf on two old FreeBSD boxes. The client box is a Dell Optiplex GX300 (P3-800 256MB) running FreeBSD 6.1-RC1 with a 100Mbit network card. The server side for my netperf test (neptune) is a Dell PowerEdge 4400 (P3 XEON-800 512MB) running FreeBSD 6.1-BETA1 with a 100Mbit network card. These are plain installs of FreeBSD on old hardware, nothing exciting. The network path between these two systems runs through two Netgear gigabit switches. I ran the same test they did more than a dozen times and always got results back at more than 11MB/sec. Here is the fastest result from my test runs:

> /usr/local/netperf/netperf -fM -H neptune -tTCP_STREAM – -m1472
TCP STREAM TEST to neptune
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    MBytes/sec

 65536  32768   1472    10.00      11.21

Here is the slowest result from my test runs:

> /usr/local/netperf/netperf -fM -H neptune -tTCP_STREAM – -m1472
TCP STREAM TEST to neptune
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    MBytes/sec

 65536  32768   1472    10.00      11.07

I don’t understand how they got 4.86MB/sec with gigabit and I got 11.07MB/sec with 100Mbit ethernet. The difference is just huge, 6.21MB/sec, well over twice their throughput. Unfortunately I don’t have spare FreeBSD systems with gigabit network cards to test against, but based on my 100Mbit tests I’m going to assume that they would be much faster than TextDrive’s 4.86MB/sec. What may be even more disturbing is that the folks at TextDrive and Joyent have been running their systems in this condition and didn’t see any large red flags go into the air indicating that they were doing something wrong.

A few minutes looking around on Google revealed that this came from Jason Hoffman on a post to TextDrive’s blog: Comparative netperf (network performance) of FreeBSD versus Solaris on identical hardware. I’d leave a comment for Jason on that post but there doesn’t appear to be any way to leave a comment. A post on the Joyent blog: Can you check my math? (Perhaps we’ll also call this, “What’s in a web server”?) posted on the same day, also by Jason, does seem to allow for comments so I’ll leave one pointing to this post.

I just can’t get over how completely bogus their numbers are. I can’t believe that they would then use them in a claim about how poorly FreeBSD does in netperf tests. Just amazing.

If you know what is going here please let me know. If I’ve done something wrong in my tests please point it out, because right now it looks like Jason Hoffman/TextDrive/Joyent need to go hire new sysadmins.