Response to VMware blog response

A few months ago, we posted our test results of some testing we had done in our test lab comparing native-to-virtual performance using VMware Server. Our report could have been more clear and the title could have been better, resulting in quite a stir on Slashdot that morning.

A VMware blogger responded to our article here: http://blogs.vmware.com/vmtn/2007/04/response_to_loa.html.

The author called the test an apples-to-oranges comparison, but is mistaken in a number of ways.

1. The author assumed that our report tested a VM configured with a single CPU. This was not the case - the VM was configured for dual CPUs in order to match the native configuration. We thought that was pretty obvious...but we should have been more clear.
   
2. The author implied that the host OS would be swapping because we had not configured the memory parameters correctly. This was not the case and we said so in the article (specifically we said the VM had 2G of RAM and the host had more). We didn't state exactly how much, but for the record it had 4G of RAM, so the host OS (Linux) was most definitely NOT swapping.
3. The author claimed that the performance drop was only 16% instead of 43% because we should have been using the hyper-threading-disabled results for the native machine. This is not in the spirit of the test, which was to determine the virtualization overhead on the same hardware. To purposely handicap the hardware during the native test would have drawn invalid conclusion because (1) this is the default configuration for this hardware and (2) anybody who compared with and without hyper-threading would be running the server with this native feature enabled.
4. Item number 7, "make sure the memory, disk and networking configurations for both physical and virtual environments match" was done in the original tests as well.
5. The suggestion to run multiple VMs instead of one (#8) seems a bit odd in this context. The point of the test was to compare the performance of a single web application in native mode and virtualized. Adding the overhead of a second VM (and a second guest OS) does not seem like something that would improve performance. Maybe it's worth testing, though.

We had initially planned to re-run the tests with the suggestions from VMware. But after more internal discussion, they do not seem to align well with the purpose of the report, which was to determine the performance cost of virtualization using VMware Server measured by the capacity of the system's ability to serve multiple simultaneous users while meeting specific performance requirements.