It’s been over a year since I first addressed the question of Box2D performance on various VMs and compilers, and I’ve been meaning to revisit it for some time. This time, I’m concerned primarily with the performance of virtual machines, as this is where we’ve seen most interesting developments of late.
I’ll start with raw numbers, since everyone seems to want those first. Afterwards, I’ll follow up with a bit more detail and some observations. All of these tests were performed on my MacBook Pro with a 2.5 GHz i7. There does not seem to be significant variance across runs, and I don’t believe the ratios will differ significantly on other modern Intel platforms (I’d like to get numbers for mobile ARM devices, which I suspect will behave differently, but that’s much more involved).
So without further ado, here are the raw numbers. The units are
followed by standard deviation and ratio compared to raw C.
|C (gcc 4.2.1)||2.62 +/- 0.24||1.0x|
|NaCl (x86-32)||3.39 +/- 0.25||1.3x|
|asm.js (Firefox Nightly)||5.12 +/- 0.59||1.9x|
|Java 1.7||6.02 +/- 0.67||2.3x|
|AS3 (Flash 11.6)||10.4 +/- 0.91||4.0x|
|Box2dWeb (Safari)||20.4 +/- 2.01||7.8x|
|Emscripten (Chrome)||23.6 +/- 4.87||9.0x|
|Box2dWeb (Chrome)||29.1 +/- 6.91||11x|
|Emscripten (Firefox)||31.6 +/- 2.48||12x|
|Box2dWeb (Firefox)||36.2 +/- 4.53||14x|
|Dart||40.7 +/- 9.21||15x|
And the following graph shows the performance of each VM, with their means and standard deviations scaled relative to native:
|C (gcc)||3.83 +/- 0.62||1.00x|
|Mandreel (Chrome)||48.33 +/- 4.82||12.61x|
|box2d-web (Safari)||68.85 +/- 5.20||17.97x|
|box2d-web (Chrome)||74.30 +/- 13.4||19.39x|
|box2d-web (Firefox8)||110.9 +/- 7.26||28.96x|
There are two ‘clusters’
AS3 is something of a special case, being a hybrid VM that will use type annotations to emit fewer runtime checks where possible, so it at least broadly makes sense that it would fall somewhere between the two clusters. But do note that I’m no expert on AS3 internals, and as I mention below this may be attributable at least partially to the use of native vector classes in the Box2D code.
AS3 is faster than you think
I was pleasantly surprised by the performance of the ActionScript VM. Coming in at 4x native performance, it’s actually closer to the C/JVM cluster than the dynamic VMs. The Box2D code is fully type-annotated, which is where I understand that AS3 shines. And it may be getting a bit of a boost from the platform’s native vector math libraries, but I’m not sure to what extent this is a factor.
[p]NaCl and asm.js
I included Dart numbers because I have them, but please take them with a grain of salt. The Box2D port is relatively new, and my understanding is that there’s still a lot more optimization work going on in the VM.
And as always, the code is available on Github if you want to run it for yourself or tweak the implementations. What I love about this approach to benchmarking is that it’s perfectly sound to tweak a given implementation to achieve better results. After all, the only goal is to show that a platform can compute the task at hand as fast as possible.
Note: I'm moving to G+ comments, but also want to preserve the old blog comments in read-only form. I just haven't gotten around to that last part yet, so they're temporarily unavailable.