Aerotwist - Bad Benchmark, Right Result

I bounced over to Twitter all excited, because from a developer point of view this is clearly way freakin’ awesomer than the string gymnastics.

The benchmark

Kris Gustaveson responded to my initial tweet with a query:

Sure enough, when I visited the benchmark in question I saw exactly the same thing as Kris. Roughly speaking it appeared 70% slower to use WebKitCSSMatrix than to create strings. On the other side I had the comment in the bug, which definitely indicated that using a matrix should be faster than passing in a string. The theory being that internally we can detect the use of the matrix and can bypass the CSS string parser.

Scrutiny

It shouldn’t have taken me an hour to figure it out why the jsPerf benchmark was squiffy, but it did. The clue lay in the benchmark’s preparation code. See if you can figure it out faster than I did.

In case you couldn’t see it, here’s what I (eventually) realised: the element doesn’t exist in the render tree, so any style changes will be ignored. This test is therefore akin to checking whether it’s faster to set a property of an object with a string or another object!

When the element has been added to the DOM and exists in the render tree, we can actually test how CSSMatrix compares to using strings because we can measure the time spent in recalculating the styles. In theory using the matrix should be way faster (no CSS string parsing, yo.)

In the end I created my own quick tests (one using strings, one using CSSMatrix) that animated 1,000 visible DOM elements, all in the hope of clearing matters up.

But all was not as it seemed… (Ooh, feel that tension.)

The right result

What is already a budget buster for desktop would be an fps killer on mobile.

I’m on a decent Macbook Air and I was seeing ~20ms of recalc style for 1,000 elements in both the string and CSSMatrix versions. On mobile that would be in the region of 6-8x, so basically what is already a budget buster for desktop would be an fps killer on mobile.

The chart below shows the cost of Recalculate Style and time spent in JavaScript vs the number of DOM elements when you use strings:

But now see what it looks like for CSSMatrix:

There are two things to notice here:

  1. The JavaScript cost of using CSSMatrix is really high in Chrome. I did a little digging here and it’s definitely where you go and assign the matrix to the element’s transform.
  2. Probably related, but when I used the Chrome DevTools flame chart I noticed that a toString() method was being called inside my requestAnimationFrame callback. That wasn’t from me, so it appears internally Chrome calls toString on the matrix and passes it over to the CSS parser, which is exactly what we were hoping to avoid.

Therefore because we’re essentially bouncing down to a string here the Recalculate Style cost is virtually identical to setting a string directly. But for some reason (must confess here, I don’t fully understand), the JavaScript cost of CSSMatrix is higher than using a string.

Conclusion

For me this was an interesting journey. Having to rationalize seemingly contrary claims was fun, and actually getting to the bottom of it was satisfying, even if the conclusion was the same: use strings not CSSMatrix. I’ve filed a bug with all my findings (where Eric Seidel, one of our engineers has confirmed the toString() theory… can I call that my String Theory? No? Too confusing. Gotcha.) and my hope is that our engineers will be able to allow CSSMatrix to be faster than it is today.

But here’s a super critical point: if you benchmark visual APIs, make certain they operate on visible DOM elements that reside in the render tree.