Generating Facebook "Look Back" Style Dynamic Videos from HTML5 using PhantomJS and FFmpeg

This post explains how we built our own HTML5 animation to video converter, similar to what Facebook has done with its “Look Back” campaign, and more recently its “Say Thanks” feature.

Background

Using our TweetBeam Twitter wall is a great way to boost exposure for an event, and we wanted a cool way to give our customers a sense of the positive impact it can have. Inspired by Facebook and Vizify’s Year On Twitter, we figured a dynamically generated, customer-tailored video would make a great animated infographic. Plus, it’s a nice post-event gift to say thanks for choosing us.

It made sense for us to build the video as an HTML5 animation, since this is already what we specialize in. Additionally, building it in HTML5 makes it easy to customize it per event. (We built a simple editor to select the best photos and Tweets per event, then we apply this configuration to the animation template.)

To make the video more shareable and run smoothly across devices, we wanted to automatically convert the animations to a real video format (mp4).

Our approach in a nutshell

Simply put, we render the complete HTML5 video frame-by-frame using PhantomJS and then combine all frames into a video using FFmpeg. The pipeline consists of 3 main parts:

  1. An animation in the form of a HTML5 + CSS3 + Javascript website
  2. PhantomJS to render the animation and save the frames as PNG image files
  3. FFmpeg to combine the individual PNG frames into a video

Taking screenshots with PhantomJS

PhantomJS makes it easy to render a website server-side. Using the page.render method we can create a screenshot of the animation.

Of course, we can’t just call this method in a loop and expect all frames of a smooth video as output. For a smooth video, we need to capture a frame of the animation at least 30 times per second, each frame exactly 1/fps seconds after the other.

The code to achieve this looks like:

var frame = 0;
var target_fps = 30;
var frames = page.evaluate(function () {
    return getTotalDurationInSeconds();
}) * target_fps;

for(var frame = 0; frame < frames; frame++) {
    page.evaluate(function (time) {
        pauseAnimationAt(time);
    }, frame * (1 / target_fps));
    page.render(outDir+'/frame_' + padWithZeroes(frame, 5) + '.png', { format: "png" });
    frame++;
}

Pausing the HTML animation with Greensock

The trick in the above snippet is that we use page.evaluate to call two methods exposed by the website (our animation): getTotalDurationInSeconds and pauseAnimationAt. The first one should speak for itself. The second one pauses the HTML animation at a specific frame based on a given point in time, and requires a little more explanation.

Unfortunately, pausing a CSS3 animation or transition at a specific time (seeking) is not possible. This means we need to use an animation library with more fine-grained timeline control.

At TweetBeam, we use the excellent Greensock library for high performance javascript based animations. Its TimelineMax library supports all the seeking features we need.

Using Greensock and TimelineMax, we can fill in the methods we need to call from our PhantomJS script as follows:

function pauseAnimationAt(time) {
    timeline.pause(time); // timeline is a TimelineMax instance
}

function getTotalDurationInSeconds() {
    return timeline.totalDuration();
}

Stitching the frames into a video using FFmpeg

Now that our PhantomJS script can load a webpage and generate screenshots of all the frames in the animation, we can use FFmpeg to combine them into a video:

ffmpeg -y -r 30 -i frame_%05d.png -c:v libx264 -r 30 -pix_fmt yuv420p video.mp4

Note: I think it should also be possible to pipe PhantomJS directly to FFmpeg instead of writing the frames to disk first, which would be faster. Update 12/5: See Peter’s comment below on how to accomplish this

The result

We can now easily generate customized videos per customer. For example this one, which we created after the MongoDB World conference:

The feedback from our customers has been overwhelmingly positive. They spend a lot of time and effort organizing their events, so it’s great to have a nice video looking back on the highlights.

It should be noted that generating personalized videos at the scale that Facebook does comes with its own set of engineering challenges. Using the methods described above it takes about 20-30 minutes for us to generate a video, which is perfectly fine for the scale we’re operating at.

I hope this article inspired you to combine some great technologies! Like it? Questions? Drop me a note @YousefED or in the comments below.

Special thanks to @erikagain for reviewing this blog post