;

JavaScript needs the compile step (on install)

Update: read the second post Precompiled JavaScript after this one.

We all pretend that JavaScript is unique because it does not require a build or compile step on the client. Yet, this is patently untrue. At the minimum, one has to admit the native bindings compilation step during NPM module install. At the maximum, one has to admit that every library requires building to the least common denominator, which is ES5. Despite having the evolving standards and wide support for most of them (see this table), the module's author has NO idea what JavaScript features a particular client installation has. Does a particular version of client's NodeJS have Promises or not? Does it support the arrow functions and the spread operator? Without 100% NodeJS v5 adoption (and seems the world is still stuck at Node 0.10) we have to transpile our concise, elegant and performant ES6/ES7 code to ES5.

Even worse, NodeJS 5 does NOT support all the features of ES6 standard. Even when using the --harmony flag (as I recommend doing in Use some ES6 in CLI apps), some of the best parts are unavailable: module import and export, default parameters, etc.

The same situation (or worse) is when the JavaScript runs in the browser. The variety of engines is good for the industry, but bad for the module's author.

Is there something we can do to write ES6/7 today and be able to run the original code without mangling it to death using transpilers?

An example solution

Let us start with an example. Imagine I have the following source code

1
2
3
4
package.json
    src/
        main.js
        calc.js

We are using ES6 syntax in both source files: import and export, arrow function, even template literals

calc.js
1
2
export const add = (a, b) => a + b
export const sub = (a, b) => a - b
main.js
1
2
3
4
5
import { add } from './calc'
const a = 10
const b = 2
const sum = add(a, b)
console.log(`${a} + ${b} = ${sum}`)

Step 1 - Roll it up

Let us transform all our individual ES6 modules into a single bundle. We can use an excellent tool rollup. The beauty of this tool (and the ES6 standard that allows it to work) is that ES6 import / export statements can be traced statically, unlike the CommonJS code that uses the require calls (see the presentation The Importance of import and export by Ben Newman).

 -- . --  .

We just created a combined bundle inside the dist folder

dist/bundle.js
1
2
3
4
5
const add = (a, b) => a + b                                        
const a = 10
const b = 2
const sum = add(a, b)
console.log(`${a} + ${b} = ${sum}`)

Notice the "tree-shaking" benefit in Rollup and ES6 modules - the tool has determined that the exported sub function inside calc.js is never used, and is dropped from the bundle.

Step 2 - Determine ES6 features used

We have eliminated the import and export calls because we just bundled all our code into a single file. What about the rest of the ES6 features? We know that we have used arrow functions, const keyword and template literals. Keeping manually the list of the features is hard, especially as parts of the code are "shaken" off the tree during the bundling.

Fortunately, there is a tool that can inspect a source file and give us a list of the features: es-feature-tests. We simple run the tool after rolling the bundle and save the list of features.

  --- --
 --. -- > -.

In our case this has generated the following JSON file

dist/es6-features.json
1
["letConst","templateString","arrow"]

When you publish your module to NPM you can only publish these two files: dist/bundle.js and dist/es6-features.json! See Smaller published NPM modules how to include only the certain files in your package.

Step 3 - Compiling bundle (on install)

Steps 1 (bundling) and 2 (determining the ES6 features used) run on the build machine. Now we need to run a step on the client's machine. This step takes the ES6 bundle and will produce a compiled bundle the user will run. Our goal is to inspect the client's environment, determine the supported features (and the missing ones), and then transpile the bundle but only the missing features. Thus if a feature like function arrows is present, we will keep the original code. If a feature, like the template literals, is missing, we will use a specific Babel plugin to replace each instance with its ES5 equivalent.

Here is the outline of the code, you can find the full source in the repo bahmutov/compiled

transpile.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

var es6features = require('./dist/es6-features')
var es6support = require('es-feature-tests')
es6support('all', function (es6present) {
    transpile(es6present, es6features, 'dist/bundle.js', 'dist/compiled.js')
});

function transpile (supportedFeatures, neededFeatures, inputFilename, outputFilename) {
  var babelMapping = {
    letConst: 'transform-es2015-block-scoping',
    templateString: 'transform-es2015-template-literals',
    arrow: 'transform-es2015-arrow-functions',
    ...
  }
  var plugins = []
  neededFeatures.forEach(function (feature) {
    if (!supportedFeatures[feature]) {
      plugins = plugins.concat(babelMapping[feature]) 
    }
  })
  console.log('need plugins', plugins)
  
  var babel = require('babel-core')
  var options = { plugins: plugins }
  babel.transformFile(filename, options, function (err, result) {
    if (err) { throw err }
    require('fs').writeFileSync(outFilename, result.code, 'utf-8')
    console.log('saved file', outFilename)
  })
}

We need to run this script when the client runs npm install <my module>. For demo purposes I am printing the node version before running the command.

1
2
3
4
5
{
    "scripts": {
        "postinstall": "echo \"Running transpile for `node --version`\" && node transpile.js"
    }
}

Whenever we run npm install, the script kicks off and the Babel transpiler creates a bundle that is exactly supported by the user's NodeJS version.

Step 4 - the main file

There is a lot of discussions around the JavaScript community how to specify two main files per bundle. One for ES5 loaders (main) and another one for ES6 loaders (main:es6 or main:next?), the discussion is still open.

For our bundle, I set the main script to point at the compiled bundle, which is built on install

package.json
1
2
3
{
    "main": "dist/compiled.js"
}

Demo

I am using the default code inside bahmutov/compiled at tag 1.1.0.

Node 0.12 and below

First, let us install a module using Node 0.12

1
2
3
4
5
6
7
8
9
10
11
12
13
$ nvm use 0.12
Now using node v0.12.9 (npm v2.14.9)
$ npm i

> compiled@ postinstall /Users/kensho/git/compiled
> echo "Running transpile for `node --version`" && node transpile.js

Running transpile for v0.12.9
need es6 features [ 'letConst', 'templateString', 'arrow' ]
need plugins [ 'transform-es2015-block-scoping',
  'transform-es2015-template-literals',
  'transform-es2015-arrow-functions' ]
saved file ./dist/compiled.js

The compiled file has all ES6 features transpiled (because none of the needed ES6 features were found)

dist/compiled.js
1
2
3
4
5
6
7
var add = function (a, b) {
    return a + b;
}; 
var a = 10;
var b = 2;
var sum = add(a, b);
console.log(a + " + " + b + " = " + sum);

Because the main property in package.json points at this dist/compiled.js bundle, we can run it simply by executing node . command

Our code, written in ES6 and distributed as an ES6 bundle works fine on Node 0.12. Even better: the same code can be installed and ran on Node 0.10 and even 0.8!

Node 4 and above

Second, let us try a modern NodeJS engine, like v4.

1
2
3
4
5
6
7
8
9
10
$ nvm use 4
Now using node v4.2.2 (npm v3.5.0)
$ npm i
> compiled@ postinstall /Users/kensho/git/compiled
> echo "Running transpile for `node --version`" && node transpile.js

Running transpile for v4.2.2
need es6 features [ 'letConst', 'templateString', 'arrow' ]
need plugins []
saved file ./dist/compiled.js

Notice that the compilation step did NOT need to transpile anything - all the ES6 features we used in our code are supported by the environment. The code is the original ES6 bundle

dist/compiled.js
1
2
3
4
5
const add = (a, b) => a + b;
const a = 10;
const b = 2;
const sum = add(a, b);
console.log(`${ a } + ${ b } = ${ sum }`);

Still runs as expected

note I have moved this example to its own repo for clarity. See bahmutov/compiled-example - this repo even includes using 3rd party library (a method from Lodash).

Conclusion

I plan to build a (yet another) CLI build tool based on repo bahmutov/compiled. It will have just two commands - build and compile. The build command will run on the author's machine (or better on CI server), and the compile command will run on the client's machine during postinstall step.

This way I will finally be able to use ES6 when writing code, avoid unnecessary transpiling if I use Node 4/5, yet be able to support older Node versions.

There are lots of open questions. I encourage everyone interested to open an issue and start the discussion.

Frequently Asked Questions

Q: Do you have examples?

A: Simple example in bahmutov/compiled-example. bahmutov/left-behind is a larger example with main and bin bundles.

Q: Is this compatible with every Nodejs / NPM version?

A: No. Babel does not run on Node < 0.11, and NPM 2 messes up installing babel plugin dependencies (for no good reason, as far as I can tell). Thus I recommend Node >= 0.11 and NPM 3.

Q: Is this for bundling all your code, including the 3rd party dependencies?

A: So far it seems so, but I don't know how this will affect CLI applications vs libraries. Rollup is smart enough to determine which modules are 3rd party, and includes these using the standard require calls. Thus only your own code will be in the bundle.

Q: Does this increase the size of the module because the Babel transformation is included with the production dependencies?

A: Yes, unfortunately we need to include Babel and plugins in order to transpile on the user's machine.

Q: What about ES7 and beyong features? What about JSX, etc?

A: I am using "ES6 Feature Tests" to determine if the environment supports necessary features. Same tool is also used to determine the necessary features in my application. We can always find or write more feature tests to add additional support.

Q: How do I go back to the original source code?

A: Both Rollup and Babel support source maps. I expect that we can trace any location inside the dist/compiled.js back to the dist/bundle.js back to the original source line.

I have implemented targeted transpiling for browsers supporting ServiceWorker in babel-service.