JavaScript Omni-packages
How to package JavaScript libraries for a smooth transition from legacy to standard modules
by Joe Honton
JavaScript is poised for a revolution in packaging. New ESNext modules are now supported across all major browsers and server-side Node.js software. This is good news.
But the legacy of CommonJS modules will be with us for a long time. Every new JavaScript project wanting to take advantage of standard import/export syntax, will need to consider whether all of their dependencies can be met with the new module loader. And at the same time, every package developer will need to consider whether they're going to support legacy consumers (CommonJS), or standards-based consumers (ESNext), or both.
This is not a small problem, and it won't go away any time soon. Python developers went through a similar transition that took 12 years to complete. Python3 was released in 2008, but the incompatible Python2 system wasn't finally frozen until April 2020.
The best strategy forward is to design and publish omni-packages that allow developers and consumers to gradually transition from old to new. Let's examine how that can be done.
Modular loading basics
Most JavaScript developers have mastered the syntax of import/export, so we don't need to go over that again. But not everyone has the same level of familiarity with how JavaScript finds and loads modules. So let's begin by establishing some common ground.
The job of the loader is essentially fourfold:
- Identify module dependencies.
- Determine where modules can be found.
- Maintain a list of modules that have been downloaded.
- Catch cyclic dependencies.
These are not trivial tasks, and thankfully we can all rely on the work that's already been done for us.
With ESNext, the module loader is part of the language spec itself, so there is nothing extra to parse and execute in order to make use of import/export syntax.
With CommonJS, the module loader is just another piece of JavaScript. So in order to use require/exports syntax anywhere in our code, our script must first instruct the JavaScript interpreter to parse and load the functions that handle that syntax. On the frontend this is the job of build tools and bundlers like Browserify, Rollup, and Webpack. On the backend, this is seamlessly handled by the Node.js system without any extra effort on our part.
In practice, it's common to use the newer ESNext syntax to target the older CommonJS loader. To accomplish this we use a transpiler like Babel, which rewrites import statements as require statements, and rewrites export statements as module.exports statements.
CommonJS loaders carry out their work synchronously and all at once. This means that everything is parsed and ready to use immediately at the start of execution. On the other hand, the new ESNext loader carries out its work asynchronously and on-demand. That means that modules are pulled over the network only if and when they are needed. This optimization is further enhanced when proper HTTP caching is used.
Four execution pathways
There are four paths that our code can follow: two for frontend and backend, and two for ESNext and CommonJS.
1. Frontend using CommonJS
For this we use a bundler to shrinkwrap the module loader and our own code into a single file. Also, if we've written our code using the newer syntax, we need to instruct our bundler to invoke the transpiler as well. In order to use it in the browser we add a <script> tag to our HTML like this:
<script src=index.js type=text/javascript nomodule></script>
For browser scripts loaded with the nomodule attribute, code should be written with require statements. If we do write them with import statements, they must be transpiled by a builder or bundler before being sent to the browser.
2. Frontend using ESNext
For this we simply identify our source code file, and add a <script> tag with type=module.
<script src=index.js type=module></script>
Browser scripts loaded with type=module can safely rely on JavaScript's native loader and make use of standard import statements without fuss. This is the target we should all be striving for.
3. Backend using CommonJS
For this we create a package.json file having a "type" property with a value of "commonjs" and Node.js will use its built-in CommonJS loader for all require statements.
// package.json
{
"type": "commonjs"
}
Node.js locates and reads the nearest package.json file in order to discover how to load submodules. While we normally think of package.json as a file to specify publishing rules, it now serves double-duty as the runtime specifier for "type": "commonjs" or "type": "module".
4. Backend using ESNext
For this we create a package.json file having a "type" property with a value of "module" and Node.js will rely on JavaScript's native loader for all import statements.
// package.json
{
"type": "module"
}
And good news for anyone experimenting with Deno ȁ they will have a free ride because Deno natively supports ESNext — there is no need for any special package.json file.
Those who have been following this topic will recall early discussions which introduced the idea of using the filename extension .cjs for CommonJS files and .mjs for ESNext files. This was only ever supported by Node.js and never formally adopted by browser vendors. Today's best practice is to stick with the .js extension for everything. This is important if we want to create an omni-package with a single source of truth.
Omni-packages
Let's define our goals for omni-packages. They should meet these requirements:
- Packages should be usable in all four executable pathways.
- There should be one set of files acting as the "source of truth".
- Package files should be debugged and tested with unmangled, fully commented sources.
- Package files should be distributed and installed by consumers with minimized code.
- Packages should be published and versioned under a single public name.
We can accomplish these goals with a modified organizational tree similar to the classic src, dbg, dist pattern that many projects already use. But instead of three top-level directories, the omni-package layout has four: esmodule, esm, commonjs, and cjs. Each of these has a complete set of the original files, using the same subdirectory layout, and the same filenames.
The esmodule directory is the original source of truth. The developer makes changes within this tree only. The work of debugging and testing the package using either ESNext execution pathway is done directly using the files within this tree. It is kept in the repo, but not normally distributed to package consumers.
The esm directory is the minified version of esmodule. It is distributed with the package, and can be used from a browser with an HTML statement like this:
<script src=/node_modules/my-package/esm/index.js type=module></script>
Or within a Node.js project with an import statement like this:
import MyPackage from 'my-package/esm/index.js'
The commonjs directory is the Babel transpiled version of esmodule. The work of debugging and testing the package using either CommonJS execution pathway is done using the transpiled files within this tree. It is not normally distributed to package consumers.
The cjs directory is the minified version of commonjs. It is distributed with the package, and can be used from a browser with an HTML statement like this:
<script src=/node_modules/my-package/cjs/index.js nomodule></script>
Or within a Node.js project with an import statement like this:
import MyPackage from 'my-package/cjs/index.js'
Just as a reminder, in order for Node.js to properly load files from esm and cjs those two top level directories must have a package.json file with either "type": "module" or "type": "commonjs" as described above. (On the frontend, browsers will completely ignore these package.json files.)
Publication and distribution
For publication and distribution purposes the four top-level directories are placed within an outer project directory, together with the usual license and readme files, plus a project-level package.json file that looks something like this (with no "type" property):
{
"name": "my-package",
"repository": {
"type": "git",
"url": "https://github.com/username/my-package.git"
},
"files": [
"esm",
"cjs"
]
}
Omni-packages are the best way through JavaScript's transition period. They allow package developers and package consumers to migrate from legacy modules to new ECMAScript-standard modules at their own pace and without a tumultuous "stop-the-world" interruption.
