This post is split into 2 parts aiming to answer the question:

How do I use library “xyz”?

The first post defined the Problem and described the solutions people use today, this post will go over how shadow-cljs will handle the issue going forward.

The Goal

I want to automate using JS dependencies as much as possible while eliminating the issues I described in the first post. It should require as little configuration as possible but still allow the most common scenarios that people commonly use in JS projects (eg. using jQuery or react from a CDN). Interop should be seamless.

Using JS dependencies

CLJSJS popularized the use of :foreign-libs which introduced pseudo-namespaces that had no functionality beyond telling the compiler to include that certain lib in the build. You would (:require [cljsjs.react]) in your code but then access it using js/React. This is problematic for various reason and would require changing the code when importing the code via Closure.

Going forward the recommended solution is:

  • CLJS/Closure you :require by using a symbol (eg. (:require [clojure.string :as str]))
  • JS dependencies are required using a "string" which exactly matches the string you’d use in JS via require or import.

The JS version

import { createElement } as react from "react";
import { render } as rdom from "react-dom";
// or
const react = require("react");
...

then becomes

(ns your.app
  (:require [clojure.string :as str]
            ["react" :as react :refer (createElement)]
            ["react-dom" :as rdom :refer (render)]))

You could always use (def react (js/require "react")) in CLJS (which most node.js users do) but the problem here is that the compiler does not understand this information. It does not collect js/require calls and therefore is not aware of any JS dependencies it might need to deal with.

By moving it into the ns form it becomes static metadata (just like ES6 import) so tools can easily work with this data.

A small rant: Don’t use symbols for JS

Other CLJS contributors do not share my opinion that you should use strings for JS requires. They allow using (:require [react]) as a short-hand which is a magical symbol that only becomes available when the ./node_modules/react package exists (or a :foreign-lib provides it).

IMHO it is much cleaner to make it immediately obvious whether you are including a CLJS/Closure namespace by using a symbol or a JS dependency by using a "string". Not only does this make it easier for tools to do their work it also deals with all ambiguities you might encounter when using npm packages and trying to convert their name to a valid symbol (eg. - vs _, "@scoped/packages", "react-dom/server").

Just use a "string" to include JS.

shadow-cljs will support symbols for compatibility but really doesn’t like it.

Choosing a Provider

shadow-cljs introduces a new :js-provider build config option that controls which mechanism will be used to provide the JS support. There are currently 3 options to choose from:

  • :require will map directly to JS require calls, which works natively on node or when using other build tools to fill them in for us (ie. webpack, react-native packager, etc)
  • :shadow is the default for :browser builds and will package the npm code into a format we can consume in the browser.
  • :closure will attempt to import all JS dependencies using the Closure Compiler and should only be considered for :browser builds as well.

:shadow vs :closure

Both options attempt to import code from npm and rewrite it in a browser friendly way.

:closure is much more aggressive and will rewrite basically the entire file. It is the better option when it works since it will work reliably with :advanced optimizations. In theory no :externs would be required and the produced code will fully benefit from everything the Closure Compiler has to offer (dead-code-removal, code motion, etc). The downside is that it does not yet understand every form of npm package out there. A lot of work is being done to improve this situation but it might not yet work for some npm packages you want to use.

:shadow is the default until that changes. It basically only rewrites require calls since the browser does not natively understand those. It initially used browserify to bundle JS code for us but that had significant issues and did not work reliably enough. Going forward shadow-cljs uses its own JS bundler implementation with the goal of supporting every package on npm.

:shadow already has pretty good support for most npm packages I tested. Since there are so many different ways npm packages are built there might be some it does not yet support. Please let me know if you run into issues.

Using :closure is the goal and ideal path but it might take some time to get working properly. Definitely try and see if it works for your project.

Either way there are some issues we need to deal with going forward.

Externs

The downside of :shadow is that :externs are required since the JS code will not be run through :advanced compilation. It will minify the code for release builds which should still produce acceptable bundle sizes similar to what webpack and others would achieve but far from :closure.

Writing :externs is annoying but shadow-cljs check might be able to help.

We should be able to create a repository of :externs so the compiler can automatically apply the matching externs depending on the packages you use. Similar to the way CLJSJS automated most :externs.

Some :externs will still be required even for :closure since code on npm does not follow the strict coding style Closure expects. Some more “dynamic” styles are not properly detected when compiling and :externs sort of work around those cases (eg. react still needs externs with :closure).

BREAKING CHANGE: Removing :foreign-libs support

Popular libraries like reagent just include cljsjs.react but then use js/React to call the actual code. That is a problem since the compiler doesn’t (and shouldn’t) know that some foreign JS provided that global. reagent has a newer alpha version that removed the cljsjs.react require and now uses react (as a symbol, not a string. :() but pretty much all libraries need to be updated to do this.

Pre-bundled :foreign-libs (ie. CLJSJS) themselves are a problem due to the “scaling” problem mentioned in the first post. The systems simply do not mix well and frequently conflict with each other. One library might use cljsjs.react while another uses the newer "react”. This would lead to two versions of React being included in your page.

shadow-cljs will not support :foreign-libs going forward (which includes CLJSJS).

This might make it impossible to build your project with shadow-cljs without changing some code.

Since most CLJSJS packages should have npm packages as well you can work around a few issues by manually creating the cljsjs namespaces. Take react for example, usually you’d use cljsjs.react and then the global js/React. To make that available you can create that namespace and manually add the global.

(ns cljsjs.react
  (:require ["react" :as react]))

(js/goog.exportSymbol "React" react)

This should only be done if you are using a library that uses a cljsjs namespace that has not been updated. Your own code should be updated to use string requires.

I’m happy to provide an interim solution to make the transition easier if someone has a good idea how to do so.

:npm-deps

CLJS core supports the :npm-deps compiler option to control how things are compiled. In addition it also attempts to manage JS dependencies by trying to install packages for you.

I do not think that :npm-deps is adequate to cover this topic. shadow-cljs does not support this now and won’t manage JS dependencies for you. You must manually install packages using npm or yarn and manage them using the usual package.json.

Work to be done

All work was focused on npm packages and using local .js files is not as convenient as I’d like it to be. It will be properly supported at some point I just didn’t get around to it yet.

Conclusion

I decided to do an entirely different implementation than what is currently available in CLJS core. I do think that it is a tooling concern anyways and necessary to be able to deal with all the complex build scenarios users might have.

Due to the problems described in the first post I never used many JS dependencies in my projects. Going forward I feel much more confident that I can safely use them and not worry about all the problems (well, except :externs I guess).

I do think that using JS dependencies is easier and more reliable this way. There are some adjustments you might need to make to your code migrating off :foreign-libs but I do think it is worth it in the end. You gain access to the whole npm ecosystem and do not rely on third-party re-packaged CLJSJS packages.

Feedback wanted!

I need testers to make it as reliable as possible. Just yarn add shadow-cljs and get started.

A simple example can be found here.

Please report a Github issue in case something doesn’t work as expected. You can find me on reddit, Twitter or @thheller in the Clojurians Slack if you have questions.