Why I Prefer Makefiles Over package.json Scripts

On any moderately-sized Node.js project you’ve likely already outgrown the package.json “scripts” section. But because the growth was gradual, with no single acute pain point, you might not have noticed. There’s a better way.

What Are NPM Scripts?

In most Node.js projects you’ll find a scripts section in the package.json file, containing convenient shortcuts like “build” and “test”:


"scripts":{
    "build":"tsc",
    "test":"jest -w 1"
}

These can be run with e.g. npm run build or yarn test.

In addition, there are special meanings behind scripts with certain names, like “install” or “prepublish” (npm,yarn).

On a new project it usually starts out innocent enough, with a small collection of simple, self-explanatory commands. But then…

Common Patterns

One pattern that I see come up often in NPM scripts is multiple variations on a single script:


    "go": "node go.js --do-some-stuff",
    "go:debug": "echo debugging..; DEBUG=1 yarn go",
    "go:there": "yarn go --to=\"over there\""

This is reasonably readable, but check out the same thing in Make:


GO_DO_STUFF=node go.js --do-some-stuff

go:
    ${GO_DO_STUFF}

go-debug:
    echo debugging..
    DEBUG=1 ${GO_DO_STUFF}

go-there:
    # watch out for the space in that argument:
    ${GO_DO_STUFF} --to="over there"

Variables. Multiple lines. No more escaped quotation marks. Comments.

Another pattern that comes up — and this is a more sinister one — is the chain of dependencies:


  "shared-prereq": "echo shared prereq!",
  "another-prereq": "echo another prereq!",
  "task-one": "yarn shared-prereq && echo doing task one..",
  "task-two": "yarn shared-prereq && yarn another-prereq ; echo doing task two.."

What if your build tool’s syntax had a way to express dependencies?


shared-prereq:
    @echo shared prereq!

another-prereq:
    @echo another prereq!

task-one: shared-prereq
    @echo doing task one..

task-two: shared-prereq another-prereq
    @echo doing task two..

Amazing. But wait, it gets better…

Avoid Extra Work

In the above example we have a series of always-run steps, asking only of Make what we asked of our NPM scripts. But Make can do better.

If your commands are reading and writing a predictable set of files, then Make can track them and avoid redundant work.

This is a perfect fit for code generators (plug for graphql-code-generator, openapi-typescript, and json-schema-to-typescript).

Here’s an example of using Make to describe the operations of a code generator:


JSON_SCHEMAS = $(shell find schemas -type f -name '*.schema.json')
JSON_SCHEMA_DST = $(JSON_SCHEMAS:%.schema.json=%.schema.gen.ts)

JSON2TS = yarn run json2ts

schemas/%.schema.gen.ts: schemas/%.schema.json
    $(JSON2TS) -i $< -o $@

CODEGEN_DST = ${JSON_SCHEMA_DST}

codegen: ${CODEGEN_DST}

build: ${CODEGEN_DST}
    @echo "I depend on those generated files!"

clean:
    find schemas -type f -name "*.gen.*" -delete

It’s a little arcane, so here’s what it does:

  • The first time you run `make build`, it will find all the `.schema.json` files, generate a `.gen.ts` for each one, and then continue building the app.
  • The second time you run `make build`, Make will see that the generated files are up-to-date, skip the generator, and build your app.
  • If you edit one of the schema files, Make will notice that just that file changed, run the generator for it, then build your app.

Can your build tool do that?

Discoverability / Organization

On a large project, you can sprinkle multiple Makefiles in different directories, where they’ll offer discoverable shortcuts related to that area of the application.

It’s even more discoverable if your shell has smart tab completion: for example, on my current project, if you enter the aws/ directory and type make, you’ll see a list that includes things like docker-login, deploy-dev and destroy-sandbox.

Bonus: It’s Faster

This speaks for itself:


bash-3.2$ time yarn task-two
yarn run v1.22.5
$ yarn shared-prereq && yarn another-prereq ; echo doing task two..
$ echo shared prereq!
shared prereq!
$ echo another prereq!
another prereq!
doing task two..
✨  Done in 0.63s.

real    0m0.814s
user    0m0.508s
sys 0m0.138s

bash-3.2$ time make task-two
shared prereq!
another prereq!
doing task two..

real    0m0.021s
user    0m0.008s
sys 0m0.010s

Admittedly the tool overhead won’t matter if your command takes any meaningful amount of time, but you’ll feel the difference if you happen to have any scripts that execute instantly.

The Straw That Broke the Camel’s Back

JSON was meant for serializing objects. It’s a lousy config format, and it’s even worse at expressing a complicated build.

As you pile more complexity into your NPM scripts, it never feels like you’re the one placing the straw that breaks the camel’s back. But please, take a step back, look at your poor camel, and consider using another tool.

I don’t even care if it’s Make. Use something else. Maybe your language ecosystem has a sweet build tool that can do all of the above. I’d love to be proven wrong, but as far as I know in Node.js (especially with Typescript), there isn’t.

If you have more than a handful of extremely simple npm scripts, try Makefiles!

Further reading:

Conversation
  • Nice post. As a long-time Make user I too grew frustrated with the limitations of package.json and so I built

    https://www.npmjs.com/package/bajel

    This has Make semantics (roughly) but is a bit simplified and fits in better into the JavaScript build ecosystem.

    I’d be interested in your thoughts.

    • Paul / Appurist says:

      Bajel looks great. I also looked at “just” below, and both are steps forward in keeping with the article above, but just looks great in general but a overkill for most cases. I love the way you kept Bajel so easy to understand and use. I’m definitely going to give it a spin.

  • Raven Flores says:

    Make scripts are not cross-platform.

    • John Ruble John Ruble says:

      Indeed, they’re not, or at least not trivially. I might not use them if I had to maintain Windows compatibility. (I assume you’re talking about Windows?)

      Are you working with the Node.js ecosystem on Windows? Do you have a favorite cross-platform way to do the kinds of things in the post’s examples?

      On the other hand, I’ve had success in the past with installing make on Windows via Chocolatey:

      https://chocolatey.org/packages/make

      And I’d be curious to try recent versions of WSL:

      https://docs.microsoft.com/en-us/windows/wsl/

      • I went to great lengths to write cross-platform build scripts for my book: https://expertlysimple.io/angular-for-enterprise-2nd-edition/. This project has good examples https://github.com/duluca/lemon-mart (exact opposite direction of this article, since I love npm scripts) and also scripts to setup development environments: https://github.com/duluca/web-dev-environment-setup.

        WSL2 is great, but slow. This is even documented in the docs, however Powershell 7 (which itself is cross-platform) with the new Windows Terminal is great. A ton of out-of-the box Unix compatibility.

        • I think it’s worth noting that “WSL2 is great, but slow” refers to accessing Windows files from inside WSL.

          Been using WSL 1 and 2 since 2018-2019 and everything checked out.

      • John says:

        * Users can install GNU on Windows (GOW) (https://github.com/bmatzelle/gow), which takes care of the missing Unix toolchain on Windows.
        * If makefile portability is an issue, make it clear in the project’s documentation that _GNU Make_ is required. Either that, or stick to using simple Make syntax that’s supported both by GNU and BSD Make flavours.

        • KJ says:

          There’s also Cygwin and GnuWin32, which have been around for decades (and let me have awk and m4 for when I need them).

      • no says:

        Everything in your post can be done with a rudimentary knowledge of node

        Attempting to force people to install non-portable build tools from other languages is foolish.

        No, the portability problem isn’t about windows. Make isn’t portable between unices.

        This is just terrible advice

  • Les Orchard says:

    Yeah, I ended up leaning on scripts in package.json because make is not reliably available everywhere my node package needed to run – i.e. Windows, minimal VM containers, etc. Small and venerable though it might be, make is one more dependency

  • Tom Spencer says:

    On the topic of cross platform support, I’ve had luck using a make-like tool called just:

    https://github.com/casey/just

    It’s syntax is inspired by make, without all of the subtle but significant platform-specific differences that the various make implementations incur. There are prebuilt binaries and support for installing via the popular package managers (homebrew, etc). If you are considering make I would suggest looking at just and seeing if it suits your needs.

    • John Ruble John Ruble says:

      This looks fantastic! It might be just the thing to break me of my make habit. Also bonus points for Rust. Thanks Tom!

      • Tom Spencer says:

        I’ve got nothing but praise for it, I did a lot of research in this space a couple of years back and it was the front-runner. Hopefully it should be easy to adopt, the syntax will be very familiar!

  • Ivan says:

    Lets take a hypothetical developer. They have NPM. Which means they have Node. Which means they know JavaScript, which is a scripting language. One of the most used scripting languages in the world with libraries available for literally anything. One of the most cross-platform languages in the world. The language which, having Node, our dev can debug with a push of the button. And instead of having
    “scripts”: {
    “build”: “node build.js”
    },
    you propose our dev to learn obscure, non-debuggable, non-cross-platform, sudo-script language. Why???

    • Jose Rodriguez says:

      Simple answer: a) superior concept and technology b) your arguments against make are all false: it is not obscure (simple syntax, known by everyone), it is easy to debug, the non-cross-platform comes from Windows world which has had make for say 20 years (ever heard of nmake?) and it does not have a script language.

      Now that Javascript developer wants to stick to the worst language ever invented, let him or her. Got a bright future ahead.

      • gal anonim says:

        Oh my. Superiority complex much? Hating much?

        • KJ says:

          At least his comment had content, not just a juvenile ad homen.

          One of my employees spent a day or so writing some Python (another “One of the most used scripting languages in the world with libraries available for literally anything”, but not the mess that JS is) to automate one of our processes. I duplicated it with a makefile and a roughly 20-line awk script. 1/10th or better the code to maintain, took an hour to write and test. And I didn’t have to go find libraries to make it work. No dependencies. My developer thought that was clever then sat down and learned make and awk. Took her about a day to figure out the core.

          Pick the right tool for the job, even if you have to learn a new one. If all you have is a hammer, everything looks like a nail.

          If I really had a ‘superiority complex’ I would add that anyone who thinks ‘make’ is ‘obscure’, but Javascript isn’t, hasn’t really seen much outside the JS bubble and probably shouldn’t be making value judgements about the utility of tools some of us have been using successful for more than 40 years.

  • Doug says:

    Thanks for this article. I’ve recently been looking at make for use as a general task runner. Although it works it may not be the best tool for this purpose.

    For another cross platform alternative check out:

    https://github.com/go-task/task
    https://taskfile.dev/

  • Aleks says:

    I don’t normally post comments, but why use makefile when grunt/gulp task runners exists for this precise reason? Everything is well organized. Why reinvent the wheel?

    • Dr.bob says:

      You’re joking, right? Reinventing the wheel? Make and makefiles have existed for decades, so who actually reinvented the wheel?

  • Bill says:

    I agree about the issue with the out of the box behaviour of node scripts but using make seem like a step in the wrong direction for me.

    I know some people swear by them, but the poor out of the box error handing/reporting of bash scripting make them pretty awkward to debug and maintain in my experience and you typically end up writing something that doesn’t look very clean to get robust failure handling (I’m not exactly versed in bash best practice to be fair). Also, I might be completely wrong about this but it feels like it doesn’t make as much sense to assume things about the system os and what is installed on it as when you could have a system with a defined package system that is guaranteed to be there as a prerequisite of the script running.

    It just feels to me like to get something usable that fails in a sensible way you’d end up building a lot of stuff js and a few libs would gives you pretty much out of the box and I wonder if a better compromise be some thing with a nicer interface within the node ecosystem (maybe Gulp?). With maybe a package json script that just prints out the commands to access it for developers that are unfamiliar.

  • Dave says:

    I love this approach. I think another very powerful benefit is that if you work across multiple codebases and languages, you can consistently have a makefile as an ‘index’ of the key commands. Even if the makefile is just a simple wrapper around things like npm, maven, gradle, the .NET CLI, etc, you have a discoverable set of recipes which everyone can easily see. You clone the project and use the makefile to build and test – makes it easier for people who might be less familiar with the project specific toolchain. I’ve done the same thing many times and wrote a bit about this approach for Docker images (slightly old post, but very in line with what you’ve said! https://dwmkerr.com/simple-continuous-integration-for-docker-images/)

    Fantastic read and thanks for sharing!

  • vitaliy potapov says:

    What about having /scripts directory and call everything via package.json?
    E.g.:
    “scripts”: {
    “build”: “./scripts/build.sh”
    }

  • James Cole says:

    I have very similar use-cases, but I found makefiles a bit limiting. I wrote lk[1] to make this sort of thing easier, and you can just write plain bash functions instead of makefiles. As someone with similar needs I’d love to hear what you think of it.

    [1] https://github.com/jamescoleuk/lk

  • I couldn’t agree more with John on this!

    In fact, I too have started with `make`, and as I used more and more advanced features, I’ve ended up writing `make` files that generate other `make` files that are included by the main one… :)

    And, when resorting to generation, this is the issue with `make` (or others), you get away from one “quoting hell” (in JSON) just to end up in another “quoting hell” (the `make` + `bash` combination)…

    As such, like many others have voiced previously, I too have ended writing my own automation tool:

    However, I didn’t settle for just `bash`… I wanted to be able to combine Python, `jq`, `bash`, even Go if necessary, all in the same script file… (**Without any quotation hell.**)

    And, the irony, in one of my projects I’ve ended up generating `ninja` scripts (alternative to `make`), that in turn call other tasks from my main `z-run` script. (For example .)

  • Jack says:

    Sorry, but I don’t see the point of any of this. I gave up using make files when I stopped using MPW on the Mac.
    Are there no decent IDEs for what you do, that know the dependencies from using / include directives, know what has changed since the last build, and can therefore rebuild the required objects in the correct order?
    I guess some cavemen actually liked making fire through the laborious act of rubbing sticks together, but give me a lighter any day!

  • Thanks for your write up! And for the references!

    https://github.com/ysoftwareab/yplatform that this a notch higher. I’ll make sure to add a link to your post.

    PS

    for those that don’t know, npm’s “scripts” functionality as we know it today was not “designed”, but it was simply a byproduct.

    First introduced as an internal functionality for lifecycle support https://github.com/npm/npm/commit/c8e17cef720875c1f7cea1b49b7bc9f1325ccff5 on 1 Mar 2010

    and later extend to run arbitrary targets https://github.com/npm/npm/commit/9b8c0cf87a9d4ef560e17e060f4ddc03b2ff1a1c on 13 Dec 2010

    This is the entire design backstory https://github.com/npm/npm/issues/298 .

    Needless to say that GNU Make’s equivalent (or any other build system) is not to be found in npm’s scripts.

  • Adam says:

    This is making the rounds on Pinboard, a neo-Luddite circle I occasionally punish myself by following. Although it appears that you spend a lot of time writing JS/TS, the people reposting this usually do it in the spirit of, “Here’s another dumb tool (npm) that dumb JS developers use when there’s a perfectly good tool built from The Wisdom of the Ancients that’s existed for decades.”

    I once encountered a Makefile (amongst other non-idiomatic things) in a JavaScript project I inherited. The version of make I had didn’t like the syntax in his file, which included leading dots. There were other environment-specific problems too. I hadn’t seen a Makefile in a decade, so it took me a day of digging through the docs to refresh my memory, make it work on my machine, and then add the functionality I needed. The colleague who built it was vocal in his disgust of JavaScript. This and every other choice he made in the course of building it made it clear that he preferred to remain willfully ignorant of common patterns and tools in the JS community and was disinterested in how much suffering his non-idiomatic design choices caused for other people. Since my name’s in the commit history, people are now asking me Makefile maintenance questions. You can imagine why I was not pleased to encounter this article.

    The best solution is the one you glossed over, which is to abstract the npm script into a node script. If you need shared variables, use .env files. This solves your complaints about verbosity, variables, and comments without introducing an additional, non-idiomatic, environment-specific tool.

  • joão melo says:

    I lost control of the package.json scripts section many times. I’ve tried some alternatives but ended frustrated with some aspect or other. Finally I build a CLI for myself: https://www.npmjs.com/package/sqript

  • Eric Haynes says:

    Inlining a bunch of codependent shell scripts in a json file is, indeed, a quick pathway to a mess. However, note that you can also just have javascript scripts called by npm tasks. This is a far more expressive (and familiar, for js devs) syntax than make. A tiny, promise-based helper function around `child_process#exec` can make it really clean.

    You went from “look how much clearer this is” to “[this is] a little arcane” rather rapidly. It’s been my experience that that snowballs rapidly with make once you start `include`-ing multiple files, ability to mutate global variables, and even “functions” leads to a convoluted mess.

    As far as “Can your build tool do that?”, you’re starting to see quite a few that can, indeed, do that much faster and in a far more scalable fashion.

  • Markus says:

    Thanks for this article. I never used Makefiles before, wrote the first one some minutes ago due to your article and seeing a colleague build one today. Feels like magic if you’re only used to running package.json scripts.

  • Comments are closed.