jq for Creating and Updating JSON

jq is an amazing little command line utility for working with JSON data. We’ve written before about how you can use jq to parse JSON on the command line, but in this post I want to talk about using jq to create JSON data from scratch or make changes to existing data.

Generating JSON

Once in a while, I’ll find myself needing to write a Bash script that uses curl to retrieve some data. In most cases, the API I’m using requires an auth token. That means I first need to authenticate with the API before making any additional requests.

Let’s take, for example, an API that takes a username and password in a JSON object:

{
  "username": "developer-1",
  "password": "qwertuiop"
}

It then returns a response object that includes an auth token:

{
  "data": {
    "token": "this-is-an-auth-token"
  }
}

Hard-coding the username and password in a script isn’t a great idea. Instead, we’ll use environment variables to provide the values. In some languages, interpolating a couple of variables into a small JSON blurb wouldn’t be that difficult. Unfortunately, Bash isn’t one of them. At least not for me – I run into issues with quoting every time I try.

I’m sure there’s some combination of single quotes, double quotes, and escaped quotes that will work. But, after learning about jq’s ability to use arguments in its output, I haven’t had to worry about it again.

jq has a --arg option that passes a value as a predefined variable, that can then be referenced in the output. And it will properly quote the value when it generates the output. We’ll use it here to generate the request body:

AUTH_BODY=$(jq --null-input \
  --arg user "$USERNAME" \
  --arg password "$PASSWORD" \
  '{"user": $user, "password": $password}')

The --null-input argument tells jq not to read any input at all (it’s used when constructing JSON data from scratch).

The resulting JSON can then easily be dropped into a curl request:

AUTH_TOKEN_RESPONSE=$(curl -s \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d "${AUTH_BODY}" \
  https://api.example.com/auth/token)

And then finally, jq can be used to parse the response JSON to get the auth token:

AUTH_TOKEN=$(echo "$AUTH_TOKEN_RESPONSE" | jq -r .data.token)

The -r (short for --raw-input) argument tells jq to not include the quotes it normally displays with a JSON string value.

Updating JSON

In the example above, we created new JSON from scratch. jq is equally useful for updating existing JSON data.

In this example, we’ll start with a JSON object that defines a command to be executed and any environment variables that should be set when the command is run:

{
  "definition": {
    "command": "/releases/32/run.sh",
    "environment": [
      {
        "name": "VERSION",
        "value": "2.3.1"
      },
      {
        "name": "RETRY_MAX",
        "value": "5"
      }
    ]
  }
}

I want to create a release script that will read the existing JSON file and output to an updated document that has the new path for the command and the new VERSION environment variable.

To do this we can use jq’s “update” operator |=. Here’s a simple example showing how |= can be used to replace the value of a field:

> echo '{ "foo": "bar" }' | jq '.foo |= "baz"'
{
  "foo": "baz"
}

Picking up the release script example again, we’ll pipe multiple updates together to make all of the necessary changes in a single command:

# These would likely be command-line arguments in a real script
VERSION=2.3.2
COMMAND_PATH="/releases/33/run.sh"

cat current.json | jq \
  --arg version "$VERSION" \
  --arg command_path "$COMMAND_PATH" \
 '(.definition.environment[] | select(.name == "VERSION") | .value) |= $version
  | .definition.command |= $command_path'

This uses jq’s powerful filtering capability to find the object in the environment array that has the name of “VERSION”, and then updates the value property. The result of that change is piped into another usage of the update operator – this time to change the command.

In this age of JSON APIs and JSON configuration files, there are endless ways a fantastic command-line utility like jq can be used to make your life easier. It’s the first tool you should reach for when you need to parse JSON on the command line. But keep in mind how handy it is when you need to create new JSON, or update an existing JSON document as well.