← Back to Blog Home

A Tiny Toolbox for Spelunking through JSON

Developer Tools
by: Karlin Fox

Article summary

I rely heavily on local instances of a web or mobile application’s API during development. Since I also need to speak fluently with my data on live instances of the API, including both test and production, I’ve found that I often need to work with structured JSON data at the terminal rather than a Javascript-native environment like node.js or browsers. I’ve discovered how important the tools curl, bash, jq, and json-diff can be for this sort of work, so I’d like to share some ways they’ve been useful to me when wrangling JSON at the command-line.

First install the tools I’ll be showing you:

brew install node jq curl
npm install -g json-diff

Neither should pose a challenge on Linux, just replace brew with your package manager of choice.

Glueing Little Bits of Shell

While I also recommend resty, I have recently just built up domain-specific toolchains for my API using curl and bash. Here’s a stripped-down example:

api()
{
  curl -s "http://localhost:8081$4"
}

G()
{
  if [[ -z "$2" ]] ; then
    api $1
  else
    api $1 | jq -r $2
  fi
}

Basically, make a super quick way to pull down URLs bursting with JSON data in your shell. Build out configuration and auth from this layer so that by the time you need to ask a question, you can do something like:

G /students/a8b007d

to get a specific resource. But notice that if you provide a second argument, that gets piped through the amazing jq, so:

G /students/a8b007d ._links.courses.href
#=> {"href": "/students/a8b007d/courses"}

would provide just the URL for the requested link. (I’ve imagined a university data domain for these examples.)

Drill into the Data

This is already an empowering start! We’ve moved any details of the connection and URL selection behind a simple Bash function, letting us get right to the data layer we care about. The ability of jq to then drill down into your data and make pieces easily digestible for familiar Unix tools is exciting, so here are some examples of what you could do.

(Note that I’m showing data for a HAL-based API.)

Give me the name and transcript URL for each student.

G /students '._embedded.students[]|{name:.name,grades:._links.transcript.href}'

Give me the names of all the CS professors (without foreknowledge of URLs).

 

    G $(G / ._links.professors.href) ._embedded.professors[]|select(.dept=="CS")|.name

Provide the GPA as of 2013 for all the students in this course.

 

    for s in $(G /courses/a11caad5 '._embedded.students[].self.href'); do
      G $s '.transcripts[]|select(.year=="2013").gpa'
    done

You can diff with this, or you can diff with that.

I looked around for a good JSON diffing tool and ended up with the json-diff package, which is in npm. Sometimes I really need plain diff, but other times json-diff provides the JSON-specific smarts to work with jq as “an effective team” ;)

I’ve made a Bash function to make its use more compatible with my previous curl wrappers, using temporary named pipes:

    jiff() { json-diff <(G $1 $3) <(G $2 $3); }

An optional third parameter will let jq narrow down the parts of the documents you want to diff, as long as the result is still valid JSON, (i.e. this won’t work if your query results in a string or number.)

Let’s look at how that can be helpful:

Show me the difference between these two students.

    jiff /students/d86d5c1 /students/120dfa1

Show me the differences between these two syllabi.

    jiff /curricula/94f0d33 /curricula/8dc0b42 .syllabus

Your Favorites

I’m sure I’m missing out on some sweet JSON command-line tools that I just haven’t found yet. Any suggestions?

See also: Use jq to create JSON

Related Posts