Replacing Heroku Review Apps on Non-GitHub Projects

If your software team develops multiple new features simultaneously, you need to be able to deploy and test them in isolation.

The gold standard for this is to use Heroku’s review apps, which are temporary environments automatically spun up for each pull request. Unfortunately, the feature only works with GitHub; if you’re using another source control provider, then prepare for some toolsmithing. Here’s what my team came up with.

What We Want

With review apps, our process would look something like this:

1. A developer pair begins work on a feature branch and works locally until they’re done.
2. They create a PR, which causes a review app to be created (e.g. `my-project-pr-123.herokuapp.com`).
3. Let the feedback begin! Just as the PR offers an opportunity for code review, the review environment allows for several other kinds of feedback. Our exploratory tester, designer, and delivery lead usually take a look, and they often find important things to talk about.
4. When the PR is closed, the review app is automatically destroyed.

What We Have

Without this slick integration, we manually created a fixed set of dev servers and established a process that works like this:

1. A developer pair begins work on a feature branch and claims a dev server.
2. Each green build in CI will autodeploy its feature branch to the dev server.
3. When the feature is done, they create a PR and leave a comment on the backlog story that it’s ready for review, with a link to the particular dev server.
4. Once the feature is merged, they either release their claim on the dev server so somebody else can use it or begin deploying their next feature branch to it.

How it Works

On our team, claiming a server means writing your name next to it on a widely-visible whiteboard, then editing the CI config to auto-deploy your branch to that environment. Our CircleCI workflow builds, tests, and optionally deploys:

version: 2.1

# To continuously deploy your feature branch to a review server,
# fill in the 'review_branch' and 'review_server' fields below.
aliases:
  - &review_branch feature/do-a-thing
  - &review_server our-project-dev-1

jobs:
  build:
    environment:
      NODE_ENV: test
    docker:
      - image: circleci/node:10-browsers
      - image: circleci/redis:4
      - image: circleci/postgres:10.5

  # (... the rest of the build definition goes here ...)

  deploy_heroku:
    description: "Deploy current branch to specified heroku app"
    parameters:
      heroku_app:
        description: "Where to deploy"
        type: string
      extra_git_push_args:
        description: "More git push flags (e.g. -f)"
        default: ""
        type: string
    docker:
      - image: circleci/node:10-browsers
    steps:
      - checkout
      - run:
          name: Deploy branch to Heroku
          command: git push << parameters.extra_git_push_args >> https://heroku:[email protected]/<< parameters.heroku_app >>.git $CIRCLE_BRANCH:master

workflows:
  version: 2
  build-and-deploy:
    jobs:
      - build
      - deploy_heroku:
          heroku_app: *review_server
          extra_git_push_args: -f
          requires:
            - build
          filters:
            branches:
              only: *review_branch

Pros and Cons

This is a lightweight solution that covers most of our needs with a minimal time investment. Compared with the Heroku+GitHub feature, though, it has a few downsides, which mostly relate to our fixed set of dev servers:

• The servers are inexpensive, but they’re always on instead of existing only when they’re needed.
• We have to keep them up-to-date as our resource specification changes. (e.g., adding environment variables or Heroku Add-ons).
• As demand for dev servers fluctuates, we have to manually create/destroy them to grow or shrink the pool.
• They’re reused, so a developer pair has to keep track of when `feature/a` is done with `dev-2` so that they can begin deploying `feature/b` to it.
• The only thing preventing two live feature branches from specifying the same environment and clobbering each other is our human conventions.
• We often get merge conflicts at the top of our Circle config, which nearly every branch is guaranteed to edit. (We try to remember to edit this file in a separate commit and drop it later, but this fails for similar fragile-human-conventions reasons.)

The more I think about it, the more I appreciate the design of the review apps feature we’re mimicking. An open pull request very closely approximates the useful life of a review app, and managing them without the integration is a lot of overhead.

I wish it didn’t make me think so much.

What’s Next

Instead of claiming servers on a whiteboard, we’re planning to try using some form of physical tokens that we can pass around and keep on our desks.

The automation we have is working for now, but I have a few ideas for improvements:

• Automatically create new environments using Heroku’s SDK.
• Provision resources repeatably and deterministically with Heroku’s app.json specification.
• Automatically expire/destroy review environments somehow. (Any ideas?)
• Implement a naming scheme like `my-project-feature-foo.herokuapp.com` or `my-project-pr-123.herokuapp.com`.

Do you use dev servers/review apps on your project? Did you build any of the infrastructure yourself? I’d love to hear about your experiences in the comments.