Deal with Flaky Cypress Tests: Cloud vs. Parallelized Testing

Dealing with flaky Cypress tests can be a relentless journey for a dedicated developer. For instance, last week, my team saw our once-reliable CI pipeline transform into a source of frustration as tests began to intermittently pass and fail, eroding our confidence in the automated testing process.

We spent hours on a cycle of investigating false failures, navigating through a maze of asynchronous operations and possible race conditions. Attempting to merge pull requests became a lengthy endeavor, overshadowed by the unpredictability of flakiness and constant mashing of the “retry” button. With each attempt to reproduce failures locally, we grappled with the elusive nature of the bugs, consumed by the pursuit of stability in a once-trusted test suite.

Ok, so maybe that is a bit over-dramatic. But, we were experiencing a frustrating number of flakey test issues that dramatically slowed down development. In my desperation to break the seemingly-relentless cycle of failures, I chose to reach for a new tool: Cypress Cloud. Though the temptation to fully jump ship and move to Playwright was strong, we did not have the capacity on the team to rewrite our existing test suite. I was drawn in by Cypress Cloud’s free tier and promises of parallelization, load balancing, flake detection. I figured, it’s worth a shot.

So, was it worth it?

Setup

Integration with Cypress Cloud could not have been easier. I simply created an account and added the generated ID to my cypress.config.ts file and I was ready for my first run! This run went great. The test suite ran locally in record time with zero failures. But our real issue was running tests in our CI pipeline, using Github Actions. Thankfully, Cypress Cloud supplied a cypress.yml file for GitHub Actions. I was able to easily integrate my existing configuration with this example file, which came complete with a parallelization strategy. The GitHub Action I used looked something like this:

name: Cypress Tests
on: [push]
jobs:
  cypress-run:
    runs-on: ubuntu-latest
    env:
      # Environment variables for project testing
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2] # Uses 2 parallel instances
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Set up Node.js version
        uses: actions/setup-node@v3
        with:
          node-version: "18.x"
      - run: docker-compose up -d
      - name: npm install, build
        run: |
          npm install
          npm run build
      # prepare the test database
      - run: npm run db:create:test
      - run: npm run db:migrate:test
      - name: Cypress run
        uses: cypress-io/github-action@v6
        with:
          start: npm run test:server
          wait-on: "http://localhost:3000"
          record: true
          parallel: true
        env:
          CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Results

After a day filled with agonizing failures on every CI run, I was astounded to see the entire test suite pass in CI on the first run. Due to parallelization, it also ran in about half the time. The CI gods were finally smiling upon me it seemed. I was also able to view analytics and test run history on the Cypress Cloud dashboard to monitor test runs and keep an eye out for any signs of flakey tests. This blissful feeling lasted for about a day. As soon as I had my new Github action up and running and started sharing the wonders of Cypress Cloud with the dev team, the emails from Cypress.io started coming in with subject lines like these:

You’re nearing your usage limit

Heads up! You’re out of test recordings

With Cypress Cloud, it doesn’t take long before you have to pay the piper. The free tier only allows for 500 test results per month and a limited number of recordings. Unless we wanted to start paying $75 per month or more, Cypress Cloud was not the answer to our CI testing woes.

An Alternative

Not ready to invest in such a pricey solution, I began to look for alternatives to Cypress Cloud and found this blog post by Glen Bahmutov, a former employee of Cypress.io who created cypress-split, a Cypress plugin that can automatically split an entire list of Cypress specs to run in parallel on any CI — including GitHub Actions! The GitHub action I used for this method looked something like this:

name: Cypress Tests
on: push
jobs:
  prepare:
    runs-on: ubuntu-22.04
    # explicitly set the output of this job
    # so that other jobs can use it
    outputs:
      matrix: ${{ steps.prepare.outputs.matrix }}
    steps:
      # generate the list using a bash script
      - name: Create matrix ⊹
        id: prepare
        # for reusable workflow, must use the full action reference
        uses: bahmutov/gh-build-matrix@main
        with:
          n: 3 # number of containers to output
  test-split:
    needs: prepare
    runs-on: ubuntu-22.04
    env:
      # Environment variables for project testing
    strategy:
      fail-fast: false
      matrix: ${{ fromJSON(needs.prepare.outputs.matrix) }}
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      - name: Set up Node.js version
        uses: actions/setup-node@v3
        with:
          node-version: "18.x"
      - run: docker-compose up -d
      - name: npm install, build
        run: |
          npm install
          npm run build
      # prepare the test database
      - run: npm run db:create:test
      - run: npm run db:migrate:test
      - name: Run split Cypress tests
        # https://github.com/cypress-io/github-action
        uses: cypress-io/github-action@v5
        with:
          start: npm run test:server
        env:
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }}
          DEBUG: cypress-split

While this solution doesn’t offer the bells and whistles of the Cypress Cloud dashboard, it certainly has benefits. Running tests in parallel uncovers failures faster, allowing for a shorter feedback loop. Splitting test runs into shorter jobs also makes re-running failed tests quicker, as we do not have to re-run the entire test suite every time the job fails.

Navigating the Path to Stable Testing

My quest for stability amid the chaos of flaky Cypress tests took me through the promises and pitfalls of Cypress Cloud. The allure of seamless parallelized testing, load balancing, and flake detection initially captivated me, offering a glimmer of hope for streamlined CI testing. However, as the intoxicating scent of success lingered, so did the realization that free rides in the realm of Cypress Cloud are fleeting.

The initial euphoria of a passing test suite and swift CI runs gave way to the harsh reality of usage limits and impending charges. Cypress Cloud, with its price tag, seemed poised to be a luxury beyond my current project budget.

However, I discovered an alternative path, thanks to Glen Bahmutov’s cypress-split plugin, which offers the prospect of parallelized testing without the hefty price tag. While it may lack the graphical allure of a Cypress Cloud dashboard, its practical benefits became apparent: faster failure detection, shorter feedback loops, and efficient reruns of failed tests.

While Cypress Cloud emerged as an excellent solution for those who can afford it, I was pleased to find a road less expensive that may offer some of the same benefits. Whichever option you choose, may your testing endeavors be swift, reliable, and kind to your budget.

Happy testing! 🚀