How to run multiple test suite commands in Heroku CI?

Heroku CI allows specifying only single command in app.json to run your tests on Heroku CI.

If you have multiple test suite for instance in RSpec and Cucumber and you would like to run both of them on Heroku CI then you can create a bash script bin/knapsack_pro_run_tests and inside of it, you can run multiple commands with tests.

# app.json
{
  "environments": {
    "test": {
      "formation": {
        "test": {
          "quantity": 2
        }
      },
      "addons": [
        "heroku-postgresql"
      ],
      "scripts": {
        "test": "bin/knapsack_pro_run_tests"
      },
      "env": {
        "KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC": "rspec-token"
      }
    }
  }
}

Option 1: Run all Knapsack Pro commands and exit with non zero exit code

We need to store exit code from each Knapsack Pro command and ensure bash script bin/knapsack_pro_run_tests exits with non zero exit code if one of Knapsack Pro commands fail. Thanks to that Heroku CI will know that our tests failed and it will mark your CI build as failed.

#!/bin/bash
# bin/knapsack_pro_run_tests

# run tests in parallel with Knapsack Pro Queue Mode for Cucumber
bundle exec rake knapsack_pro:queue:cucumber
# $? does read exit code from last executed command
export KNAPSACK_PRO_EXIT_CODE_1=$?

# run tests in parallel with Knapsack Pro Queue Mode for RSpec
bundle exec rake knapsack_pro:queue:rspec
export KNAPSACK_PRO_EXIT_CODE_2=$?

# process should exit with non zero exit code when one of test suites failed
if [ "$KNAPSACK_PRO_EXIT_CODE_1" -ne "0" ]; then
  exit $KNAPSACK_PRO_EXIT_CODE_1
fi

if [ "$KNAPSACK_PRO_EXIT_CODE_2" -ne "0" ]; then
  exit $KNAPSACK_PRO_EXIT_CODE_2
fi

Option 2: Fail fast on first failed test suite

Below approach is recommended only if the second Knapsack Pro command is using Knapsack Pro Regular Mode.

Here is bin/knapsack_pro_run_tests bash script.

#!/bin/bash
# bin/knapsack_pro_run_tests
# exit when any command fails, thanks to that we stop running tests when one of commands fails
set -e

# run tests in parallel with Knapsack Pro Queue Mode for RSpec
bundle exec rake knapsack_pro:queue:rspec

# run tests in parallel with Knapsack Pro Regular Mode for Cucumber
bundle exec rake knapsack_pro:cucumber

This option is NOT recommended if you use Knapsack Pro Queue Mode for the second Knapsack Pro command defined in bin/knapsack_pro_run_tests.

When first Knapsack Pro command will run tests and one of tests fail then bash script bin/knapsack_pro_run_tests will exit with non zero exit code due to flag set -e. Then the parallel CI node won't run tests for the second Knapsack Pro command. Because of that other parallel CI nodes that run successfuly tests for the first Knapsack Pro command would run the second Knapsack Pro command in Queue Mode and it would consume the whole Queue of tests. This would lead to running too many tests on parallel nodes that have no failing tests. The tests won't be well balanced across the parallel nodes.

Other

Learn more how to run tests with Knapsack Pro only on a few parallel dynos with Heroku CI?