How many API tokens/keys do I need?

Each knapsack_pro command defined in the CI pipeline should be used with a unique API token. The purpose of an API token is to track a specific set of tests. When running two different sets of tests within the same test suite (for example, unit tests and feature tests run separately), you need two separate API tokens.

Examples include:

• Step 1: API_KEY_A for bundle exec rake knapsack_pro:cucumber.
• Step 2: API_KEY_B for bundle exec rake knapsack_pro:rspec.
• Step 3: API_KEY_C for running a subset of RSpec tests with KNAPSACK_PRO_TEST_FILE_PATTERN="spec/features/*_spec.rb" bundle exec rake knapsack_pro:rspec.
• Step 4: API_KEY_D for bundle exec rake knapsack_pro:rspec[--tag tagA].
• Step 5: API_KEY_E for bundle exec rake knapsack_pro:rspec[--tag ~tagA].
• Step 6: API_KEY_F for Queue Mode with bundle exec rake knapsack_pro:queue:rspec.

Notes:

• If you use the KNAPSACK_PRO_TEST_FILE_PATTERN environment variable to run a specific subset of your test suite, a separate API token is required. This approach is helpful for tracking the same test file pattern across branches, such as running only unit tests without feature tests.
• Applying --tag tagA or --tag ~tagA runs a subset of your test suite, necessitating a separate API token.
• If you wish to use both Regular Mode and Queue Mode, you will need a separate API token for each mode.

Multiple API Tokens for Different Environments

If you have a single RSpec test suite that you want to run concurrently against different environments, such as mobile and desktop, you should generate separate API tokens for each environment, e.g., RSpec Mobile and RSpec Desktop.

Multiple API Tokens for Sequential Steps on CI

When running parts of your test suite in steps on the CI server (e.g., unit tests first and feature tests second), you need a separate API token for each unique test file pattern set with the KNAPSACK_PRO_TEST_FILE_PATTERN environment variable. This configuration allows for sequential or parallel execution within the same CI build ID.

Using the Same API Token

There is a scenario where using the same API token for different test sets is appropriate. If you run all RSpec tests on the main branch but only a subset on feature branches related to specific code changes, use the same API token. This approach enables tracking across all branches in the Knapsack Pro dashboard under the same CI build list for that API token.

Examples:

• Run all tests on the main branch and use the KNAPSACK_PRO_TEST_FILE_PATTERN environment variable on feature branches to run a subset of tests.
• Users who use the crystalball gem to select a minimal test subset for feature branches should also use the same API token. Even if the KNAPSACK_PRO_TEST_FILE_LIST_SOURCE_FILE environment variable defines different test files for each CI build, using the same API token allows Knapsack Pro to optimize test balance in Queue Mode effectively.

Switching Between Regular Mode and Queue Mode

We recommend recording the timing of a new test suite with API token A using knapsack_pro in Regular Mode. After recording the test suite timing, you should use the same API token A to run your tests in knapsack_pro Queue Mode. This approach ensures that Queue Mode benefits from the test suite timing recorded quickly with Regular Mode, preventing the first Queue Mode run from being slow due to the initial recording of test file timings.

However, if you decide to switch back from Queue Mode to Regular Mode, using the same API token could lead to edge cases where some builds might not be well balanced in Regular Mode. Therefore, we recommend using separate API tokens for Regular Mode and Queue Mode to avoid such issues. If you plan to use only Queue Mode, then this concern does not apply.