How many API tokens/keys do I need?

Each knapsack_pro command defined in the CI pipeline should use an individual API token. The purpose of an API token is to track a set of tests. When you run 2 different sets of tests within the same test suite (e.g., unit tests and feature tests are run separately), you need two separate API tokens.

Here are examples:

• Step 1. API_KEY_A for bundle exec rake knapsack_pro:cucumber
• Step 2. API_KEY_B for bundle exec rake knapsack_pro:rspec
• Step 3. API_KEY_C for KNAPSACK_PRO_TEST_FILE_PATTERN="spec/features/*_spec.rb" bundle exec rake knapsack_pro:rspec
• Step 4. API_KEY_D for bundle exec rake knapsack_pro:rspec[--tag tagA]
• Step 5. API_KEY_E for bundle exec rake knapsack_pro:rspec[--tag ~tagA]
• Step 6. API_KEY_F for bundle exec rake knapsack_pro:queue:rspec

Note:

• If you specified the KNAPSACK_PRO_TEST_FILE_PATTERN environment variable to run a subset of the test suite, then you need a separate API token. This is useful when you want to track the same test file pattern across your branches. For instance, you want to run only unit tests without feature tests.
• If you pass --tag tagA or --tag ~tagA, then you run a subset of the test suite, so you need a separate API token.
• If you want to use Regular Mode and Queue Mode at the same time, then you need a separate API token for each mode.

When you run the same test suite for the different environments then you need multiple API tokens.

Let's say you have a single RSpec test suite and would like to run it concurrently against different environments like mobile and desktop. You should generate separate API tokens: RSpec Mobile and RSpec Desktop.

When you run the same test suite in steps on the CI server then you need multiple API tokens.

For instance, you run only unit tests and feature tests as a second step. Therefore, for each unique test file pattern defined with the KNAPSACK_PRO_TEST_FILE_PATTERN environment variable, you need a separate API token.

This allows you to run both steps sequentially or in parallel as part of a CI build with the same build ID. Typically you would configure sequential steps in your CI provider, but some CI providers allow you to start parallel steps so that you can run unit tests and feature tests at the same time.

When should I use the same API token?

There is a scenario where you should use the same API token even though you are using a different set of tests.

Let's say you run all your RSpec tests on the main branch. But on your feature branches, you want to run only a subset of the test suite that is related to the feature code changes. In such a case, you should use the same API token. This way, you can see all your branches in the Knapsack Pro dashboard on the list of CI builds for a given API token.

Examples:

• Run all tests on the main branch and define a test file pattern with the KNAPSACK_PRO_TEST_FILE_PATTERN environment variable for your feature branches to run a subset of the test suite.

• Some users use the crystallball gem. Its primary purpose is to select a minimal subset of your test suite which should be run to ensure your changes didn't break anything. You could run an RSpec test suite that takes 10 hours on the main branch, but on the feature branch you may use custom tools or the crystalball gem to detect what part of the codebase was modified and then run only a small subset of the 10 hours test suite. You can define a small subset of the test suite with the KNAPSACK_PRO_TEST_FILE_LIST_SOURCE_FILE environment variable (learn more). In such a case, you should use the same API token even though a test files list that's defined in the KNAPSACK_PRO_TEST_FILE_LIST_SOURCE_FILE environment variable can be different for each CI build on your feature branch. The same API token is recommended so that Knapsack Pro can leverage collected test execution data to help you balance tests in Queue Mode well.