How to split slow RSpec test files by test examples (by individual it)?

❗ RSpec requirements:

• You need RSpec >= 3.3.0 to use this feature.
• Please ensure you don't use deprecated RSpec options because it could lead to skipping tests!
• It's not allowed to use RSpec --tag option

In order to split RSpec slow test files by test examples across parallel CI nodes you need to set environment variable:

KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES=true

It allows a single test file to be auto split by test examples between parallel jobs (parallel CI nodes).

Thanks to that your CI build speed can be faster. It works with Knapsack Pro Regular Mode and Queue Mode.

We recommend using this feature with Queue Mode (see knapsack_pro gem installation guide) to ensure parallel CI nodes finish work at a similar time which gives you the shortest CI build time.

If you find any issue to enable the feature, see common problems.

What are the benefits of RSpec split by test examples feature?

• Your slow test files are the bottleneck and delay the whole CI build time. When you split slow test files by test examples in parallel CI nodes it means you remove the bottleneck and enable yourself to run much more parallel CI nodes than before to get much faster CI build time.

• The more parallel CI nodes you use the more slow test files will be detected by knapsack_pro gem and they will be split by test examples.

• We recommend running at least 2 CI builds after you increase the number of parallel CI nodes. This way Knapsack Pro API can learn about your test examples execution time to better predict tests split in the future CI builds.

How to verify that RSpec split by test examples works as expected?

You can open one of your latest CI builds from the user dashboard.

When slow test files are detected then they will be highlighted in yellow.

When you use RSpec split by examples feature then tests will be split by examples. RSpec adds ID of the test example [1:1] at the end of a spec file path, for instance, spec/example_spec.rb[1:1]. Here is a list of test files split by examples:

As you can see test file paths to test examples are not highlighted in yellow when they are not a bottleneck anymore. Sometimes it can happen that you have a single test example (it test which does a super slow work, for instance, capybara test). In such a case when the test file path for the test example is highlighted in yellow then this is your bottleneck. You can refactor the test or think how to write 2 smaller and faster test examples.

How does the split by slow RSpec test files work?

Let's see an example. You have 2 test files:

spec/features/dashboard_spec.rb it takes 7 minutes to run.
spec/features/homepage_spec.rb it takes 3 minutes to run.

In total, those 2 test files take 10 minutes to execute.

You run the test suite on 2 parallel CI nodes. In a perfect scenario, we would like to run each parallel job for 5 minutes because 10 minutes / 2 parallel jobs are 5 minutes. But we can't because one of the test files is slow. File spec/features/dashboard_spec.rb takes 7 minutes to execute which is more than expected 5 minutes per parallel job. Because of this problem, your CI build will take 7 minutes to run instead of 5 minutes.

To fix the above problem we can use knapsack_pro auto split slow RSpec test files by test examples feature.

knapsack_pro knows that ideal expected time per parallel CI node is 5 minutes so it will look for test files that take more than 70% * 5 minutes = 3.5 minutes to run. It means if the test file takes >= 3.5 minutes then it will be split by test examples between parallel CI nodes. We use the 70% threshold to adjust to randomness in the execution time of test files. The slower the test file the more probable is that it will run not always the same amount of time. This is especially happening for feature/capybara specs or when CI provider has some performance hiccup.

Is the test file split only by it?

RSpec test file is split by test examples. In RSpec syntax test example can be marked as it or specify.

Why does knapsack_pro not split all test files by test examples?

Running test files by test examples adds additional overhead for memory usage in RSpec. If you have a very large test suite (>= 1000 test files) then running all test files by test examples would lead to running out of RAM on your CI provider. There would be no point splitting all test files by test examples. Instead, knapsack_pro gem determines which test files pose a real bottleneck for your CI build and ensures those specific files are split. It guarantees optimal performance.

Common problems

Warning: don't use RSpec --tag option

Do you pass RSpec option --tag in command like bundle exec rake "knapsack_pro:queue:rspec[--tag mytag]"? If you do it and you use at the same time KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES=true then --tag option might not be respected by RSpec due to a bug in the RSpec. RSpec has a higher priority to run test examples like spec/a_spec.rb[1:1] than respect the --tag option. This can result in running test cases that you didn't intend to run.

You should not use the --tag option when you use RSpec split by test examples feature KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES=true.

Instead of the --tag option we recommend using environment variables to specify a list of test files to run.

Warning: don't use deprecated RSpec run_all_when_everything_filtered option

If you use the deprecated RSpec option run_all_when_everything_filtered (learn) it can cause skipping some of the tests when knapsack_pro will try to run a test example and a test file at the same time. Do not use below deprecated flags!

# DO NOT USE THIS DEPRECATED OPTIONS
RSpec.configure do |c|
  c.filter_run :focus => true
  c.run_all_when_everything_filtered = true
end

The latest RSpec version provides a new option named filter_run_when_matching (learn). You can use it instead of the above deprecated option.

# this can be used instead of run_all_when_everything_filtered
RSpec.configure do |c|
  c.filter_run_when_matching :focus
end

Why are some of my test files not executed?

If your RSpec output contains a line mentioning the focus tag, e.g.:

Run options: include {:focus=>true, :ids=>{"./spec/example_spec.rb"=>["1:1:2"]}}

it means that RSpec will run only tests tagged as focus. You could have this option set accidentally. It must be disabled in order to use the RSpec split by test examples feature.

Please ensure:

• you don't have --tag focus or -t focus options in your .rspec file,
• you don't set the focus tag in spec_helper.rb, e.g. RSpec configure { |c| c.filter_run :focus => true }. See explanation.
• you don't pass the --tag focus or -t focus option as an argument to the knapsack_pro command.

Why do I see the LoadError: cannot load such file -- example-ruby-gem error?

You load example-ruby-gem gem in your Rakefile so when knapsack_pro gem is running a rake task the example-ruby-gem gem is not loaded. You can fix your project configuration by not loading example-ruby-gem in RAILS_ENV=test environment. Or if you need it, then please ensure you add example-ruby-gem in :test group in the Gemfile.

Why do I see an error: Don't know how to build task 'knapsack_pro:rspec_test_example_detector'?

If you see an error like:

Don't know how to build task 'knapsack_pro:rspec_test_example_detector' (See the list of available tasks with `rake --tasks`)

It probably means bundler can't find the rake task. You can try to remove the default prefix bundle exec used by knapsack_pro gem by setting KNAPSACK_PRO_RSPEC_TEST_EXAMPLE_DETECTOR_PREFIX="".

Why do I see the error Could not generate JSON report for RSpec. Rake task failed when running RACK_ENV=test RAILS_ENV=test bundle exec rake knapsack_pro:rspec_test_example_detector?

If you see the below error:

Could not generate JSON report for RSpec. Rake task failed when running RACK_ENV=test RAILS_ENV=test bundle exec rake knapsack_pro:rspec_test_example_detector
/usr/local/bundle/gems/knapsack_pro-2.11.0/lib/knapsack_pro/base_allocator_builder.rb:53:in `fast_and_slow_test_files_to_run'
/usr/local/bundle/gems/knapsack_pro-2.11.0/lib/knapsack_pro/queue_allocator_builder.rb:5:in `allocator'
/usr/local/bundle/gems/knapsack_pro-2.11.0/lib/knapsack_pro/runners/queue/base_runner.rb:15:in `initialize'
/usr/local/bundle/gems/knapsack_pro-2.11.0/lib/knapsack_pro/runners/queue/rspec_runner.rb:14:in `new'

knapsack_pro is failing exactly at this line:

/usr/local/bundle/gems/knapsack_pro-2.11.0/lib/knapsack_pro/base_allocator_builder.rb:53:in `fast_and_slow_test_files_to_run'

You can see the source code of fast_and_slow_test_files_to_run method here.

As you can see knapsack_pro is running RACK_ENV=test RAILS_ENV=test bundle exec rake knapsack_pro:rspec_test_example_detector rake task via shell command with Kernel.system(cmd). This rake task is failing to run.

Here is the source code of the knapsack_pro:rspec_test_example_detector rake task.

The main purpose of this rake task is to run RSpec to generate a JSON report with test examples for slow test files. Thanks to that you will be able to split slow test files by test examples into parallel CI nodes with knapsack_pro gem.

RSpec itself fails and returns a non zero exit code which means something failed in RSpec. This causes an exception. That is why you see an actionable error in the output like: START of actionable error message. You could follow the actionable message to run only RSpec on your own and verify why RSpec is failing. We hope maybe then RSpec will show some error message to the output that would give us a clue why RSpec is failing to generate JSON report with test examples.

E, [2021-03-30T17:33:12.199274 #103] ERROR -- : [knapsack_pro] ---------- START of actionable error message --------------------------------------------------
E, [2021-03-30T17:33:12.199329 #103] ERROR -- : [knapsack_pro] There was a problem while generating test examples for the slow test files using the RSpec dry-run flag. To reproduce the error triggered by the RSpec, please try to run below command (this way, you can find out what is causing the error):
E, [2021-03-30T17:33:12.199348 #103] ERROR -- : [knapsack_pro] bundle exec rspec --format json --dry-run --out .knapsack_pro/test_case_detectors/rspec/rspec_dry_run_json_report_node_0.json --default-path spec spec/models/user_spec.rb spec/features/articles_spec.rb
E, [2021-03-30T17:33:12.199368 #103] ERROR -- : [knapsack_pro] ---------- END of actionable error message --------------------------------------------------

As suggested in the actionable message you can run the below command on the CI server to run only RSpec without knapsack_pro to see if RSpec will produce some error to the output that would explain why it can't generate JSON report for slow test files like spec/models/user_spec.rb spec/features/articles_spec.rb. It's important to run the command on the CI server (in the same environment where you experience the issue).

bundle exec rspec --format json --dry-run --out .knapsack_pro/test_case_detectors/rspec/rspec_dry_run_json_report_node_0.json --default-path spec spec/models/user_spec.rb spec/features/articles_spec.rb

If it turns out that the above command executed on the CI server alone is working fine and it generated JSON file in .knapsack_pro/test_case_detectors/rspec/rspec_dry_run_json_report_node_0.json then you could do a few things. You could try to use fewer parallel nodes, or add more CPU/RAM. Most likely RSpec is causing your CI machine to freeze due to a lack of resources.

There is nothing we could do on the knapsack_pro gem level. It is RSpec that is failing here.

The most likely reason is some problem on your CI server level, like not enough CPU/RAM. We saw in the past that RSpec can eat a lot of resources and freeze or fail. So our best guess is that the performance issue is a root problem.

If RSpec is not showing any error to the output then it's hard to debug why it is failing. You would have to play with your CI settings like a number of parallel nodes, CPU/RAM/disk space, etc. This way you could find a configuration that is stable for your CI server.

How to auto split slow test files for other test runners

See other examples how to auto split test files by test cases on parallel jobs (CI nodes).