In our daily TDD/BDD work, we spent most of our time in test cases, we analyze requirements, write test case, run it and watch it fail, implement code to make the test pass, repeat the process "until money runs out". Thus the speed of running those test cases have significant impact on a team's performance, ! You can't tolerance writing a test case, waiting 1 minute finish; You can't tolerance CI finishes running all RSpec/Cucumber tests for half an hour! I can't! So I motivated to improve this!
In the project I was working on there are 1566 RSpec examples for all models, after I applied most tips appropriate for out project, the result below is a comparison before and after boosting, the command is:
time bundle exec rspec spec/models
Before boost: After boost:
One minute and 8 seconds are saved after the boost!!! And it is only part of our all RSpect test, we definitely have tests for controllers, views, libs, etc., a 50.3% improvement!
To be honest from the beginning I admit I don't have too much confident I can make things better, until I saw this presentation: Grease your suite which surprised, inspired and encouraged me quite a lot! I did two whole days research and summarized 12 super useful tips for speeding up the RSpec/Cucumber tests!
Spork has two major advantages:
My real experience is it is awesome! I wrote a few line of code and I run the test, I can see the result in much shorter time than before! BDD development in daily work becomes more enjoyable, less frustration when waiting for slow tests times and times. Trust me, once you get used to it you'll never wanna fall back to the old slow way!
Before getting into the detailed tuning option I would like to review a few important points:
By reading the official documentation, we got the following optimizable options:
We can reference Twitter's GC tuning options below used in their production:
Twitter’s settings mean: Start with enough memory to hold the application (Ruby’s default is very low, lower than what a Rails application typically needs). Increase it linearly if you need more (Ruby’s default is exponential increase). Only garbage-collect every 50 million malloc calls (Ruby’s default is 6x smaller). The expert who did the PPT on Heroku I mentioned aboverecommended the following options:
How to enable the GC options above? Personally I store those values in my .rvmrc file under project folder, you can export them in other places. And please notice the option values above are for your reference, might not quite appropriate for every concrete project, how to set those options suitable for your real project can be determined by on a simple rule: better settings makes less times of GC running. I wrote the following code in spec_helper.rb to display how many times GC runs and how much time it cost in total after running given tests examples.
The sample output of GC stats after running 360 examples:
More resources on Ruby GC tuning:
Other than adjusting GC tuning options we can also deffer GC runs! Create a file under spec/support/deferred_garbage_collection.rb and paste code below:
By default in Rails development/test environment the logger's level is 0 which means
:debug, however, most of the time we don't need verbose information flushed into log file, we can absolutely increase the log level to reduce IO! Setting it to 3 (:error) would be best IMHO, there is no redandunt information but still be able to trace failed tests.
Reduce Devise.streches can improve Bcrypt performance, because we don't need strong encryption during testing, the default value is 10, we can set it to 1 in test environment.
Real HTTP requests can vastly slower down your tests, unless you explicitly want to "test" the request (usually in integration testing), you should mock it to isolate your testing "target" and focus on the behavior of your code.
WebMock stubs HTTP requests at low Net::HTTP level so there is no need to change any of your tests, by default it disables all real HTTP requests, you can register requests you want to test to cross the wall, or you can put
WebMock.allow_net_connect!at your spec_helper.rb for RSpec or env.rb for Cucumber, personally I don't quite suggest allow all requests at a high level, because sometimes unobtrusive HTTP request happens and it is not easy to discover by just reviewing the code, for example:
So if you disable all the real requests in testing (by default when using WebMock), those unobtrusive request can be easily found out.
Put the following code in spec_helper.rb and env.rb:
Run tests in multi CPU cores in parallel is extremely powerful, on a four-core Intel i7 processor with hyper-threading enabled, parallel_tests can kick-off 8 processes to run test separately on each core, in the idea situation it could reduce time cost down to 50%, and the more case you run in parallel, the more dramatical differences you will get.
However there are couple of things we need aware:
use_transactional_fixtures not robust since in RoR transaction works on one DB connection instance! So this enhancement forces using one shared connection when running Capybara, and this improves performance a little bit as well.
Put the following code in spec_helper.rb and env.rb:
Many Rails application uses the famous image processing library PaperClip, it uses libraries of ImageMagick, they are all great libraries, but the problem for us is we don't want to really invoke them during test in most of the time, testing functionalities in those third party library makes no sense for your project, we should isolate it and focus on our own business logics.
Add following line into test.rb, it will avoid invoking real ImageMagick command thus significantly improve tests related with PaperClip.
We can use an in-memory DB for testing environment would be sweet, because there is no need to persistent. We do have a lot of choices of in-memory DB for example:
More resources on this
In memory SQlite database for testing Rspec, Cucumber: best speed database clean strategy
Actually in idea situation, we can consider adopting "No Database strategy" for our tests! Refer UnitRecordfor reference and see how fast the author did!
Finished in 19.302702 seconds.
4920 tests, 7878 assertions, 0 failures, 0 errors
This is the last tip for testing performance, unlike above tips/techniques: it is non-technical, however it is probably the most important one!
rspec . --tag speed:'fast'
rspec . --tag mandatory:true
Daily BDD/TDD with fast, "responsive" tests which can give you feedback in short time makes development life more enjoyable and happy, so does CI! The 12 tips above might not all suitable for your real project, but a number of them should do. If you didn't get a good result after boosting, run rspec with --profile/-p, RSpec will then track the top 10 slowest examples, for Cucumber pass "usage" as a output format will also reveal slowest step definitions.
Wish the tips help you!