How to Stop Hating Your Tests by Justin Sears

In notebook:
Article Notes
Created at:
JavaScript testing
How to Stop Hating Your Tests on Vimeo

Isolation of tests

the better we isolate tests the better it communicates what the units job is 


- Big objects are hard to work with. People who practice TDD tend to hate big objects, they are harder to deal with
Tests make big objects even harder do deal with (they have many logical branches, many dependencies, etc.)

Rule of product  if you have a function that takes four arguments (​valid(a,b,c,d)​) then the number of test cases = a ×b × c × d 

"I'll just add one more argument..." will double your test cases

suggests: "Stop the bleeding" don't add new stuff to your object
"Limit new object to 1 public method and at most 3 dependencies"
=> write lots of small units

Off Script

Tests can and should only do three things:
  1.  Sets stuff up
  2. Invokes a thing
  3. Verifies behavior 
  • Arrange 
  • Act 
  • Assert
  • Given
  • When
  • Then
always respect this order

Minimize each phase to 1 action per line

jasmine-given, mocha-given  

let > given
before -> when
then -> expect

Test design smells:

  • Lots of "Given" steps? -> Too many dependencies or too complex arguments
  • Multiple "When" steps? -> The API is too complex or hard to invoke
  • Many "Then" steps -> The code is doing too much

Hard to read, hard to skim code

"Logic in tests confuses the story of what's being tested"

Test code is untested code (might contain errors) and is hard to read
passing green is "fantasy green"

If you refactor your tests to make them more terse you make them more brittle and harder to understand

The Sim tests

uses context  for logical branches
Arrange, Act, Assert should easy by found

Tests that are too magic (or not magic enough)

Balance between expressiveness (small or big API)

spec-given reduced the API to
natural assertions

smaller testing apis are easier to understand, but create bigger tests and more one-off helper methods that you will have to carry
bigger test apis yield terse tests, but look like magic to outsiders

Tests that are accidently creative (are very bad)

always uses the subject and result consistently

inconsistent test can have a message (only if 1 in 10, otherwise it becomes a mess)

Make unimportant test code obvious to the reader. Don't use real-life looking values, but words like "pants" - it has an important meaning to the reader

Test Isolation

Unfocused test suites

Defining success: Is the purpose of the test readily apparent & does its test sutie promote consistency
The testing pyramid
Some tests call to real db, some fake third parties, some use third-party apis, etc.etc.
Start with two suites, each approaching one extreme:

One test suite(A): as realistic as possible, as integrated as we can manage (top of the pyramid)
Other suite(B): as isolated as possible (bottom of the pyramid)

Recently worked on an Ember project.
Had to write "components tests". They established the rules:
  • Fake all APIs
  • no test doubles objects
  • trigger actions, not UI events
  • Verifiy app state, not HTML templates
it bought them consistency

Too realistic

The boundaries of what exactly is considered "realistic" is very implicit in many projects (this is not good). They test for some scenarios, but not EVERYTHING, so if something blows up in production, they reply by increasing realism even more.

some practical issues with realistic tests:
  • are slower
  • take more time to write/change/debug
  • higher cognitive load
  • fail for more reasons
Solution: have clear boundaries 
Increases the focus on what's tested and what's controlled

Clear Boundaries:
if something blows up in production - they can analyse better the situation and just (as one solution) write one focused test for that scenario

"Less integrates tests offer more design feedback & failurs are easier to understand"

Redundant code coverage

You change one unit, but then you realise that you need to change all the surrounding units as well now. Kills team morale.
How to detect redundent coverage?
Look at your coverage tool and see the column that shows how many times a certain line was hit.

One approach: identify a clear set of layers to test through
Try at outside-in TDD, isolating units via test doubles (you isolate each of your units that stuff that are underneath it) <- called as London-School TD, or mockists, or discovery_testing

Careless mocking

Test double
a fake, a stub, a spy, a mock

His process (strategy) on using test doubles:

Given the subject of his test, that use three dependencies -> he fakes those dependencies and tests how easy/convienent is it to use their APIs.
does the data flow is sound -> it's still easy use because they actually don't exist

In real-life most people don't have this approach: they mimic/replace existing dependencies so that their tests will pass. They use mocks to "shut up" dependencies that are causing them pains
They make tests confusing to read

Application frameworks

Most problems (tasks) we face: "how do I make my application talk to X?" <- it's integration issue
Talks about different strategies people use when working with frameworks (integrate them at every line <-> try to dodge them)

The framework dilemma
  1. Frameworks focus on integration problems
  2. Framework test helpers are very integrated
  3. Users only write integration tests
If some of our code doesn't rely on the framework, why sould our tests?
Answer is: if you have a lot of domain-specific logic in your code, decouple the tests from the framework

Test feedback

Useless error messages

very wasteful, it adds a lot of friction and waste, you have to somewhow print out where the test fails exactly 

"use assertion libraries based on the messages they give"

Feedback loops

Speed test is extremely important can cut your productivity into a fraction: if a feedback loop time increases by two -> you just halved your productivity (count in distractions, context switch, decide action)

The mythical 10x developer is found!!! :D (a 30second feedback loop vs. 480 second feedback loop) 

Painful test data

  • inline (for testing modules)
  • fixtures (for integration tests)
  • data dump (smoke tests)
  • self-priming tests (staging, production)
You don't have to pick just one version 
Data setup seems the biggest contributor to slow tests ("citation needed" :) )

profile your slow tests

Linear build slow down

Lot of time in a test is acutally spent in either "app code" or "setup, teardown" and not in "test code"
So the total time is not linear, but can almost be exponential

"Early on set a firm cap on build duration - and then enforce it!" -> delete tests or make stuff faster when approaching the limit

False negative

In real life, true negatives are depressingly rare (when you catch a real bug)
False negatives erode confidence in our test
top causes:
  • redundant coverage
  • slow tests (it only fails in CI not in development)
ie. lots of integration tests
analyse this data