The 4 Benefits of Tests

By Llewellyn Falco & Jacqueline Bilston

I’ve found that when I'm creating tests they provide 4 categories of benefits. Having these categories helps me to see how to write better tests and helps me see what to improve when I am feeling a particular pain. These benefits tend to occur in chronological order, which is how I’m going to lay out the explanations.

Note: This article is about tests as artifacts, not testing as performance. There is another area of testing known as exploratory testing. It is a different practice and provides different benefits.

#1 Specification

“What am I building?”

At the very beginning, before any code is even written, writing a test scenario can help you understand what it is you’re trying to build. These scenarios often work much better than requirements, as the specifics of a scenario will often surface hidden issues. Programming is often done between two or more parties. Sometimes it can feel that way even if you are doing it by yourself. Scenarios will surface misunderstandings where you thought you were in agreement; edge cases you weren’t considering; and logical errors before they are written.

Even if this is as far as you proceed with your tests, they will still bring value.

#2 Feedback

“Does it work?”

After you have a test scenario, you can immediately start getting feedback on whether or not it is completed. This feedback can come in many forms:

Programming by red: Compiler errors can guide you on what needs to be built next. Failing tests can guide you on what needs to be implemented differently.

Debugging / Logging / Inspecting: Running code and seeing the results is critical to understanding that you’ve built what you meant to. Just imagine if you shipped code that you never ran :-/ 

People don’t know what they want until they see what they don’t: Showing the results to a customer will bring insight into if you are building the right thing.

#3 Regression

“Does it still work?”

Just because code worked yesterday does NOT mean it still works today. Rerunning your tests allows you to know that things still work. A suite of passing tests gives confidence that things still work. Better still, a single failing test proves that something doesn’t.

Note: In my experience, you won’t get good regression without automated tests.

#4 Granularity

“Why did it break?”

Granularity can show up in many forms, but it’s always about figuring out how to fix what went wrong. I tend to see 2 main forms of it.

#1 Temporal Granularity: 

You just changed a line of code and it broke the build. That line is the problem. Maybe you just roll it back, maybe you fix it and roll it forward. Either way it’s fixed and it was easier than if you had changed 1,000 lines of code.

or...

5 people commit and the nightly build breaks, who caused it?

vs.

CI runs after each commit and Jonny's commit broke it.

or ...

The tests passed 2 minutes ago and now they are broken

vs.

The tests passed this morning and now they don't.

#2 Differential Granularity

‘Exception thrown’ doesn’t help anybody

‘Array index out of bounds’  is a little better.

‘Array[10] is out of bounds for length 10’ means you have an off-by-one error.

‘Array[-2] is out of bounds for length 10’ means you have calculation bug.

‘Array[two] is out of bounds for length 10’ means you have to verify user input .

‘Array[5] is out of bounds for length 0’ means you have to check if result came back empty.

Logs in JSON are queryable which helps you find issues faster.

Objects with useful toStrings help you understand the state of your system.

Diff tools help to find what changed between two outputs.

Giving meaningful output helps us understand the causes for failures and the potential fixes.