What's the impact of how long your tests take?

Preface: There has been a lot of talk lately about whether or not Test First is Dead.This is a complicated and tangled issue, so I want to simplify many aspects of this debate in this blog post to focus on a single aspect of TDD: 

Test Run Time.

So for this blog, let's ignore the distinction between unit, integration, acceptance tests, even manual tests. I'll lump them all together into a single box - 'tests' so we can focus on the 2nd benefit of Tests: Feedback

How fast is fast enough for you tests to run? There is a lot of discussion. minutes? seconds? fractions of a second?

Usually there is a very important step skipped in this discussion. People never seem to ask:

What does it matter how long they take?

Cycle Time

Have you ever needed to make 'just a small fix'? You change a line of code. Compile, save, deploy (maybe to your phone or websever) look at result. 

"Damn." That didn't actually fix it. You change the line again, Compile, save, deploy, look at the result... "Damn". 

4-5 tries later and it's fixed. less than 20 characters typed. Less than 60 seconds of work and yet the clock says it's an hour later than you started.

What is going on here?

Again a lean graph is helpful in mapping out and understanding this system. 

  The problem is for every 10 seconds of work you do, you have to wait 10 minutes to get your feedback. This is a highly infective way of working, which is why we don't do it that offend. 

Although this phoneme is well know enough to inspire xkcd comics

To discuss this part it is better to use the repeat notation to display the above lean graph

Work to Wait Ratio

Now, we tend to want high feedback but we also want a good work to wait ratio. I find that in practice we want at least a 3:1 work to wait ratio.

That means if it takes you 10 minutes to get feedback (test it out) then you will probably end up coding in 30 minute or longer chunks before running the code. If compiling and running takes you 3 minutes you'll work in at least 9 minute chunks.

This allows a lot of time for mistakes to fester and complexity to compound.

A lot of good Test Driven Developers I know like to run their test's at least 3 times a minute. That means 20 seconds per cycle, which means an absolute max of 5 seconds per run to achieve the minimum work to wait ratio.

But this changes it get to the 'very very quick' stage. If you are typing in word it underlines typpos  as you go. This might seem like a minor connivence over the 'Check Spelling' feature but, then again, when is the last time you remember running a spell check manually? 

It worth noting I know many legacy systems where the compile time alone can take 3 hours. That means if I come back from lunch and program for 1 hour, I will not be able to see the effects of my changes before I go home for the night.

Discovery 

One last thing to consider is the aspect of discovery when creating. When you are trying to be creative and discover 'the right thing' then the immediacy of feedback is very very important. This is why many people find themselves preferring things like paper, whiteboards,  wysiwyg editors when they are brainstorming. 

This is part of why you are seeing more things like continuous test runners and website monitors that automatically refresh on file change. 

You'll find that the longer the cycle times the more deliberate each action becomes. This drastically lowers the chance for a joyful discovery. Not many people played around with punchcards.

If you are interested in this aspect of feedback, I suggest you check out this talk [inventing on a principle] by Bret Victor. Who has though about this more than anyone else I know of.  

Test First vs Test After

There has been a lot of talk lately about whether or not Test First is Dead.
This is a complicated and tangled issue, so I want to simplify many aspects of this debate in this blog post to focus on a single aspect of TDD: time.

So for this blog, let's ignore the distinction between unit, integration, acceptance tests. I'll lump them all together into a single box  - 'tests'.

Let's focus on the speed aspects of Test First vs Test After.

Test After

To start, we'll look at the time it takes to write code using test after.
I'm going to use a lean graph to help map the differences between 'actual work' and 'non-work'. I get paid to write code, not tests, so I'll map code to work and tests to 'non-work'.

In this example, it took me 60 minutes to code a feature. Then more time to write the tests for that feature.

Now for the key question:

"Is it faster to not write the tests?"

In other words, which is true:
 a)  60  < 60 + X
 b)  60  > 60 + X

This is obviously  a)  60  < 60 + X  regardless of how long or quickly you can write the tests.
This doesn't mean that in the long term it isn't faster to write tests, but it does mean in the short term, if you write tests after, it will always take you longer.

so what about...

Test First

When you have a test before you start it can make it easier for you to write the code. This is because the Tests can aid with Specification & Feedback of the task at hand. This saves some time. Again, we'll assume for this example that it saves me 30 minutes while writing the code.

Now for the key question:

"Is it faster to not write the tests?"

In other words, which is true:
 a)  60  < 30 + X
 b)  60  > 30 + X

This is a more difficult question. It depends on how long it took to write the tests (what the value of X is). If X is less than 30 minutes it is faster. If X is longer than 30 minutes it is slower.

This means that the techniques, technologies & fluency of the programmer all affect the result of this equation. So it is not a clear cut answer & it will vary depending on the individual.

Conclusion

For me, it is often quicker to write the test. I have the advantage of years of fluency and tools like approval tests. However, the interesting point to me is while Test After means:  writing tests takes more time

Test  First means: writing tests may or may not save you time

I believe this possibility is worth exploring.