How Can You Do TDD When You Don't Know What The Answer Is?

I'm afraid my issues with TDD are going to be a recurring theme on this blog.

Let me state up front that tests are awesome. If you can set up unit tests, they are a fantastic development aid. I can't begin to say how nice it was to be able to pound on my unit tests when I did my recent major rewrite of my code. I love those tests, and wish I'd been more thorough writing them for the last decade. And my overall test suite is an essential development tool. I'd be lost without it.

But... the entire "test first" TDD development strategy depends on knowing what your code is supposed to be doing at a detailed level AND being able to write a test for it. (At least as I understand it.) For instance, it's easy to imagine how to check your amortization program to see if it is giving you the correct answers. TDD examples are full of stuff like this. But how do you test your MP3 encoding program? Your regular test suite cannot incorporate a double-blind panel of listeners with a variety of sound reproduction equipment to ensure the music sounds good to human ears, or better than some standard. You can do all sorts of tests to make sure low-level code is working the way you expect it to. But that testing will always miss the essential issues. You can automatically test that the component algorithms work, but you cannot automatically test that you are using them correctly.

Needless to say, my work looks more like the messy second case than the tidy first case. A lot of my development work is essentially experimental reverse engineering -- what interpretations of this data are needed to make coherent geometry from them? I have the advantage over the MP3 developers that the basic coherency tests can be automated (though they take hours to run in parallel on a pretty fast quad core machine). But they are just rough and ready tests that the data we are generating from the file mostly seems sensible, and I've never been able to get them close to having no failures. It's always a statistical test -- is the success rate greater after the change -- rather than an absolute right or wrong test.