Moritz++

Going over the #perl6 backlog today, I noticed that Moritz had two suggestions for me based on my last post. The first was a nice suggestions to use given in my tests:

{
    my $match = "d'+p+<<<+accent+_B" ~~ m/ <ABC::broken_rhythm> /;
    isa_ok $match, Match, '"d+p+<<<+accent+_B" is a broken rhythm';
    given $match<ABC::broken_rhythm>
    {
        is .<note>[0]<pitch><basenote>, "d", 'first note is d';
        is .<note>[0]<pitch><octave>, "'", 'first note has an octave tick';
        is .<note>[0]<pitch><accidental>, "", 'first note has no accidental';
        is .<note>[0]<note_length>, "", 'first note has no length';
        is .<g1>[0], "+p+", 'first gracing is +p+';
        is .<broken_rhythm_bracket>, "<<<", 'angle is <<<';
        is .<g2>[0], "+accent+", 'second gracing is +accent+';
        is .<note>[1]<pitch><basenote>, "B", 'second note is B';
        is .<note>[1]<pitch><octave>, "", 'second note has no octave';
        is .<note>[1]<pitch><accidental>, "_", 'second note is flat';
        is .<note>[1]<note_length>, "", 'second note has no length';
    }
}

This is far from the amazing new testing form I was hoping to find, but it sure is a big improvement over what I had. If I don't think of something else better, I will go back and redo all of the longer test cases this way.

He also said about the barline regex, "Looks like it would need LTM to work." I believe he's talking about longest-token matching, but I don't fully understand the issues, I fear. I did finally write tests for this case:

for ':|:', '|:', '|', ':|', '::'  
{
    my $match = $_ ~~ m/ <ABC::barline> /;
    isa_ok $match, Match, "barline $_ recognized";
    is $match<ABC::barline>, $_, "barline $_ is correct";

And they didn't work -- sometimes it would match the first barline it recognized rather than the longest. Moritz also suggested, "Reordering the longest alternatives to the front would help," so I did, and then all the tests passed. I hope that means the problems are actually gone, and not just that I've managed to hide them for the moment.

Anyway, big kudos for Moritz! Thanks to him, it definitely feels like I am getting closer here. Now if I can just figure out the proper way to ask for an entire line of ABC music in Perl 6, I will be there!

Humbled by (Poor) Testing

I thought I was all ready to finish up with the Nubs / Polynomial thing last night. I started a blog post, then I looked at the code I wanted to show off. Well, it didn't seem clear enough to me to make me happy. Wait, I said to myself, this would make more sense as a Nubs member function. And that rewrite was so nice, I told myself it would be even better if I rewrote Nubs.Evaluate in the same fashion, because then the real beauty of this approach would shine out.

And then the excrement hit the fan, because after that perfectly straightforward rewrite, tests started failing left and right, with really weird errors. It took me about a half hour to figure out that this wasn't some strange Rakudo issue, it was a basic failure of my algorithm. For several weeks now, the code had been broken. But my tests were too clumsy to detect the failures until I stretched what I was doing really far out there.

Basically, the testing issue was this. The basis vector for the very first point of the NUBS curve should look like (1 0 0 0 ...). That means the very first point is just the very first control point. However, if in all your tests that first control point is some variant of (0, 0, 0, ...) (ie the origin), then a result of (0, 0, 0, ...) for the first point only shows you that the basis vector looks like (x 0 0 0 ...), where x can be anything at all. Whoops.

So it's back to the drawing board. Clearly I need more tests, and then I need to puzzle out where my algorithms have gone wrong. At least I'm pretty confident that I've got the Perl 6 issues worked out; now it's just a matter of fixing the math.

Testing and Solutions

Turns out submitting the bug I had that was messing up the search test was the right thing to do, as Moritz++ quickly pointed out it wasn't a bug at all. [<=] is a listop, so where I thought I was sending two arguments to ok, [<=] @array and "array is sorted properly", actually I was sending just one, the equivalent of [<=] @array, "array is sorted properly", and it is certainly not true that 8 <= "array is sorted properly". Problem solved!

Meanwhile, Hacker News pointed me to a very nice article by Peter Seibel on TDD, "Unit testing in Coders at Work". What's great is that it points out both cases where unit tests were absolutely crucial AND it points out great work done without them.

It also points out a hilarious example where Ron Jeffries goes completely off the rails trying to write a Suduko solver using TDD. Peter Seibel even coins a name for this "pattern": "Going in Circles Means You Don’t Know What You’re Doing".

Seriously, it's worth looking at Peter Norvig's solver to get an idea what a working Suduko solver looks like, and then check out those five posts of Jeffries'. I don't think I can say it better than Norvig did: "I think test-driven design is great. I do that a lot more than I used to do. But you can test all you want and if you don’t know how to approach the problem, you’re not going to get a solution."

BTW, Norvig's approach looks like it would be an absolute blast to code in Perl 6. If no one else has done it, I may take a stab once I've gotten a bit further on my current project...

Testing

I intended to talk about testing a lot more on this blog when I started out, before I got thoroughly sidetracked by the coolness of Perl 6. However, seeing another furor about it on the various programming sites last week, and having varied experiences with it in my own work in the same time, made me think it was time to revisit it.

First the good. A lot of the Perl 6 work I've been doing is perfectly suited to TDD. The tests are easy to write and nicely concise, with the median test just a single line long, and the longest maybe six lines. The tests run pretty quickly, even as slow as Rakudo is today. They provide direct and useful feedback about the code, and make it easy to make larger changes to the code with the confidence that the tests will find any problems you create that way.

I guess even here I wouldn't be following the rigorous ways of TDD. Typically I would code a little bit of library first, to get a feel for where it is going, then write tests for what I have done, and more tests that occur to me. Then make those tests work, rinse, and repeat. Sometimes the code would lead the test, sometimes the other way around.

But as I said, I find this really effective for this sort of work.

My problem with advocates of TDD, then, is that a lot of them seem to imagine that this sort of work is the only wort of work. But it isn't! Testing to see that two vectors are approximately equal (within a tolerance) is trivial. Testing to see that two B-reps are approximately equal (within a tolerance) is monstrously hard. Seriously, I've been doing professional work with B-reps for fifteen years now, and I have no idea how I would practically go about such a thing. (If I had code to fill the B-rep with cubes of varying sizes, you could then compare the cubes, to make sure that all the cube vertices of one B-rep were inside the cubes of the other, and all the voids were likewise empty. But that's a pair of complicated O(N^3) algorithms, and it still wouldn't handle a vast horde of common cases (like NMT B-reps and open shells).)

Putting this in concrete: last week I was working on a bug involving the orientation of B-rep faces on a simple box model. So I wrote up a test to look them. It took me several hours to write approximately 150 lines of code for the test, and it was fairly hard work. At the end of that, the test ran -- and confirmed that the model was correct as far as it could tell. (Admittedly the tests would have been easier to write if I could have written them in Perl 6 (with a well-written B-rep library) rather than C++. A lot of common B-rep operations are terribly verbose in C++.)

The end result is that I write a good many unit tests, but they are testing around the edges of things. So you can have unit tests for an assembly structure, but the assembly components are straight lines rather than B-reps. The tests can be quite helpful, but they are hardly conclusive proof your code is working.

A third example is a project I considered when I was dreaming of buying an iPhone. Wouldn't it be great, I thought, to have a little button accordion app, so you could fire up an accordion at any moment? The program would obviously consist of two main units: the user interface that allows you to press the buttons, and the music-making engine.

How would you go about unit testing something like that? One of the major components requires having someone pressing buttons on the iPhone to test it properly. The other component is generating audio. You could mock both components, but that would only let you test the interface between the two, but that's the trivial part of the program.

And if the app was good, both components would need to be finely tuned to get the proper feel. It seems to me this would require hours of sitting there playing the thing, and unit testing would help little, if at all...

Vector: Testing

Our testing logic comes via the Perl 6 Test.pm module. This comes distributed with Rakudo, but initially I cloned it to Vector just to simplify setting up the tests. The magic of Configure.pm handles the path to Rakudo's Test.pm properly, so I've deleted the Vector's local version.

The top of the file sets everything up:

etc. I'm sure there's a better way to define is_approx_vector, but this version works okay. (Actually, if - worked properly for Vector, and we defined abs to be a synonym for Length, I believe Test.pm's is_approx would compare two Vectors quite nicely.) We use test * to declare we have no plan -- that means instead of specifying the number of tests to run, we have to use done_testing at the end of the tests. Then it defines a bunch of Vectors to use in the tests, and we are off and running.

I only use a tiny subset of the functions available in Test.pm. isa_ok let's me test if the type of the supposed Vector objects is actually Vector. is checks its first two parameters for string equality. Absolute equality is a bad idea with floating point numbers, so is_approx checks that the absolute difference between two numbers is less than a small epsilon. My is_approx_vector does the same sort of thing for two Vectors.

dies_ok takes a closure and tests that it dies as expected. For instance, I use this to test that taking the dot product of Vectors of different dimensions fails instead of generating a meaningless result. lives_ok does the opposite, testing that code doesn't die without specifying what it actually does.

Put them together and it's possible to do a decent set of tests. In some cases (like 7D cross product) I had no idea what the correct answer of the operation is (other than just duplicating the formula for it again, but that seems pretty useless), but I did test that the results have the expected properties. Overall at the moment my test suite runs 256 tests, which is a lovely number IMO.

And that's pretty much it for Vector for now. I'm going to tackle doing a few simple projects using it and see what I learn from them. And I promise to revisit Vector when changes to Rakudo make it work better.

Vector: "Joining the Perl 6 Ecosystem"

After my comment yesterday about setting up a webpage pointing to the active projects coded in Perl 6 out there, the #perl6 channel reminded me about masak's proto. It is an attempt to create a simple CPAN-like tool for Perl 6. I had read about it months ago, run into some sort of difficulty using it, and just forgotten about it. But work has progressed, and masak and a few others are working on making it more powerful actively.

I've decided to try to get Vector on there, and document what I'm doing as I do it. Not that I think Vector is particularly brilliant, or that it should go in the Perl 6 CPAN when there is such a thing. But I hope it is at least both a pretty good example of how to do this sort of thing in Perl 6, and potentially a useful tool someone else could build on.

So I'm looking in the proto PIONEER file. It lists four conventions that need to be followed, the first of which is creating a deps.proto file. Vector doesn't have any dependencies; I'm not sure if having an empty file or no file is a better way of indicating that. I'm guessing an empty file, as that suggests that I have at least considered it. (Or better yet, with a comment indicating there are no dependencies?)

Next is building. Vector doesn't need a build stage, and PIONEER indicates that if there is no Makefile.PL or Configure.pl file, it just assumes the build worked, which sounds perfect for my purposes.

Step three is running tests. If there is a makefile, proto will make test. If there isn't, it will try to run prove recursively on the t/ directory. Assuming it's smart enough to run prove with the system's working Rakudo, this should work just fine with what we already have. Errr, assuming the LIB paths are set up properly.

Which is the last issue, I guess. I've just been running with Vector.pm and Test.pm in the top-level Vector directory, no need for a PERL6LIB environment variable. Will prove test figure out the paths automatically? And if I switch over to that system, should it be lib/Vector.pm or lib/Math/Vector.pm or something like that?

I think my next step is to check in what I've got now, make this post, e-mail a link to it to masak to get his comments, and head off to the pub. I will report on what happens later.

Testing in Perl 6

So, here's my first hesitant step to building Perl test code (specifically Perl 6 in this case, but I've never used any of the Test modules for Perl 5, either).

Simple and short, but with it I learned a significant limitation of my STEP extract script and turned up a Rakudobug. All in all, a most successful first foray into testing Perl code.

Now all I need to do is figure out the proper directory structure, and how to automatically build a makefile with a "test" target, and get the hacked version of Test.pm which properly handles planless testing.

Perl 6 Parsing Generously?

I just started a new little Perl 6 project by grabbing a little 99 line Perl 5 script, sticking use v6; at the top, and running it through Rakduo. Much to my surprise, a lot fewer things were complained about than I expected. (So far, I'm only about one-third through the first stage of the port.)

It did complain about my $in_file = shift; to store the next value from @ARGV/@*ARGS, about foreach $arg (@*ARGS), and about using dot from string concatenation. But it did not complain about if ($arg =~ /^\-sd/), for ($i = $1; $i <= $2; $i++), or open IN_FILE, $in_file or die. I wouldn't have expected any of those to get through the parser? (I'm assuming they'll fail when the code finally is ported enough to reach proper execution.)

I'm also trying to figure out how to automatically test this script. I guess I can easily enough throw a few files at it and check to make sure the results are correct. Does it make sense to try to unit test something this small? (As it is, the Perl 5 version has no subs at all!) And what is the accepted way of doing such testing in Perl 6?

Ovid on TDD

Hacker News just gave me a pointer to a nice post of Ovid's on test-driven development. "Many people realize that exploratory programming and TDD don't always play well together." That's a lot of my issue with it. More often than not, when I program I'm looking for the right approach to interfacing with someone else's code or file format. It's very hard to create meaningful unit tests ahead of time in those circumstances, because you don't know what the right answer is ahead of time.

How Can You Do TDD When You Don't Know What The Answer Is?

I'm afraid my issues with TDD are going to be a recurring theme on this blog.

Let me state up front that tests are awesome. If you can set up unit tests, they are a fantastic development aid. I can't begin to say how nice it was to be able to pound on my unit tests when I did my recent major rewrite of my code. I love those tests, and wish I'd been more thorough writing them for the last decade. And my overall test suite is an essential development tool. I'd be lost without it.

But... the entire "test first" TDD development strategy depends on knowing what your code is supposed to be doing at a detailed level AND being able to write a test for it. (At least as I understand it.) For instance, it's easy to imagine how to check your amortization program to see if it is giving you the correct answers. TDD examples are full of stuff like this. But how do you test your MP3 encoding program? Your regular test suite cannot incorporate a double-blind panel of listeners with a variety of sound reproduction equipment to ensure the music sounds good to human ears, or better than some standard. You can do all sorts of tests to make sure low-level code is working the way you expect it to. But that testing will always miss the essential issues. You can automatically test that the component algorithms work, but you cannot automatically test that you are using them correctly.

Needless to say, my work looks more like the messy second case than the tidy first case. A lot of my development work is essentially experimental reverse engineering -- what interpretations of this data are needed to make coherent geometry from them? I have the advantage over the MP3 developers that the basic coherency tests can be automated (though they take hours to run in parallel on a pretty fast quad core machine). But they are just rough and ready tests that the data we are generating from the file mostly seems sensible, and I've never been able to get them close to having no failures. It's always a statistical test -- is the success rate greater after the change -- rather than an absolute right or wrong test.

Should You Test Code You Use From Other People's Libraries?

So, I just lost hours (possibly days) because of a phenomenally stupid bug in a library my software uses. (The reason I'm not sure how long is because I don't yet know if this bug has been causing the slowdown I've been tracking down for a week, or if it has just been making the real bug harder to find.)

If the makers of this library had even the crudest unit test in place for this function, they would have detected the bug instantly. But obviously they didn't have such a test. So the question is: should I have had such a test in place?

My tests didn't pick up on the bug in an obvious fashion because my code is designed to be very tolerant to faults in the incoming data. (That's an absolute must in my line of work.) In particular, the bug only showed up this time because I was using the results of the broken function in a new and different way. In that case, the difference was startling -- one test run took 3 seconds before that change, and appeared to run forever afterward. (I think I let one test run go for over a week.)

How paranoid should your unit tests be? Should you test all the functions you use in a third-party library? What about the system libraries? What about your compiler?