Now With Grammar And Tests

I've made a huge amount of progress with the ABC project in the last 36 hours. At this point I think I've just got a few more rules to write and debug before we are able to completely parse the sample ABC tune I posted several days ago. (Naturally they'll be the trickiest, I imagine.)

grammar ABC
{
    regex header_field_name { \w }
    regex header_field_data { \N* }
    regex header_field { ^^ <header_field_name> ':' \s* <header_field_data> $$ }
    regex header { [<header_field> \n]+ }

    regex basenote { <[a..g]+[A..G]> }
    regex octave { \'+ | \,+ }
    regex accidental { '^' | '^^' | '_' | '__' | '=' }
    regex pitch { <accidental>? <basenote> <octave>? }

    regex tie { '-' }
    regex note_length { [\d* ['/' \d*]? ] | '/' }
    regex note { <pitch> <note_length>? <tie>? }
    
    regex rest_type { <[x..z]> }
    regex rest { <rest_type> <note_length>? }
    
    regex gracing { '+' <alpha>+ '+' }
    
    regex broken_rhythm_bracket { ['<'+ | '>'+] }
    regex broken_rhythm { <note> <g1=gracing>* <broken_rhythm_bracket> <g2=gracing>* <note> }
    
    regex element { <note> | <broken_rhythm> | <rest> | <gracing> }
    
    regex barline { '|' | ':|' | '|:' | ':|:' | '::' }
    
    regex line_of_music { <barline> | [<barline>? <element>+ [<barline> <element>+]* <barline>?] }
}

Much, much nicer than just having "abc_" at the beginning of every regex name. And wow, compared to any other parsing tool I've ever used, this is really, really easy. This comes very close to matching the ABC BNF, though I've simplified a lot, and changed !trill! to +trill+ (etc) to match the version of ABC present in this file.

So far the only downside I've found is that it is ugly to test:

{
    my $match = "d'+p+<<<+accent+_B" ~~ m/ <ABC::broken_rhythm> /;
    isa_ok $match, Match, '"d+p+<<<+accent+_B" is a broken rhythm';
    is $match<ABC::broken_rhythm><note>[0]<pitch><basenote>, "d", 'first note is d';
    is $match<ABC::broken_rhythm><note>[0]<pitch><octave>, "'", 'first note has an octave tick';
    is $match<ABC::broken_rhythm><note>[0]<pitch><accidental>, "", 'first note has no accidental';
    is $match<ABC::broken_rhythm><note>[0]<note_length>, "", 'first note has no length';
    is $match<ABC::broken_rhythm><g1>[0], "+p+", 'first gracing is +p+';
    is $match<ABC::broken_rhythm><broken_rhythm_bracket>, "<<<", 'angle is <<<';
    is $match<ABC::broken_rhythm><g2>[0], "+accent+", 'second gracing is +accent+';
    is $match<ABC::broken_rhythm><note>[1]<pitch><basenote>, "B", 'second note is B';
    is $match<ABC::broken_rhythm><note>[1]<pitch><octave>, "", 'second note has no octave';
    is $match<ABC::broken_rhythm><note>[1]<pitch><accidental>, "_", 'second note is flat';
    is $match<ABC::broken_rhythm><note>[1]<note_length>, "", 'second note has no length';
}

On the plus side, this does show how to get at the parsed bits. On the downside, it's not really good at testing what is not present in the match, and it seems like any refactoring to the grammar will lead to massive changes in the tests. I'm guessing there will be a better way of testing this in the future... or there already is and I just don't know about it.

At this point, it seems to me the biggest obstacle is figuring out how to formulate line_of_music so that it actually returns its results in a usable matter. The thing is, the interleaved order of the barlines and the elements is very important to make sense of the music. The way I'm doing it now will return an array of barlines and an array of elements, with no idea how those two arrays interact....

Ack: Forgot to include mention of the word Perl here so it would get picked up by Ironman.

Regexes are Wow

I've just spent the last little bit playing around with adding more regexes, and I think I'm in love. It feels both easier to use than Perl 5 regular expressions and vastly more powerful. Here's the latest batch I just added:

regex abc_basenote { <[a..g]+[A..G]> }
regex abc_octave { \'+ | \,+ }
regex abc_accidental { '^' | '^^' | '_' | '__' | '=' }
regex abc_pitch { <abc_accidental>? <abc_basenote> <abc_octave>? }

regex abc_tie { '-' }
regex abc_note_length { [\d* ['/' \d*] ] | '/' }
regex abc_note { <abc_pitch> <abc_note_length>? <abc_tie>? }

The only difficulty I had here was not realizing at first that abc_basenote needed angle brackets around the character class specification. That caused me a few minutes of confusion, because I was trying to jump to (informally) testing more complex things built up around it, and so it wasn't immediately obvious that the base note regex was simply not working.

But hey, I know how that works now, and wow! The rest of it was dead easy and expressive to boot. And using .perl to dump the match structure is awesome.

There are two things to ponder here, however. First, having abc_ at the beginning of every regex is an obvious wart. I think I can get around that by putting these in a grammar? Must investigate.

The second is that at this point, I've got enough structure to what I'm doing that I really need a test suite. I suppose if I package the regexes in a grammar, then I can use it from another file, and test that? Must experiment.

Getting Started with ABC

So, in classic style, I've started off this project by just flailing around a bit. My first thought was to try TDD. But I very quickly realized I didn't have the first clue what the interface being tested should look like. So I dashed off the roughest idea for some classes. But I don't know how they are best filled. So I am reversing course, and will start by playing around with regexes a bit, trying to learn how the new facilities in Perl 6 work.

So my real starting point is pulling up The Book and reading up on regexes. No pretense I understand what's going on, let me just document what happens as I go along. BTW, this project is at github.

So, my first attempt looked like this:

use v6;

my $abc = q¬´X:64
T:Cuckold Come Out o' the Amrey
S:Northumbrian Minstrelsy
M:4/4
L:1/8
K:D
A/B/c/A/ +trill+c>d e>deg | GG +trill+B>c d/B/A/G/ B/c/d/B/ |
A/B/c/A/ c>d e>deg | dB/A/ gB +trill+A2 +trill+e2 ::
g>ecg ec e/f/g/e/ | d/c/B/A/ Gd BG B/c/d/B/ | 
g/f/e/d/ c/d/e/f/ gc e/f/g/e/ | dB/A/ gB +trill+A2 +trill+e2 :|»;

regex abc-header-field { ^^ \w ':' .* $$ }
regex abc-header { <abc-header-field>+ }

if $abc ~~ m/ <abc-header-field> /
{
    say $<abc-header-field>;

Note how I'm trying to get with the program and use dashes in my Perl 6 identifiers! Unfortunately, this doesn't actually work in this case:

Confused at line 15, near "abc-header"
in Main (file , line )

I played around a bit, and finally ended up changing the dashes to underscores. Changed that way it runs, but say $<abc_header_field>; prints out the entire string rather than just the first header field I had expected.

Aha! Apparently the .* $$ sequence in abc_header_field doesn't give you every up to the end of the line, as I'd expected. Instead it matches all characters (including newlines) up until it hits the end of the file, then backtracks until it finds the last end-of-line. Good to know. Changing it to \N* $$ works nicely.

Now I tried making it match abc_header instead of abc_header_field. No joy, we still just get the first header line. Why? Because we aren't matching the newlines between header lines! So tweak abc_header to account for the newlines, and bingo!

regex abc_header_field { ^^ \w ':' \N* $$ }
regex abc_header { [<abc_header_field> \n]+ }

if $abc ~~ m/ <abc_header> /
{
    for $<abc_header><abc_header_field> -> $line
    {
        say "header: $line";
    }

And here are the new results:

header: X:64
header: T:Cuckold Come Out o' the Amrey
header: S:Northumbrian Minstrelsy
header: M:4/4
header: L:1/8
header: K:D

Wow. After a couple of false starts, I've got some simple, powerful code there. I think I like these regexes...

Updated: Ugh, some weird formatting issues with the Perl 6 syntax highlighter. Too tired to mess around with now. Rest assured that there is a closing curly bracket in each example code.

Intro to ABC

So, the ABC format is a simple ASCII format designed primarily to make it easy to work with single line musical notation. It's great for, say, traditional dance tunes, like this tune from Northumberland.

X:64
T:Cuckold Come Out o' the Amrey
S:Northumbrian Minstrelsy
M:4/4
L:1/8
K:D
A/B/c/A/ +trill+c>d e>deg | GG +trill+B>c d/B/A/G/ B/c/d/B/ |
A/B/c/A/ c>d e>deg | dB/A/ gB +trill+A2 +trill+e2 ::
g>ecg ec e/f/g/e/ | d/c/B/A/ Gd BG B/c/d/B/ | 
g/f/e/d/ c/d/e/f/ gc e/f/g/e/ | dB/A/ gB +trill+A2 +trill+e2 :|

My favorite ABC program renders that like this (click on it for a high resolution version):


So, my goal here is to write a Perl 6 module that can read and understand this format, maybe do some light processing, and output it again. I'm hoping a grammar will make this pretty easy and straightforward, but I'm fairly ignorant of them so far. It will be an adventure!

I guess my first step is to set up a class to represent a tune, and then write some tests for it. Next time...

NUBS and Polynomials, with graphic

As has happened before, while putting the finishing touches on the code for this post, I suddenly realized I needed to rewrite the code before I could be happy with it. So this post will be devoid of Perl 6 code, and just discuss what is going on in the (now-improved) graphic.

The basic idea I'm illustrating here is that a NUBS curve is basically a convenient way of merging a series of partial polynomial curves. In this graphic, the NUBS curve is in black. The polynomials are in red, green, and blue.



If you start where the red doesn't overlap the black, and then follow it to the black and follow the black beyond that, you can trace the curve and get an idea of what is going on. Basically, the black NUBS curve follows one section of the red polynomial, then smoothly switches to a section of the green polynomial, and finally switches to a section of the blue polynomial. It is piecewise polynomial, in other words, with smooth transitions. (It is possible to generate unsmooth transitions too with a NUBS, or even out-and-out discontinuities, but generally this is not wanted.)

The other great thing about NUBS is how natural they are to specify, because their control points conform roughly to the shape of the curve, and modifying them changes the curve in a fairly natural fashion. I don't have time to go into great depth here, but here's a quick comparison. First, here's the code to specify the curve in the picture:

my @control_points = (Vector.new(-1, -2),
                      Vector.new(1, 0),
                      Vector.new(1, 1),
                      Vector.new(0, 1),
                      Vector.new(1, 2),
                      Vector.new(1, 2),
                      Vector.new(1, 2));
my @knots = (-1, -1, -1, -1, 1, 2, 2, 3, 3, 3, 3);
my Nubs $nubs = Nubs.new(3, KnotVector.new(@knots), @control_points);

Then here's the red polynomial:

(0.194444444444444, 0.111111111111111) x^3 
+ (-0.916666666666667, -0.666666666666667) x^2 
+ (0.583333333333333, 1.33333333333333) x^1 
+ (0.694444444444444, 0.111111111111111) x^0

Actually, that's a fairly clean polynomial in this case, but even so, it's hard to work with. The bit we're interested in is parameterized from -1 to 1, so for instance, to find the starting point we need to evaluate the polynomial at x = -1:

- (0.194444444444444, 0.111111111111111)
+ (-0.916666666666667, -0.666666666666667) 
- (0.583333333333333, 1.33333333333333) 
+ (0.694444444444444, 0.111111111111111)
= (-1, -2)

Whereas the starting point of the NUBS version is just the first control point.

Ack. I could go on, but I fear I've exhausted the patience of the Perl readers out there. And I feel like I'm constantly butting up against the limits of Rakudo with this project, so I think I'll put it aside for a few months and tackle something else. Maybe try to whip up a grammar for ABC files, or something like that. Then return to Vector when after Rakudo has had a chance to mature a bit more...

Ack in TextMate

For a while now I've known about ack, the slick grep substitute written in Perl. And I've known that the multi-file search function in TextMate was lousy, slow and memory-eating.

What I didn't know was that there was an Ack bundle for TextMate. It is a dream come true as far as I'm concerned. Okay, I've only briefly played with it, but it's screamingly fast and produces good-looking results. If there's a better add-on for TextMate, I don't know about it.

Good Thing Perl is Dead

PerlPilot comments that the "Perl is dead" meme annoys him. Personally, considering the apparent exponential growth in Perl-related Advent calendars this year, I think it's a good thing Perl is dead. If this is dead, a living Perl community would probably overload the Internet with Advent calendar traffic...