Was talking to folks from a local agency (which shall remain nameless, but has initials) the other day, and the conversation came to development practices. I asked what kind of automated testing they were doing, and was met by a series of blank stares; their testing procedure amounted to “load it up in a browser and see what it looks like.”

At this point I probably had trouble keeping the incredulity off my own face. Now, I’ll admit I came to the automated testing party late, and pretty much only because Rails makes it so easy to get started in it. But, having finally made it to the party, I went off in search of automated PHP testing tools. I found PHPUnit, and dove right in.

And I gained a little sympathy for PHP testers.

While on the learning curve, I tweeted in frustration, and in one of those magic Internet moments Sebastian Bergmann responded, asking for ways to improve the beginner’s experience with PHPUnit. Now that I’ve climbed the worst part of the curve, and have put the carabiners and pitons away, I’m going to try and satisfy that request. I would request of my readers (of which there are few) that if they see something wrong with the way I’m using PHPUnit, they patiently explain to me why I’m wrong. While I’m more than convinced the tool is useful, I’m still learning how best to use this thing.

Let me first acknowledge that some of my problems arose from the fact I came to PHPUnit from Rails, and while the testing process is similar there, it’s not the same, and the similarities will cause confusion if you don’t recognize the differences.

The first thing I did was check out the command line details, to figure out how to run it. From those command line details I discovered that it can automatically create a test skeleton from an existing code file, and since I was trying to use it on some legacy code, I said “Terrific!” and did just that. My first mistake.

Now I had a test file for the code, containing one test for each method in the class file. That was new to me, but I went with it; my first inclination with new tools is that the toolmaker knows something I don’t, or I would have been the one writing the tool, so that must be the best way to test PHP applications. I think that was my second mistake. Tests don’t need to be segregated by method, in fact the method is almost irrelevant to a good test. A good test focuses on a single condition of inputs and outputs; a single class method may have dozens of tests written about it, depending upon the domains of the inputs and outputs.

I gave up too quickly on the manual, my third mistake. In all fairness, however, the manual encouraged that behavior. I know the principle is to start simple and work toward more complex examples, but the simple example was too simple to be of value, and the more useful examples were found several chapters later, with chapters on several other topics in between. This encouraged me to think the simple examples were all I was going to get, and to extend the confusion even farther, the form of the simple examples actually contradicted the form of the PHPUnit-generated test skeleton, because the simple examples followed the traditional scheme of different tests for different facets of the same function, rather than one test per method in the skeleton.

More confusion.

The rest of my learning curve consisted mainly of untangling the confusion and unlearning the things I thought I knew about PHPUnit. The tool is useful; I still don’t know how best to use it.

Suggestions:

Change the test skeleton output, or put words in the very first chapter on writing tests explaining good testing procedure, and acknowledging that the generated test skeletons don’t conform to it. The latter is probably simpler, as to get the skeleton right PHPUnit would have to understand what happens in the function it’s looking at.

Reorganize the manual. This is the biggest need. I wouldn’t have used the samples that were in chapter 4, for one thing. Instead it should start with a simple function someone might have written in a useful routine from real life. Everyone’s got some laying around — a class method that does only one small thing. Show that function, and then show writing tests for that function. Work upwards from such a simple function to one of moderate complexity, that maybe requires 6-10 tests with some of the tests having 2-3 assertions for the result.

That goes for chapter order as well. Instead of “writing tests” with very simple examples in chapter 4, with the more complex examples not showing up until several chapters later, get the writing tests together in one place. For example, the 4-page chapter on the test runner contains just a little more information than the command-line help will give you — that’s an appendix; putting it where it is simply distracts from the test writing. Instead, combine using the command-line runner with the writing of tests, showing the written test in action.

Expanding the test writing material will then lead more naturally into the test suite material, which should be revised and expanded, because frankly (and here I’ll display my own density in hope it makes things better for everyone else) I still don’t understand how to write a test suite, after reading the chapter 4 times!

The chapter on fixtures could be integrated into writing tests and test suites, since it goes with both procedures. The idea would be that early in the progression from simple to moderate complexity in the writing tests examples the idea of using a fixture for the input data would be introduced. Then the fixture would gain complexity with successive examples, just like the test in general does.

In general, the problem with the book’s organization is that it looks too much like bad corporate websites from the mid 1990’s. Back then corporations thought the right way to structure their website or intranet, for that matter, was along business group divisions: technical specs in the engineering area, features in the marketing area, etc. (Yes, I’ve seen those websites; to my shame I’ve even worked on one or two).

It seems like this was the way the book was organized: one chapter for each feature, when in reality the users are going to come to the book wanting to learn to write tests, and as an extension of that, write test suites. Nobody approaches PHPUnit with the idea “I’m just going to write a test fixture.” They’re going to write a test fixture because the test they’re writing needs or would benefit from one, and what they need to know before they learn how to write one is when to use one and how it makes the test easier to write. And that’s shown by including it in the test writing chapters.

I know I’m going to hear “If you can do it better, then shut up and do it!” That’s a sentiment that more than a bit daft, because while I might have several ideas on how to organize the material better, I don’t have the knowledge to write the material. As I’ve acknowledged before, I’m still figuring out how to use this tool. I’m gobbling up every stray bit of information I can from my own experiments, as well as Planet PHPUnit. I’m still a beginner; I don’t have nearly enough knowledge to write a book about it.

Yet.