Stream-of-Consciousness Testing

I just used the doctest module for the first time today, and I have to say, “Wow! Gosh! Gee!”

To date, PEAK incorporates some 878 automated tests, all but one of which are based on the unittest module, so I guess I’ve gotten a fair amount of experience with unittest. But my few hours today working with doctest have made me realize that unittest carries an immense amount of overhead baggage that came from its Java-based predecessor, JUnit. In a way, it now reminds me of some of the early versions of peak.naming, before I figured out how to reduce the complex Java “JNDI” API down to a clean “Pythonic” API. The thing is, what’s easy and sensible in Java (Javaic?) is not at all the same as what’s easy and sensible in Python.

Anyway, I’m rambling a bit, so here’s the deal. There was a module that I wrote an initial, unfinished draft of the other day, and I had also written a unittest-based test suite for it. This afternoon, I downloaded the Python 2.4 version of doctest (that has all the nifty Zope 3-inspired doctest+unittest integration features), backported it to Python 2.2, and rewrote the test suite for the module in question as a doctest text file.

Now, I have a top-notch piece of documentation for the module (peak.util.unittrace), complete with examples, in barely twice the number of lines of the unittest-based test suite alone, even though the doctest version tests more things, is more readable, and includes documentation!

And as if that wasn’t enough, as I added new features (after adding tests for them), I found that my programming went much more smoothly. Why? Because I was documenting my stream of consciousness. When I thought, “Well, the X needs to do Y before we can test Z”, I would write, “Before we test Z, we need to do Y to X,” followed by the setup code or whatever. Because the documentation part of a doctest needs no markup (like a docstring or comment), there’s no interruption to my train of thought; I just type. But, the desire to have a coherent document leads me to structure my thoughts, in outline and “story” form, continually “refactoring” the document to a cohesive whole.

As the “story” progresses, I take low-level implementation details and try out high-level assemblies of them, almost as if I were using the interpreter directly. Then, when an assembly works, I yank out the test code and stick it into the actual implementation, replacing it in the documentation with a call to the new, high-level method that I just implemented, but leaving the test output intact. In short, I was “refactoring” tests into implementation… while still keeping the tests!

When I was done adding functionality, I then “refactored” the document by grabbing the text that dealt with public APIs, and pushed them to the top of the document, leaving all of the low-level implementation tests under a heading called, “How It Works (subject to change!)”. Then, I spent a few minutes polishing the visual appearance of the document when formatted with reST.

The entire process was as seamless and fluid as I have ever experienced during test-driven development. More so, even. Ordinarily, I hesitate over every new test, trying to figure out how to translate from the result I want, to a test expressed in unittest-ese. With doctest, however, the testing just “disappeared” from perception. It was like I was just writing down my thoughts while playing with the interactive interpreter, only it was even better than that, because the interpreter and my notes were a single, continuous stream, and my work was saved in a file where I could edit and re-run it at will.

I’m always telling people about how Python just gets out of your way and lets you focus on the problem, but I didn’t even realize that it was unittest that was in my way before. Heck, I just thought unit testing was hard. I’ll bet that that’s what happens with Java programmers, too. They probably just think programming is hard. 🙂

On that note, I also thought documentation was hard. Not in the sense that writing any one document is difficult, but in the sense of ensuring that documentation always gets written, and that it stays correct and up-to-date as the software changes. Now I see that the obvious answer for that problem is the same as it is for testing: write the docs (and tests) first, and make them executable. In PEAK, we’d already dabbled with that principle in the peak.ddt “Document-Driven Testing” framework, but that system was intended for acceptance tests and communicating with non-developer project stakeholders. I realize now that doctest is the same idea, but applied to unit tests and developer documentation.

For more information on doctest, I suggest checking out this PyCon presentation by Jim Fulton. It makes the case for why Zope 3 development moved from unittest to doctest, and gives examples of how to use doctest. My initial impression of doctest via the Zope 3 sources was bad, because embedding doctests in docstrings appeared to lead to modules’ code being completely obscured by massive amounts of documentation and tests. But, now that I know how to incorporate tests into separate documentation files, while still integrating them with an existing unittest-based test framework, I’m pretty pumped about using doctest from here on out.

(P.S. I suggest using the doctest.ELLIPSIS option, as it makes it easier to do comparisons on partial output. See the Python library reference for details!)