In XML.com: Should Python and XML Coexist?, Uche Ogbuji liberally quotes from and analyzes two of my XML-v.-Python rants, and actually gets it completely right. Since at least one of those rants has been cited as meaning I think XML is the spawn of Satan, I'm glad Uche read closely enough to get the context and nuance, without projecting things into it that I didn't say. Kudos!

Well, we're about as prepared as we're going to get. Nothing to do but wait, and take advantage of the simple amenities like TV, air conditioning, and internet service while they last. I've been doing some OSAF work, and answering emails generally.

It looks like the storm isn't slowing down, which means it'll get here sooner but be weaker. The NHC wind forecast from 11am shows only about a 50% chance it'll be a hurricane this evening, and I believe that was when they were still expecting it to slow down and be offshore longer.

(Hurricanes, in case you didn't know, are kind of a solar-powered steam engine. They're powered by warm water, so being on land hurts them, although obviously not as much as it hurts us!)

It's funny how the mind sometimes doesn't really start working until you take "your" mind off of things. I repeatedly had this experience yesterday. For example, I was out in the garage looking for the power cord for my portable mini-refridgerator, as Leslie wanted to use it in her office at the store. (This was before we knew about the hurricane situation.) Anyway, I was looking high and low for that darn thing. Every now and then, I would try to do my Jedi mind trick, but it just wasn't working. (In fact, it hasn't worked that often since the first time I discovered it.) Finally, I took a deep breath and relaxed, emptying my mind as completely as I could, and sort of "let myself be".

At that precise moment, my eyes fell upon a box that was partially hidden under a pile of wood. So I opened the box, and there were the cables that I'd spent the last 15 minutes tearing up the place for. I get the impression that when you're sitting there in your conscious mind going, "Do the trick, find me the cables, yeah, do it now," the subconscious mind can't hear itself think.

Actually, I think it's more like this... when you focus consciously on a thing, the metadata you're tagging it with is more like "focus on this", not "do this". And those are not really the same thing! To "just act", you need to quiet and empty the conscious mind a bit, to sort of take the consciousness filter offline. While the conscious mind is "in the loop", the system's bandwidth is more constrained. I suspect that one of the most valuable spinoffs of "bompu" (secular) Zen is practice at clearing the conscious mind. Maybe hypnosis works by letting the hypnotist bypass that nosy consciousness filter and have a chat directly with the subconscious, who knows?

Anyway, I had the same experience with a bunch of other things yesterday, just not so much in the physical looking-for-stuff sense. For the last few days I'd been chewing on the problem of designing a configuration syntax for WSGI deployment, trying to get something that in the simplest case could specify an application to find, download, and deploy with maybe one line of configuration text, something like "someapp from SomeProject" as a minimal "hello world" sort of configuration for some WSGI application.

I'd been bouncing some ideas off of Ian Bicking on the Web-SIG list, and made some progress, but the whole thing still seemed too complicated to me, and there were lots of annoying little parsing ambiguities in the grammar I had come up with. The whole thing also seemed way too WSGI-specific in some ways, even though the syntax seemed to have potential for doing other kinds of configuration and component assembly, from logger configuration to GUI widget hiearchies, or to create aXML-free ZCML or schema-free ZConfig variants.

So, having spent many a spare hour struggling consciously with the design, I of course solved the essential problem yesterday while my conscious mind was distracted by worrying about the hurricane. The light that came on in my head as I was getting in the car to drive to the grocery store was blinding, so much so that I was tempted to go back inside and try out the idea. But luckily, sanity prevailed. :)

I've dubbed the idea SCALE, which stands for Syntax for Configuration and Language Extensions. Thus, files created using it can be thought of as "Python scales", which is a nice double entendre suggesting both the skin of a snake, and the idea that Python is a scalable language. And if we apply it usefully in the web field, we can perhaps say that the alternative to Ruby On Rails is "Python that Scales". ;)

Seriously, though, the concept is more or less a user-defined statement facility, implemented via runtime execution of "scope" objects (similar to PEP 343, but different). There are some very small similarties to Logix, except that SCALE is actually for the most part a subset of Python rather than a superset. At a lower level of the SCALE system, it'll be possible to have scopes in which the language isn't actually Python any more, but that will be a non-trivial thing to implement, which should help to keep most people from doing it. :)

SCALE's main purpose is to make it easy to textually specify some kind of hierarchical object structure, especially ones that may involve cross-references or need to be able to refer to an arbitrarily large assortment of potential object types. Thus, I don't see a point to enabling widespread mini-language proliferation of the sort that Logix supports. While Logix turns all Python statements into expressions, SCALE simply does away with non-expression statements altogether, but allows any expression to have a nested block beneath it. There are a couple of other little twists on the basic syntax for dealing with assignment and context, but basically, SCALE is Python assignments or expressions, with the ability to have an indented block under a given assignment or expression. This means that creating nested components and their configurations are easy. Each block has its own scope, nested lexically as in Python functions.

The execution model is rather like SAX parsing for XML; each scope is an object that receives events describing the expressions and

[3 hour hiatus as the power goes out; my UPS kept this PC going long enough for me to "Save as Draft" to blogger's servers.]

Wow, I'm absolutely amazed that our power is back already, since when I called FPL they gave me a spiel (the same as they're repeating almost hourly on every TV and radio station) about how, once the winds die down, they'll go out and assess things, and then 24 hours later tell everyone when they might get power back. So, the fact that it's back on (at least for now) 3 hours after it went out is very nice. During the intervening time, I thought of lots of interesting things to talk about in this post, none of which I now remember. Oh well. I'm going to go ahead and publish this now, before any chance of the power going back off again. We're still in a zone with a high probability of tropical storm force winds. The counterbalance is that we're very near to both a fire station and an FPL regional staging area for hurricane response, which means we're frequently among the first to get power restored during these things. Still, the lights are flickering every now and then as I type. I'd like to think we're past the worst of it, but the hurricane's stronger eastern side is still coming on shore.

It's interesting to look at the Ruby Gems project documentation, as it's amazingly similar in some ways to Python Eggs. Well, similar to what I want eggs to be, anyway. Eggs have got an awful lot of Python-oriented and plugin-oriented features that gems don't appear to, but gems are a lot more, um, polished. :) More specifically, they've already got their basic code signing approach worked out, along with tools to list installed packages, uninstall packages, and they even have a built-in webserver to let you share your local packages. Which is wild, but cool. It's sort of like if PyPI were to work by scanning eggs in its local filesystem in order to serve a package index. I imagine that might make it rather slow to start up, but at least with eggs you have nearly all the useful information in the directory entries and don't have to open the files. So, I imagine you could actually get a pretty good index going that way.

A webserver isn't the top priority for eggs at the moment, though. The 0.6 development version in CVS is going through some growing pains, like the need to finish a basic manual for the pkg_resources module, and to add some more explanatory messages to ez_setup and setuptools about the downloading process. There are also some discussions in progress about the "test" command, and supporting "py.test". I think the unfortunate thing about py.test is that it doesn't extend Python's unittest module.

unittest has gotten something of a bad rap, I think. Regardless of whether you like its basic testing facilities or not, it is an extremely good framework. In fact, I think it's one of the most beautiful frameworks in the Python standard library. Its functionality is cleanly separated into four roles, each of which can be filled by any object implementing the right interface: runners, loaders, cases, and results. Because of this exceptionally clean factoring, the basic framework is amazingly extensible. But a lot of people don't realize it, and so they create competing, incompatible frameworks like twisted.trial and py.test.

I'm not saying these other frameworks are bad, it's just that the additional functionality could usually be added by implementing a replacement loader, runner, case, or result class, depending on what kind of features you want to add. For example, all of py.test's many advertised features could be cleanly implemented as backward-compatible extensions to the unittest framework, that would then run under other test runners like unittestgui, cleanly integrate with the existing setuptools "test" command, etc.

People gripe about stuff like finding and running tests with unittest, but I think that's because they don't know about extending it, or perhaps how to use it properly in the first place. Earlier this year, I wrote a simple 20-line "loader" for unittest that scans subpackages for modules whose names started with "Test". And the unittest.main() function lets you specify what loader you want to use. So, by passing a module or package name on the command line, I can recursively scan all packages for tests, and I didn't have to write a whole new test framework to do it.

I gather, from the hype around various unittest replacements, that this is considered a big deal. Unfortunately, it seems like nobody realizes how easy it is to extend unittest to do these things.

So, a brief tutorial is in order, I think. A "runner" is the top-level thing that runs tests and reports on the results. A "results" object records the success or failure of individual tests as they are executed, possibly reporting on the progress as it occurs; usually it is created by the runner and is specific to the runner. For example, there's a TextTestResult class that outputs the dots or "ok" messages, and a similar class that updates the progress bar in the GUI version. A "loader" finds and accumulates cases to be run into a TestSuite. (A TestSuite is technically a "case" too; but you'll probably never need to subclass it unless you want to do something fancy like implement py.test-style incremental gathering instead of the default gather-then-test behavior.) The default loader can scan a module for test case classes, or run a function returning a case or suite of cases, among other things.

Contrary to apparent popular belief, it is not necessary for you to subclass any particular unittest implementations of any of these ideas. (The default loader uses isinstance() and issubclass() to identify test cases in a module, but this is easily changed in a custom loader if needed.) A "case" object need only implement __call__(result), __str__(), and shortDescription() to be fully compatible with the runner. The __call__ method should call result.startTest(case), result.stopTest(case), and in between call one of result.addSuccess(case), result.addError(case, sys.exc_info()), or result.addFailure(case, sys.exc_info()), as appropriate. The rest is up to the case to manage.

The only methods you need to implement a custom loader are "loadTestsFromName" and "loadTestsFromNames". However, if you subclass the default loader class (unittest.TestLoader) you can selectively override various aspects of its functionality. For my recursive scanning loader, I just overrode the "loadTestsFromModule" method, so that it checked whether the module was a package, and then also searched for subpackages, and any contained modules whose name began with "Test". This was easier than writing a completely custom loader, but if I wanted to create a py.test-like test finding algorithm, I'd definitely start out the same way, by subclassing TestLoader and adding new functionality. That's because creating a custom script to run unittests using a custom loader is just a couple of lines: "from unittest import main; main(module=None, testLoader=MyCustomLoader())". That's not so hard, is it?

If I also wanted to change the progress reporting, I'd create a new "result" class, and then subclass TextTestRunner, overriding _makeResult() to return an instance of my altered result class. unittest.main() also accepts a "testRunner" keyword argument, so that's maybe another line to add to my script.

In short, there's very little reason to create a whole new testing framework; the one we have is just fine, even if its default features may not be to everybody's liking. But it's a framework, which means you're supposed to put something in it. Because it's a standard framework, we have the opportunity to let all our testing tools work together, instead of forcing people to jury rig their tests together. The doctest module in Python 2.4 provides APIs to create unittest-compatible test cases from doctests, which means that I can use doctest in conjunction with my existing 800+ unittest-based test cases. I can't do that with py.test or twisted.trial or any other from-scratch framework that discards the superlative design of the unittest module. unittest may be a mediocre tool, but it's an excellent framework that would allow us to all develop tools that work together, much like WSGI enables interoperability between web applications and web servers. unittest deserves a lot more recognition than it gets, as it could be just the thing to stop us ending up with as many mutually-incompatible testing frameworks as we have mutually-incompatible web frameworks.

Well, it's finally official. Chandler's parcel.xml format is now deprecated and will soon be gone altogether, replaced entirely by simple Python APIs. Some of you may be thinking back to my Python Is Not Java rant, in which I said that using XML for core application functionality like this was, well, unwise. :) At the PyCon Chandler sprint, it was discovered that the Chandler's homegrown XML schema definition language was a terrible hardship on developers, and so I proposed to replace it with a descriptor-based Python API. That migration was completed recently. With that done, only initialization of data items (such as Chandler's UI components) was done using XML. So, a few weeks ago, I implemented an experimental API for initializing data items, which quickly became quite popular, with some even pointing out the advantages of being able to factor out repetition.

For a while, there was also a proposal to create a new XML format just for UI definition. But my counterproposal for using a simple template class and a classmethod instead was met with great rejoicing.

Many people misunderstood and/or misrepresented my previous position on XML; the case of Chandler should help to clarify it. Chandler still uses XML for WebDAV, for .xrc files, for sharing, and numerous other use cases where it makes at least some sense to do so. The parcel.xml format, however, was pure excise: a verbose additional language to do things that are more cleanly (and efficiently) done in Python code. It was developed to serve a vision of Chandler as a "data-driven" system, and it was supposed to ultimately support things like GUI editors.

Of course, the real sin here was not so much XML per se, as overengineering in advance of requirements. If you're not developing the feature now, it's best not to make a bunch of other design decisions based on what you think the feature will need. A little thing like choosing to put data in XML form can result in a wide variety of additional costs like:

• Designing the XML format
• Implementing a parser
• Documenting the format
• Developing a bunch of stuff in the format
• Evolving and fixing the parser to handle more and more complex use cases that weren't thought of previously
• Productivity losses versus what it would've been with Python
  
• Converting all the data once you decide it was a bad idea, or else paying the ongoing marketing and education costs to get third-party developers over the hump, or the cost of not getting those developers on board

The cost of adding things you don't need is really, really high. Luckily, OSAF believes that it's more important to get things right, than it is to keep throwing money down a rathole to justify the money already spent. I've certainly worked for organizations where the reverse is true, though, including one that threw away tens of millions of dollars trying to replace a small, well-designed Python application with an expensive piece of "enterprise" crapware. Ah, the things I could've done with that budget! Well, probably I just would've given everybody raises and maybe hired a few more people. Or maybe spun off my group as a company that would sell the software to other companies. Heck, we could've used it to buy free sodas for life for everybody working in the company and got more value for the investors than what was actually done with the money!

But I digress. The point is this: delaying feature investments good, sunk cost fallacy bad. Any questions?

My patch to implement PEP 342 is now in the Python CVS trunk. It probably could've gotten there sooner, but I've been terribly busy. I decided to go ahead and check it in now, rather than waiting until I'd polished every possible thing, so there are a few loose ends, not the least of which is documentation. Tests are there, though. If you're the sort of person who builds your own Python versions from CVS, then enjoy!

(In case you didn't read my earlier post on PEP 342, PEP 342 enhances the semantics of Python's generators to make them more coroutine-like, with many spin-off benefits for asynchronous I/O, simulations, and that sort of thing.)