Until I saw this neat traceback from the innards of reddit, the success of WSGI and eggs was a lot more abstract.  But now, it appears they've got some street cred (along with Pylons, Paste, Beaker, and flup, it looks like).

OTOH, since reddit was first built with Lisp and then rebuilt with web.py, (making this version 3), and since now they've gone all mainstream and "sold out" to Conde Nast, maybe that means my work is "corporate" and "enterprisey", now, instead.  ;-)

Every so often, I see a new package listed on the Python Package Index that claims -- in its tag line, no less -- that it offers some feature "without external dependencies".

The authors of such packages must think it's a truly valuable feature, to list it in the tag line ("description" field of their setup script).

But I'm a little confused by why they're uploading it to the Cheeseshop.  After all, anybody who believes that a lack of dependencies is good...  isn't going to be able to use it!  ;-)

And anyone who doesn't care that much about dependencies (perhaps because they use setuptools?) was probably already using whatever package the new package is intended to replace.

Sure, if the package also offers some other features, a better API, or something like that, it gives people more choices, encourages competition and all that other good stuff.  And I certainly don't want to discourage anyone from uploading useful things to PyPI.

I'm just puzzled by the idea of advertising that your package has no external dependencies, when, as far as anybody else is conerned, it is an external dependency.  Seems like that space could be better used to promote whatever other benefits the library brings.

Just one of those "things that make you go hmmm" I suppose.

I don't usually like to blog about my client work, but in this case I'll make an exception, since so many other people are blogging about it who don't have any idea what they're talking about.

A lot of these people, it seems, think that the Chandler project is dead, dying, or a "failure" of some kind.  And that's simply not true.

Could the project have delivered more, in less time?  Sure, absolutely.

Does it have anything to do with the tools used?  Absolutely not.

Is the project dead?  Not at all.

Did it achieve its original goals?  No.

Is it likely to achieve its original goals?  I'm inclined to say no.

Does that matter?  Hardly!

Why?  Because in addition to creating a nice desktop application that:

• does online/offline calendars with overlays, recurrence and timezone support, date parsing, etc.
• has an extensible sharing framework that allows peer-to-peer or server sync with a variety of protocols, including CalDAV and gData
• allows plugins to extend the data model and include that data in peer and server sharing
• has an Eclipse-like plugin model allowing extenders and extendees with sophisticated UI extensibility

and a nice CalDAV server with an AJAX UI, the Chandler project has also funded (through paid developer time) a lot of open source libraries and tools, especially for Python.  Here's a partial list of other projects that are or were funded by Chandler, in whole or in part:

• setuptools and EasyInstall
• Windmill
• PyLucene
• PyICU
• jcc
• zanshin
• simplegeneric
• vobject
• Trellis
• PEP 342 and related features of Python 2.5
• wxPython
• m2crypto

I'm sure I've missed some, even among the things that I personally worked on.

Anyway, the point is that between the client, server, and libraries, the Chandler project has produced quite a lot of working code, all of which would remain useful and beneficial to the community,  even if the organization had already been disbanded.

But it hasn't disbanded.  On Tuesday I'll be in San Francisco, meeting with the other members of the team, as we hash out our strategy for moving forward.  It is certainly possible that we'll decide to just wrap up the outstanding work on bug fixes, quality improvement, and so on for the existing packages, and not try to continue past the end of this year.  It's also possible that we'll decide to push forward with new feature work, or focus on integrating our cool calendar/plugin platform with other open source applications.  Really, there are quite a few possibilities there.

What a lot of people outside OSAF don't get about this is that the re-org is a good thing.  Don't get me wrong - it's not so good for the people who got laid off.  But for the project, it was a godsend.

See, out of all the junk that people have been writing for years and years about the project, almost nobody has actually seen what the real problem is, why Chandler didn't get anywhere near the original, highly-ambitious goals.

Sure, people have pointed fingers at lots of things, and the idea that there was plenty of time and resources with no hard deadlines is often brought up as a culprit.  But that's not quite right, either.

Having worked on, in, and around the project for about three years now, I can tell you quite simply what the problem was, and why the re-org fixes that problem.

There was no objective basis for decision-making.

It's that simple, really.  Without an objective basis, there was no way to argue from anything except opinion, with nobody's opinion being more important than anyone else's.  There was no benevolent dictator but Mitch, and Mitch had already stopped being available day-to-day before I even started working for them.  (And I've heard that even when he was the benevolent dictator, he was perhaps sometimes a bit too benevolent -- i.e., inclined to just let people choose their own direction/vision for what the project was going to be.)

Thus, there was no unified design, architecture, vision, nothing.  We had fiefdoms, not because anybody wanted to shut anybody else out, but because the natural response of a good developer faced with chaos is to find a way to organize the part that he or she can deal with.  It was easier for each person to just go and focus on the things he or she cared about, than to try to build consensus in the absence of an objective idea of what "success" was supposed be.

Now, you can point to inadequate specification, lack of constraints, and all sorts of other contributing factors as to why there was no objective criteria for success.  But to me, those aren't really central.  You could have every single one of those things correct, for example, and still find some other way to create a culture that lacks objective criteria!

And it's the lack of these common, objective criteria that does you in, regardless of why the criteria are lacking.  Without them, you can't really have productive discussions or planning, whenever the necessary action crosses organizational boundaries.  (Since different sub-groups will have their own views and criteria, with no common criteria to sync against.)

Anyway, next week, we're actually going to sit down and work on defining some objective criteria for the Chandler project as a whole, going forward.  Those criteria may not be what Mitch originally had in mind, and they may be considerably less ambitious.  But, my sincere hope is that they will be sufficiently objective, to allow us a chance at achieving them this year.

So, if and when the project is really dead, we'll certainly say so.  In the meantime, IMO, the obituaries are more than a little premature.  We're only pinin' for some criteria, you see.  ;-)

CompUSA closing?  It's a bit hard to believe it.  I remember going there back in the 80's, for goodness sakes.  Bought lots of books and software there, not to mention keyboards, mice, memory, cases, motherboards, you name it.  They've been around for about as long as I've been a programmer.  Now where am I going to buy motherboards when the old one dies on a Saturday morning?

True, I've been buying most of my peripherals from Circuit City these days, but they don't carry very many motherboards, and none of them in stores.  It seems that computer shows are a dying breed these days, too.

I guess there are too many places to buy assembled computers these days, and the gamers and other people who like to customize buy their parts online.  Ah well.  A cup of auld lang syne, CompUSA, we knew thee well.

The other day I was helping my wife debug a report she was trying to set up in her store operations software, and there was something screwy about the totals.

She was trying to set up an inventory change report that would reflect the changes of inventory levels at the store, for items that are also listed on her website, so she could update the website when items sold out or came back in.

For some reason, though, only the inventory arrivals were showing, not the sales.  I went in and changed the selection criteria to only select the sales, and that worked.  I put back in the other part of the query, something like "(arrived>0 or sales>0)", and then only the arrivals showed.

Frowning, I took out the "arrived>0" part again.  The sales showed up.  Then, just for the heck of it, I changed the query to "(sales>0 or arrived>0)", and only the sales showed up.

"This is weird," I said.  "You know, I'll bet NULLs are probably the problem somehow."

And she said, "What makes you say that?"

I laughed.  "Because with SQL, NULLs are nearly always the problem."

(Postscript: I'll leave it as an exercise for the reader to figure out how they were the problem.  It'll be good for your SQL chops.)

As my self-help business has grown, I've been needing to ship dozens of newsletters and CDs to my members each month, not counting incidental orders for other products.  Last year, I did all my shipping manually through the USPS website, but as soon as I started offering subscriptions, I switched to using Endicia.com, as previously recommended by Joel on Software.

Endicia offers a Windows client that lets you copy and paste a slew of addresses from the clipboard, and print shipping labels to a Zebra label printer (or pretty much any other kind of printer).  And since the May 2007 rate changes, I can now print customs forms right on the shipping label for shipping to every one of the countries my subscribers are in.  I don't even have to hand-sign and date the customs forms any more!

However, one of the most annoying things about using the client manually is that it has to be carefully configured before printing each kind of label, and it cannot print more than one non-US label at a time.  Almost exactly half of my subscribers are outside the US, so it literally takes me hours to copy-and-paste their addresses one at a time, double-check all rate options, and print the labels.

So this month I got fed up with that process and wrote a Python library to interface with Endicia via XML: PyDicia.

PyDicia is an industrial-strength (well, light industrial, anyway!) interface to Endicia's DAZzle client for Windows.  It can not only send arbitrary packages to anywhere in the world with any shipping option or label type supported by Endicia and the USPS, it can retrieve address corrections, delivery confirmation numbers, customs IDs, etc.  If you have application objects like a "Customer" or "Invoice", you can register callbacks to turn them into address data that PyDicia understands, and to receive the address corrections, delivery confirmation numbers, etc.  (e.g. So you can mark an order "shipped" in your database.)

In my current actual use so far, I've only printed out labels from a script, and haven't done any application integration yet.  That's because my "customer database" currently consists of an Excel spreadsheet and a plain text file of addresses.  I'll probably replace all that with an Access database soon and implement business rules to figure out what should go into each person's package(s).  (Eventually, the Access database wouldl probably move to some sort of server-side database, but it's a YAGNI for now.)

However, I've already saved a couple of hours of data entry...  at the cost of two or three days programming time.  :-(  Oh well, so it'll take a few more mailings (or a handful more members!) before the time savings really starts showing.  But on the flip side, I had fun writing it, and now I don't dread the process of doing the international shipments as much.  Even more important, I no longer dread the idea of getting a lot of new members, which has been causing me to avoid promoting the group as effectively as I should!

A few tips for using PyDicia, by the way: be sure to set up your DAZzle layouts first, including saving the printer setup with each layout and enabling "stealth" postage.  This will prevent constant prompting to enable stealth postage for international shipments, and avoid device errors or badly printed labels due to incorrect printer information.  That way, you can disable the prompts and just spew labels out at lightning speed.  Yay!

Every so often, I get inquiries from people about the status of PEAK -- the Python Enterprise Application Kit.  Progress on the core has been glacially slow in the last few years, because I no longer work in an "enterprise" shop.

But nestled within the big PEAK framework are a lot of little frameworks struggling to get out.  So this weekend, I posted a status report on these and future frameworks to the PEAK mailing list.  So, if you've been wondering what's been going on with RuleDispatch, PyProtocols, PEAK-Rules (whose core design was the basis for my initial version of PEP 3124), or any other pacakge of mine, whether on the Cheeseshop or off, you might want to have a look.

Yesterday was the final deadline for Python Enhancement Proposals (PEPs) for Python 3.0 .  In theory, this was the date that PEPs were to be reviewed by, but in practice, lots of people (myself included) were submitting new PEPs at the last minute, to make sure they got in.

Read more about them here.

2nd Update: I'm now using a FreeSwitch-based solution.  The command-line interface is crude, but will work for now for smaller conferences.  And it supports Google Talk, as well as most SIP phones and of course calling in from regular phone lines.

Update: Some kind souls have pointed out that Asterisk's default conference bridge needs some additional hardware, and thus my dedicated server idea won't work.  There's another conferencing plugin for Asterisk, but it doesn't have the web features. 

This is just the kind of information I was looking for; thank you very much!  One person also suggested checking out FreeSwitch, which appears to have most of the conferencing features I want accessible via a command-line interface (which may well be good enough). 

Anyway, if you have more information about either or both tools, or want to offer your services to set something up, I'd appreciate hearing from you.  I'll update this page again when the project is done.  Thanks!

We now proceed with the original article below...

Dear Lazyweb,

I'm looking for a better way to do international conference calls for Owners' Circle sessions than the various off-the-shelf providers I've used to date -- and I'm willing to pay generously for somebody to help me make it happen in a very short time frame.

I need web moderation controls and I want VoIP support so people don't have to make international toll calls (almost 40% of Circle members are in Europe or Australia), but I also need the ability to let people who don't have decent 'net connections call in to a good old-fashioned voice phone number.  And I'd greatly prefer being able to do all this at a flat monthly cost for up to say, 50 people, without any per-minute per-person charges needing to factor in there.

My initial research seems to suggest that slapping Asterisk on a dedicated server with enough CPU, memory, and bandwidth ought to be just the ticket, since there is already some kind of conference facility in the software with some sort of web UI.

But here's the problem: I don't really have the time to hack around with this stuff for several hours, or to spend several hours more learning enough VoIP acronyms to figure out what the heck I need to buy in order to get regular phones hooked up to this thing.  I definitely don't need to spend all that time and find out I'm barking up the wrong tree here.

In particular, I should note that cost savings is NOT the goal of this project.  I already have a cheap conferencing provider that meets all my requirements except one: unreliable call quality.  Sometimes they're great, sometimes they suck.  Last weekend, I ended up having to manually create a Skype conference with a small group of coaching clients because the quality was so bad.

Yes, I realize that people will have different levels of quality based on their bandwidth, latency, etc.  But that wasn't the problem last week -- a direct Skype conference with the same people produced fantastic quality.

Skype conferencing, however, has a limited group size, no moderation controls, and an abysmal user interface that doesn't allow you to re-conference a dropped participant without re-creating the whole damn conference.  It also doesn't address the regular-phone issue.  (Which is why Skypecasting isn't any better, especially since I hear its quality is variable, too.)

So, I'm pretty much stuck with needing to have something better to use in less than two weeks, and highspeedconferencing.com says it's going to be three months or more before they have a guaranteed-quality offering available.

Therefore, I'm contemplating this VoIP server idea.  Is it possible?  More important, is it practical?  I'm worried that I may be making assumptions rooted in ignorance and wishful thinking, and I really don't want to spend all my spare time hacking together proofs-of-concept.  So if you've actually done Asterisk-based conferencing using a hosted server, I'd really like your input.

And, if you're available to do the work to actually set the thing up for me, I'll happily pay you.  Especially if you can tell me what I do or don't need, and can or can't do with the result.  I'll even pay for your time if it turns out I can't do what I want, as you'll still have saved me time learning that on my own!

So, if you're interested, or just want to share your input or experiences, please drop me a line at voip-help "at" dirtsimple.org, or leave a comment on this post via my blog.  Thanks!

<rant mode="on" type="get-off-my-lawn-you-crazy-younguns">

WSGI middleware is hazardous to your API's health.  There, I said it.

It's not the actual idea of WSGI middleware I have a problem with, but what a lot of people seem to think the idea of WSGI middleware is.

Unfortunately, the way I wrote the part of  PEP 333 that talks about how maybe frameworks can someday become middleware, has produced an entire school of thought that's taken this to mean that middleware should be used to create protocols for extending WSGI itself!

More precisely, people seem to think that if you're going to offer a service via middleware, you have to stick the API for that service into the WSGI environment.  Apparently, they think that because WSGI puts things in the environment, and because servers are allowed to provide extension APIs via the environment, that it makes sense to have middleware whose sole purpose is to put APIs into the environment.

But this doesn't make any sense.  If your application requires that API to be present, then it's not middleware any more!

Middleware, you see, exists in order to wrap applications with additional behavior.  If you can't arbitrarily decide whether to stick it in front of an application, it's not middleware any more.  It's part of the application.

And if it's part of the application, trying to put it on the outside of the application is a complete freaking waste of time!

Meanwhile, you then have to pull stuff out of the environment in order to use it, doing crap like environ['some.session.service'].whatever() in order to do something that could've been written more clearly as SomeSessionService(environ).whatever(), and doesn't require you to stack a bunch of so-called "middleware" on top of your application object.

So please, end the madness now.  If you are tempted to put something into the environment using middleware, so that an application can pull it out later by direct access to the environment, please, just don't.

Instead, write library functions or classes that do whatever needs to be done, and which (maybe) store data in the environment, using library-specific keys.  Do not make users of your library access these keys directly; instead, give them functions or methods that do useful things with this private data.

Do not make users wrap their application with your so-called "middleware", when a callback or decorator will do.  Yes, technically that's still middleware, but practically it's a lot easier to use middleware expressed via a decorator or by passing a callback to a library function, rather than having to configure a complex middleware stack at the server deployment level.

Reserve such true middleware...  i.e., deployment middleware, for tools that are not specific to the application.  Tools like testers, debuggers, loggers, proxies, routers, caches, compressors, and so on.

There are many, many legitimate applications for deployment-stack middleware like this, and almost without exception they are tools that don't care what application they're used with, and neither does the application know they exist.

But application "middleware", while being WSGI middleware in a technical sense, should not be forced into the deployment stack, where it's harder to deploy.  Give your users a real API, for crying out loud, and let them use it inside their application.

Okay.  'nuff said.

</rant>

P.S. One more reason to avoid implementing APIs as middleware, is that it's extremely difficult to write correct, WSGI-compliant middleware, and very easy to do some very stupid things in the process.  Always test your middleware using wsgiref.validate on both sides of the middleware.  That is, wrap a validator around not only the middleware itself, but also the application the middleware is wrapping!  If you don't do that, you'll miss all kinds of compliance bugs.

P.P.S. Notice that this means WSGI 1.0 has failed in its goal of making middleware "both simple and robust".  I have some ideas for a WSGI 2.0 that would drastically simplify both applications and middleware, but at the cost of eliminating the write() callable for unbuffered streaming.  My idea for 2.0 would allow WSGI 1.0 apps to be called from a 2.0 server (using 2.x middleware, of course), but write() operations could only be streamed if the middleware used greenlets or threads or some other async mechanism.

Here's what a WSGI 2.0 application could look like:

def simple_app(environ):
    """Simplest possible WSGI 2.0 application object"""
    return (
        '200 OK', 
        [('Content-type','text/plain')],
        ['Hello world!\n']
    )

I'll leave it as an exercise for the reader to figure out how much simpler everything (including libraries, applications, and middleware) can be under this approach, at the expense of driving write()-driven streaming applications into second-class citizens that have to be run in another thread... which they already have to be under servers like Twisted, anyway.