Monday, July 11, 2005

The Way Forward is to Look Back

Backward compatibility is a blessing for users and a curse for developers, but often the only way to move a technology forward is by providing backward compatibility. With that in mind, I'm pleased to announce setuptools and EasyInstall version 0.5a9, which now attempt to deal sanely with such things as old "unmanaged" package installations (i.e., direct to site-packages or elsewhere) and packages that may or may not be using __file__ or __path__ to look at source or data files. (This was a lot of work, even though I somehow managed to squeeze it in this weekend while also working nearly 14 hours on helping my wife get her store's new location set up, toting heavy boxes and stacks of building materials about!)

In addition to all the cool new features, I also just finished the Developer's Guide and Command Reference for setuptools. The "extending and reusing" section is still all XXX's, but anything else you might want to know should now be there, including stuff on how to use it with Pyrex, how automatic package and source file finding works, how to easily do daily builds and tagged snapshot builds, automatically delete outdated snapshots, how to manage different options for multiple projects while keeping your scripts simple, zipfile safety analysis, and much much more.

Indeed, it's practically a tome on "everything you might want to know about practical build and distribution management for Python libraries" using setuptools and Python Eggs. Well, as much of a tome as has ever existed for that, anyway. It does assume, however, that you have read and understood the distutils manuals, especially Distributing Python Modules. (But Installing Python Modules is a good read, too.)

Anyway, I think I'm finally satisfied with the end-user and developer/packager functionality provided by EasyInstall and setuptools, respectively, so it's time to go back to work on the pkg_resources runtime module, which still needs some improvements. About six weeks ago, I shifted gears from work on the runtime to work on all the user-visible flash and dash, because I was worried that my "market window" for getting Python Eggs some "mindshare" in the community was expiring. Now that EasyInstall and setuptools are getting some attention and use among the SIGs, I can now afford to back off on the marketing/features side, and go make the core engine really work right. :)

For that matter, I can now finally start using setuptools for the use cases I had in mind when I first conceived of it almost a year and a half ago. Namely: distributing packages without bundling their dependencies.

Back then, I envisioned a future bustling marketplace of Python libraries, where reuse was a practical reality instead of just a theoretical possibility. That day isn't here yet, but I can now begin setting an example with the gradual breakup of PEAK into lots of narrowly focused packages like DynamicImports, SecurityRules, VersionTool, PyUUID, PyCLI, and so on. I'm also looking forward to being able to reuse others' packages, without giving more than a moment's thought to "how will I make sure users have this?", and a few seconds to add it to my setup.py.

Yes, looking back is definitely the way forward, because if Python Eggs were only a solution for new and refactored packages, they would be no solution at all. Looking back means looking towards our "legacy" - the wealth of packages that we inherit via PyPI and indeed any web page that links to a Python package's source distribution or win32 binary distribution. With "legacy" often being considered a bad word in the IT field, it's important sometimes to remember its original meaning as an inheritance, especially an inheritance of great value. EasyInstall now lets us connect our rich past, with our even richer future.