Backward compatibility is a blessing for users and a curse for developers, but often the only way to move a technology forward is by providing backward compatibility. With that in mind, I’m pleased to announce setuptools and EasyInstall version 0.5a9, which now attempt to deal sanely with such things as old “unmanaged” package installations (i.e., direct to site-packages or elsewhere) and packages that may or may not be using __file__ or __path__ to look at source or data files. (This was a lot of work, even though I somehow managed to squeeze it in this weekend while also working nearly 14 hours on helping my wife get her store’s new location set up, toting heavy boxes and stacks of building materials about!)
In addition to all the cool new features, I also just finished the Developer’s Guide and Command Reference for setuptools. The “extending and reusing” section is still all XXX’s, but anything else you might want to know should now be there, including stuff on how to use it with Pyrex, how automatic package and source file finding works, how to easily do daily builds and tagged snapshot builds, automatically delete outdated snapshots, how to manage different options for multiple projects while keeping your scripts simple, zipfile safety analysis, and much much more.
Indeed, it’s practically a tome on “everything you might want to know about practical build and distribution management for Python libraries” using setuptools and Python Eggs. Well, as much of a tome as has ever existed for that, anyway. It does assume, however, that you have read and understood the distutils manuals, especially Distributing Python Modules. (But Installing Python Modules is a good read, too.)
Anyway, I think I’m finally satisfied with the end-user and developer/packager functionality provided by EasyInstall and setuptools, respectively, so it’s time to go back to work on the pkg_resources runtime module, which still needs some improvements. About six weeks ago, I shifted gears from work on the runtime to work on all the user-visible flash and dash, because I was worried that my “market window” for getting Python Eggs some “mindshare” in the community was expiring. Now that EasyInstall and setuptools are getting some attention and use among the SIGs, I can now afford to back off on the marketing/features side, and go make the core engine really work right. 🙂
For that matter, I can now finally start using setuptools for the use cases I had in mind when I first conceived of it almost a year and a half ago. Namely: distributing packages without bundling their dependencies.
Back then, I envisioned a future bustling marketplace of Python libraries, where reuse was a practical reality instead of just a theoretical possibility. That day isn’t here yet, but I can now begin setting an example with the gradual breakup of PEAK into lots of narrowly focused packages like DynamicImports, SecurityRules, VersionTool, PyUUID, PyCLI, and so on. I’m also looking forward to being able to reuse others’ packages, without giving more than a moment’s thought to “how will I make sure users have this?”, and a few seconds to add it to my setup.py.
Yes, looking back is definitely the way forward, because if Python Eggs were only a solution for new and refactored packages, they would be no solution at all. Looking back means looking towards our “legacy” – the wealth of packages that we inherit via PyPI and indeed any web page that links to a Python package’s source distribution or win32 binary distribution. With “legacy” often being considered a bad word in the IT field, it’s important sometimes to remember its original meaning as an inheritance, especially an inheritance of great value. EasyInstall now lets us connect our rich past, with our even richer future.
You’ll be glad to know we’re in the process of converting to eggs. We’ve got one of our biggest projects being built a managed from egg sources (we really needed something like the ‘requires’ functionality). It’s looking really good and, as a colleague said to me the other day, it’s really tempting to start egg’igying the whole standard library :-).
It would certainly let the standard library breathe a little and would avoid some of the problems of what to and not to include. I still can’t make my mind up whether a core python install should include anything? (but thats a phiposophical questions for another day).
Just a quick question (it seems like here would be a good place for the answers so it looks like a good place to as). How do you approach packages like Zope interfaces where they aren’t using setup.py?
To some extent this routes around OS-level packaging (.rpms or .debs), which may not be the best thing to do… Ideally eggs would be an extension to, instead of replacement for, that packaging. Not sure how that works out in practice, though. Maybe some kind of automatic egg -> .rpm|.deb converter is in order? To me part of the appeal of a packaging system is reproducability: given the list of packages on a machine I can exactly duplicate that environment (modulo /etc and /var and /home) just by installing the same list of packages. Eggs (and CPAN too!) break this and I’m wondering how to fix this.
“”” To some extent this routes around OS-level packaging (.rpms or .debs), which may not be the best thing to do”””
Eggs were designed to meet the needs of Python *applications*. Frankly, OS-level packaging SUCKS for applications. You can easily distribute an application as a script plus a directory full of eggs.
“””Ideally eggs would be an extension to, instead of replacement for, that packaging.”””
They are; if the OS packaging installs eggs, then Python programs can know what packages are installed or not installed, and read any data or metadata from them.
“””Maybe some kind of automatic egg -> .rpm|.deb converter is in order?”””
Well, it’d be fairly straightforward for someone familiar with those systems to write, since all it’d need to do is dump the egg file or directory in site-packages. If the egg contains a ‘not-zip-safe’ in its ‘EGG-INFO’ subdirectory, then you unzip it and package as a directory instead of a zipfile.
“”” given the list of packages on a machine I can exactly duplicate that environment”””
And given a list of the eggs on a machine, you can duplicate its Python environment — even on another platform, using a different packaging system, or even if it has no packaging system. (E.g. Windows).
However, there’s an even deeper point, which is that since eggs can know what other eggs they depend on, you don’t need a list of “all the eggs”, all you need is the list of eggs for the applications you use, and installing those will install the dependencies. (This means, by the way, that if a platform has maintenance scripts that use certain libraries, they should package those scripts as an egg, and then wrap the egg in an OS-level package.)
So really, this is a platform-independent packaging system for Python, but if you want to create OS-specific wrapper packages, nothing stops you, as long as there’s some algorithm to map PyPI project names to a namspace in your platform’s packaging system. (Otherwise, you can’t automatically generate a dependency list from egg dependency info.)
This doesn’t stop Python applications from installing their own eggs, but who cares? Either the application is packaged by your OS, in which case it wouldn’t need to install additional eggs, or else you installed the app separately anyway.
“””How do you approach packages like Zope interfaces where they aren’t using setup.py?”””
But they do; check out the releases at http://www.zope.org/Products/ZopeInterface.