Python Eggs are popping up everywhere now. The Trac project will apparently become the first extensible, open-source Python application to use eggs as a plugin format. (I had thought Chandler would have earned that distinction, but it’s going to be a good few months before it will really get on the roadmap.) There’s been talk on the Python Mac SIG about platform version tags for eggs on OS X. Ian Bicking is blogging about eggs as a packaging and deployment mechanism for web components. The ever-linkable Django uses setuptools to simplify its setup, although none of the tutorials or installation instructions mention it. (If they would just run “setup.py register” after adding a little more info to “setup.py”, then people could use “easy_install Django” to find and download it via PyPI.)
I’ve also gotten emails from a few people who are using eggs to manage software projects in their workplaces, as well as to manage external dependencies, and Ian Bicking also recently asked about using egg metadata for an at-work project, too. Not bad for an 0.5 alpha. Not bad at all.
There’s a CVS version of 0.6 in the works too. If you’re planning to extend or embed setuptools or the pkg_resources API, you should be working with the CVS version, because the APIs went through some changes relative to 0.5, as I refactored them to conform with eggs’ new architecture overview and terminology guide.
All in all, it looks like the beginning of a quiet revolution in open source software development with Python. I’m still not promoting these things to the wider world of Python users, though, as I don’t want too many people stubbing their fingers and toes on my rough alpha edges. (In fact, I sometimes worry that Django’s current popularity might lead to lots of people bumping into problems with setuptools, and creating a backlash. So far that hasn’t happened, though.)
So how long till beta? How the heck would I know? Try the alphas and send me bug reports, feature requests, and patches. Better yet, try the CVS version and build more tools that work with eggs. Paul Moore has volunteered to create some simple package management tools, for example, to list or delete installed packages. What ideas do you have?
I’m also using eggs for application deployment in my workplace, I even have a natty little quick and dirty updating wizard, which currently ignores the easy_install updating mechanism completely, prototyped.
I was almost tempted to make eggs for all my dependencies, too, but a lot of the packages i use have non-distutils windows installers, and I didn’t see a reason to try and work out how they work and shoehorn my own egg version of it.
So far the only complaint I have is that it doesn’t handle basic auth out of the box, but that’s more of a urllib2 limitation, as I can’t pass it http://user:pass@host.com/ urls and have to do fancy things with install_opener.
Is there yet a specifcation for a repository layout using, say, standard Apache directory indexes that I can stick my various components in and with some fancy configuration bits, or a site-specific .ini file, get easy_install to look there for updates too?
“””a lot of the packages i use have non-distutils windows installers”””
Pity. EasyInstall knows how to convert Windows installers to eggs, but only ones created using the distutils bdist_wininst command.
“””So far the only complaint I have is that it doesn’t handle basic auth out of the box, but that’s more of a urllib2 limitation, as I can’t pass it http://user:pass@host.com/ urls and have to do fancy things with install_opener.”””
Note that you can override the ‘open_url’ method of PackageIndex in a subclass, if you want to. You then have to set your easy_install command’s ‘create_index’ attribute to your PackageIndex subclass before its finalize_options() method gets called.
“””Is there yet a specifcation for a repository layout using, say, standard Apache directory indexes”””
Give –find-links the URL of an Apache directory index, and it will “notice” any eggs, source distributions, or distutils win32 installer .exe’s listed in that directory. Also, –find-links now accepts local directory names as well as URLs, so if you have a network mount with eggs it’s an easy way to share them for deployment in a LAN.
“””that I can stick my various components in and with some fancy configuration bits, or a site-specific .ini file, get easy_install to look there for updates too?”””
To make EasyInstall look there every time, just add an [easy_install] section to a distutils config file (e.g. your ~/pydistutils.cfg file) with “find_links = space-separated URL list” in it. This is actually documented as an example in the EasyInstall manual since 0.4a2.
Incidentally I’ve made such an index for Paste dependencies, for cases where they haven’t made it into PyPI yet (or in one case to patch a package with a bad setup.py). It’s as easy as listing the files, and if you can put them all in the same directory then Apache can even do that for you. For packages internal to our company, I’m expecting we’ll do just that with IP-based access restrictions.
For Django, right now they are making changes way too fast to make any release (mostly to make the system easier to use). What a project like Django needs (and Paste still kind of needs) is the easy_install.py –develop option; with the new (still in CVS?) ability to find svn repositories based on name, and by adding the appropriate line to ~/.pydistutils.cfg, you could install Django that way. Young packages are often “leaky” (especially frameworks) so the boundary between the user’s application and the framework isn’t as firm — you really want to be working directly from an svn checkout.
What line you should add to .pydistutils.cfg, I’m not sure — one for every project you are interested in (with its own package index) isn’t much better than explicit URLs. But a Wiki page would be a giant security hole. Or maybe PyPI can hold unreleased packages better than I think; I haven’t thought that part through yet.
“””Or maybe PyPI can hold unreleased packages better than I think”””
If you look in the CVS version of the setuptools documentation, under “Making your package available for EasyInstall”, you’ll see an explanation of how to include a Subversion link in your package’s “long_description” metadata so that EasyInstall will see it on your PyPI page. So yes, you can use PyPI to track a project in active development that’s not making releases.
For that matter, the documentation also explains how to use setuptools to make snapshot releases numbered by Subversion revision number or build date, and rotate out (delete) older snapshots. If you build the snapshots to an Apache-indexed directory, you need only set your PyPI “download_url” to point to that directory.
So in fact PyPI+setuptools should be able to handle this scenario just fine, as long as you take the time to plow through all of the tips in the setuptools developer guide. I have much the same situation with some of my packages (e.g. RuleDispatch), but I haven’t had time to actually set it up for this yet, or I’d point to it as an example.
(Admittedly, having the –develop option would be very useful here too, and I do plan to try to squeeze it in by the weekend.)
This is great and exciting!
Now I want to go write something I can actually distribute…
The potential for eggs to improve rapid application deployment is immense and quite exciting.
What I’m not getting, or missing entirely, is the security model for eggs.
Is there a mechanism built-in eggs for public-key signatures of a package to be checked against local keyrings full of signatures? [Lately I’ve been impressed with how easily yum handles this.]
“””Is there a mechanism built-in eggs for public-key signatures of a package to be checked against local keyrings full of signatures?”””
My current plan for this is to have the signature(s) be external to the egg, since PyPI already supports that, and indeed the “upload” command can already sign an egg and upload the signature. But if you want to check the signature, right now you have to do it manually.