Well, less than two weeks after I speculated on how a dirt-simple CPAN clone might be constructed, it’s ready to use for everything but scripts. Installing a Python package can now as simple as typing (say) “easy_install SQLObject”, and boom, you’re done. Cleanly uninstalling or upgrading packages is almost as easy.
EasyInstall is at version 0.4a1 right now, so there are still a number of features on my to-do list. Most important are script handling and automatic dependency installation. The docs could use a “Making your Package Work Well with EasyInstall” section, and I’d like to do a little refactoring so that it uses the logging module and has support for pluggable downloaders, so that GUI or Twisted-based applications can control the actual downloading process.
In the process of adding all these new features, I’ve also been plowing through the Python Eggs to-do list, and especially updating the Eggs developer documentation. I’d still like to write a section on creating application plug-in architectures using Eggs, and expand the API reference some more, but it should actually be pretty usable already.
The trickiest part of all this to figure out has been bootstrapping the installation process. Once EasyInstall is able to handle scripts, it’ll be able to upgrade itself, so that’s not a big deal, and initial installation can be done the old-fashioned “setup.py install” way. (Actually, if you have Python 2.4 and use “python -m easy_install” to run it, EasyInstall can upgrade itself now.)
But what about packages that depend on using a newer (or older) version of setuptools to do their own installation? That one has been making my head spin. With the package still changing rapidly, it’s tough to just tell people, “make sure you have setuptools version X installed”. On the other hand, if you just ‘import pkg_resources’ and ‘require("setuptools>=0.4a1")’ in your setup script, then it’ll exit with an ImportError, DistributionNotFound, or VersionConflict as appropriate. Or you can trap those errors and exit with a message asking the user to install (or upgrade) setuptools. I guess that’s probably the sanest, safest way to handle it, and I should probably add some sample code to the doc pages to show how to do that.
In the meantime, I took a break from writing this to see if I could come up with a way to make setuptools/EasyInstall install itself as an egg, even when you first install it using the distutils. Well, it turns out that the distutils themselves have a way to do this, using the two-argument form of ‘extra_path’. The CVS version of setuptools now installs using a ‘setuptools.pth’ file, and is installed to a directory called ‘setuptools-0.4a1.egg’, thereby creating a sort of “poor man’s egg” with no metadata, but enough information in the directory name to allow the egg runtime system to be able to treat it as if it were a real egg. The ‘setuptools.pth’ file then ensures that the package is always available to be imported, but it has lower precedence than ‘easy-install.pth’, so if you use EasyInstall to upgrade setuptools, the newer version will get used, and the setuptools.pth will be a relatively-harmless leftover.
Whenever setuptools’ version changes, so too will the extra_path setting, so each installed version will actually be separately available. The only catch is that you can’t actually change versions at runtime, because of the usual problems with reloading modules in non-trivial Python applications. Switching between versions of the egg runtime will require an interpreter restart, after changing the easy-install.pth file to point to the new version of setuptools. But, applications using EasyInstall internally to “update plugins” would probably do such a restart before activating any other library upgrades, so that’s probably not a big deal; setuptools will really be no different than the application’s other libraries in that respect.
Whew. Sometimes it really amazes me what you can do with just what’s in the Python standard library. Setuptools and EasyInstall do zip and tarfile processing, HTTP downloads, HTML screen scraping, subversion checkouts, and a whole host of other things, and I’ve only spent two weekends working on EasyInstall so far.
Now, somebody tell me why I didn’t write this thing sooner? 🙂
After using it this weekend to package up a pyrex wrapped c library I’m converted!! I really think you’re onto a winner.
The main problem I’ve had is the lack of scripts support. (oh. and Zope Interfaces completely baffling me with it’s non standard installation).
If I can help in any way, just shout – even if it’s only for testing. (I’ve set up a vanilla install of ubuntu and debian in a vmware session so I can rollback to a clean system at any point – useful for testing).
This is awesome. I tried it with the example, and it just worked, even though I’m behind a prox/firewall.
I also tried easy_install pygame, which doesn’t work, because the package isn’t avaiable on pypi.
Now say I wasn’t sure about the name. So maybe I wanted –display *game* and then a list of packages that match the pattern.
I’d find that nice.
Florian: Once the PyPI server situation is figured out, I’m hoping we can add a more straight-forward XML-RPC API for things like searching and retrieving metadata. That should make features like search pretty easy to implement.
Note, too, that searching PyPI for the package you want is as simple as going to PyPI and using the search function. I really don’t want to turn EasyInstall into a web browser. 🙂
However, there’s nothing stopping somebody from creating a super-duper PyPI client/package manager that uses EasyInstall to do installations. It’s open source, after all.
About the browser thingy, consider apt-get/emerge. I think that illustrates the point.
Yes, I actually thought that a xml-webservice for a package index would be an awesome thing, including but not limited to searches.
In fact the first thing I tried to find out is how to plug in another package index, that’s using xmlrpc.
“””If I can help in any way, just shout – even if it’s only for testing.”””
Thanks, Tim. I’m working on script support now, and it should be finished soon. I definitely can do with more testing, though, especially when I hit version 0.5, which will mark the completion of all the “installing user” features. (As opposed to features for package developers, distributers, and people integrating EasyInstall into applications.)
“””I also tried easy_install pygame, which doesn’t work, because the package isn’t avaiable on pypi.”””
Big binary packages that have tricky builds are an excellent candidate for egg distribution, since EasyInstall can use eggs without needing to do the build step. If you have setuptools installed and import the ‘setup()’ function from there, then adding an egg distribution to a package is as easy as “setup.py bdist_egg”, or adding ‘egg’ to your existing list of bdist –formats.
I’ve looked a little bit at making EasyInstall support win32.exe binaries as well, but so far I’m finding it quite tricky, because there’s no PKG-INFO and paths in the zip may include “extra path” parts.
Phillip! You rock!
About the version conflicts. The haskell folks might have an answer, and it’s language neutral. EternalCompatabilityInTheory
I’d love to see this part of a package system.
Oh. And now I see that your concern was localized to setuptools. But they still have a cool idea. 🙂