dirtSimple.orgwhat stands in the way, becomes the way
A “dirt simple” download-and-install CPAN clone

A “dirt simple” download-and-install CPAN clone

It seems that CPAN clones are in the air of late. Yesterday, Ian Bicking posted a comment here that he was working on an automated download facility, and today, there was a Planet Python post about Uraga, a CPAN clone some other folks are working on.

So it got me to thinking, how simple could this actually be? Bob Ippolito and I had talked about making something like that using eggs, where there would be some sort of manifest file that told you where to download things from. But it occurred to me today that there are already “manifest files” out there on the web that work just fine: web pages and directory listings. For example, SourceForge download pages contain a wealth of links to URLs containing “SomePackage-1.0.zip” and the like. Since Python egg filenames also include Python version and platform information, everything you need to find a file is right there in the links – just parse the HTML page and go.

To test the usefulness of this theory, I went to PyPI and randomly selected 43 packages, investigating their download links. Here are the results:

  • 15 packages had no download URL at all
  • 2 had a download URL that went to their homepage, with no direct links to downloadable files
  • 10 had a download URL that pointed directly to a .zip, .tgz, .tar.gz, or .tar.bz2 file with a specific version number in the filename
  • 10 had a download URL that went to an HTML directory listing with links to versioned files
  • 6 had a download URL that went to a “latest version” (i.e. no explicit version number) archive or .py file

So, about half of the packages could have been processed by a spider hunting for specific versions. And about half of those could easily add .egg files to their download listings. Very interesting.

Of course, not every distributor of a package is going to want to mess with making eggs for different platforms, so a really useful tool is probably going to have to be able to download a source archive and build an egg from it.

Now you may be wondering, why build an egg? If you’re going to have to build from source anyway, why not just install the package directly? Because eggs – even unpacked ones – let you keep multiple versions of a package on your system, and activate them at runtime.

So now I’m thinking, maybe there should also be an install_egg command for the distutils, that basically builds and egg and then installs it in site-packages (or wherever) for you. Then, we could use that with our hypothetical PyPI spider, to make a complete fetch-and-install utility.

Now, once we have that, let’s say that somebody wanted to make a bunch of packages available as eggs for their platform. All they’d need to do is run that fetch-and-install such that it installs to a web-accessible directory, whose contents are visible as a directory listing. Now, somebody who adds the URL of that directory to their spider’s search URLs would be able to find and download pre-built eggs for whatever they needed, without needing to do any building.

It’s starting to sound an awful lot like what everybody’s trying to make, doesn’t it? So what are the architectural components we need?

  • The “reader”: An HTML reader that scans a web page for links to eggs and/or source archives with names that match distutils-standard naming conventions
  • The “finder”: A tool that takes a list of candidate start URLs and invokes the reader on them to search for specified package(s), caching the resulting index data
  • The “source catalog”: A tool that, given a package name, finds download URLs from PyPI and determines whether they are archives or links that should be passed to the reader
  • The “fetcher”: given a desired package, it consults the finder and the source catalog, trying to download a platform-suitable egg, falling back to finding a source archive and building an egg.
  • The “builder”: given a source archive URL, download it, extract it, find the setup.py, and build/install an egg
  • Some way to decide what version of a package to build/install, if more than one version is available. (e.g. a way to select only stable versions, or whatever)

Interestingly, it might be possible to just repurpose an existing Python web spider to do a lot of this, just by spidering from PyPI with a reasonable external link depth, to build an index of package+version to download URLs. In fact, you could use that spider to simply create an HTML page with all the download links.

Given the existing capabilities of the egg runtime, and the assumption that an existing HTML-parsing spider (or browser-emulator) could be made to do the fetching and parsing, the biggest parts remaining are the “builder”, and managing the whole thing’s configuration. It seems to me there are lots of policy issues ranging from the trivial (where to put the eggs) to the critical (what versions to allow? code signing? checksums? what download sites do you trust?)

But the interesting thing about all this, I think, is that in a sense we already do have a CPAN: it’s called the web. Now all we need is a smart enough client to use it. 🙂

In the meantime, I’ve actually managed to squeeze in a few more hours’ work on Python Eggs and their documentation. Directory scanning, dependency management, and even namespace packages are implemented now, although some of these features have received rather minimal testing. The documentation has also undergone a significant overhaul to explain many more of the implemented features, although there’s still a lot more to write, just to explain what can be done with the current version of the runtime.

Follow-up: I built eggs for the “mechanize” package and its dependencies, and found it only takes a few lines of code to retrieve and analyze the links of a download directory, such as the PEAK projects download directory, or a Sourceforge file listing. Of course, actually downloading the list and parsing it can be slow, so an “end-user quality” download tool might need to do a fair amount of tweaking to make the process more friendly for impatient people like me.

Join the discussion
  • I think you make it too complicated, and scraping is too unreliable. “Dirt simple” would be 90% of the job for 10% of the effort — scraping is more like 50% of the job for 50% of the effort 😉

    There’s room for all the necessary metadata in PyPI right now, including direct links (except to SF downloads, but let’s say SF mirroring is special cased). And there’s all sorts of other useful information in there as well.

    At this exact moment it would take screen scraping to get access to that information, but XML-RPC support is already live in PyPI, it’s just a small job to add some useful functions.

    I’d like to also add metadata comments to PyPI — basically PyPI-related support requests that are emailed to maintainers — and I’d like those to show up on the web page as well. *There* it might be useful to use (much more limited) screen scraping, so you could get immediate centralized access to a URL (maybe with a warning on installation about trustability) — and of course correct metadata would supercede any comments (and comments could be deleted/hidden when resolved). But I don’t know, it might be easy enough to get people to correct things in a timely manner. It’s worth starting there.

  • For long-term world-conquest purposes, having these .egg files be readily usable by the major distributions would be a plus.
    If there is no additional effort required to get RPMs and ebuilds and Debian packages, how cool would that be?

  • “””how cool would that be?”””

    Yep, it’s almost as if eggs were designed that way…

    Seriously, yes, the key audiences for Python Eggs are people who need to *distribute* software, and people who need to *manage installations* of software. Sometimes the distributors are also the software’s authors, and sometimes the people managing installations are also developers using the software, but often they are not.

    I’m actually thinking about changing the name of “setuptools” to “easy_install” or something like that, to help emphasize the part where eggs make things easy to install, uninstall, and otherwise manage. You can have as many versions of something as you want on your system, you can remove something by deleting a single .egg file or directory, and you can select what you want to use at runtime with a simple “require()” call. What could be easier?



Stay In Touch

Follow our feeds or subscribe to get new articles by email on these topics:

  • RSS
  • RSS
  • RSS


Get Unstuck, FAST

Cover photo of "A Minute To Unlimit You" by PJ Eby
Skip to toolbar