There’s a saying that says if you improve a product’s usability by 10%, you’ll end up with twice as many people who can use your product. (I thought this saying came from Joel On Software, but if so I can no longer find it.)
Anyway, the undocumented corollary of this saying is that if your product is only 90% finished, but early publicity doubles your userbase, you are going to be swamped with complaints about that missing 10%. 🙂
A case in point: setuptools. A couple of days ago, Joe Gregorio blogged that Python library developers shouldn’t use it, partly because of some installation issues he ran into. I wrote Joe back privately, to make sure I understood the issues, and also to make sure he knew where to find the documentation that would’ve resolved the issues he encountered.
But Joe’s just the canary in the mine shaft, so to speak, and his song is just the warning that, like it or not, setuptools has gone well past its early “alpha” audience, and it needs to go into beta as soon as possible. And that means cleaning up those 10% “fit-and-finish” issues like the ones he encountered. So what else could I do? I bumped down the priority on other new features for setuptools, and got started on fixing Joe’s problems.
So, I’m happy to announce that setuptools in SVN now supports –prefix and “traditional” PYTHONPATH setups right out of the box. Unlike previous versions, it now supports arbitrary PYTHONPATH install targets without needing any special setup. Just make sure the target directory is on PYTHONPATH at install time and runtime, and away you go.
Currently, you can only get this version from SVN, but I’d appreciate testing and feedback. This version also has some hopeful fixes for /usr/lib64 issues on 64-bit Linux, and a lot of other odds and ends. I hope to release an official 0.6a10 version soon (maybe tomorrow?), and maybe an 0.6b1 by PyCon.
A big thanks to everyone who helped me figure out solutions to the problems, provided feedback (even of the complaining kind) and otherwise participated in the process of getting setuptools to where it is today.
(P.S. This post is not intended to imply that setuptools is now finished, only that I hope its usability is now caught up to the size of its current audience, and maybe a little beyond.)
Glad to see this post and to get some closure on the back-and-forth of the last couple of days. BTW, I appreciate your comments on my setuptool-related posts, especially the last ones which attempt to clarify your position. No hard feelings whatsoever, I meant it when I said that I learned a lot about setuptools in this process.
Grig
Your post on python vs java a while back really made me interested in learning python. And the post about belaboring the obvious was hilarious.
I also dugg your gripe about blogger. Btw I have another gripe to add.. when u enable word verification for comments, it even asks the owner of the blog to jump thru the hoop for adding comments to his own blog??!?. Duh? I mean, if he wanted to spam himself he could have just written more posts right? It’s not like posts are even word-verified. What do you think Philip? Have u ever been annoyed while posting a comment on your own blog? To Steven Johnson, if u’re reading this could you see if u can do something about it? Thanks.
Please add a –fetchonly flag or somesuch that downloads the egg and its dependencies to the current directory. This facilitates installs on computers not connected to the internet and shouldn’t take too long to implement.
“””Please add a –fetchonly flag or somesuch that downloads the egg and its dependencies to the current directory.”””
Use “easy_install -maxd .”. “-m” says not to add to a .pth file. “-a” means to always copy eggs to the target directory, even if they’re already installed somewhere else on your machine. “-x” excludes scripts, and “-d .” sets the current directory as the target location.
Once you’ve done this, you can transfer that directory to your target machine, where you can use “easy_install -f. *.egg” to install them.
The easy_install command is *really* flexible — too flexible, in fact. My plan for 0.7 is to have the “nest” command offer subcommands for common use cases, as most people don’t like to have to figure out in their heads what obscure option combination will do the thing they want. 🙂
“””Have u ever been annoyed while posting a comment on your own blog?”””
Didn’t occur to me, actually. I’m personally glad I see the same interface as my users, since then I know what *they’re* seeing. I’d hate to lose that just to save on typing a made up word like “plroxg” every now and then. 🙂
Usability: Please, please, please: rename ez_setup or easy_install.
When I read this post the first time, I just shook my head and held my
tongue. Maybe setuptools will be cool some day, when it works.
Then, unfortunately, I had the bad luck of having to interact with it
again. Since I can’t find a bug tracker for setuptools (alright, I
didn’t look for one actually, posting a comment in a blog burns more
frustration than filing a bug), I’ll just post this here.
All I wanted to do today was get trac running. Much to your
credit, though, they have embraced eggs and setuptools. Specifically,
I need to build an egg to install a plugin. Okay, before I can do
that apparently I have to upgrade my setuptools installation. Ubuntu
ships setuptools, but apparently not a new enough version. Minus a
point.
I downloaded ez_setup.py and ran it. Woops, it won’t even try as long
as my current setuptools installation exists. Minus a point.
So, as root (boy it’s a good thing I had root on this machine), I
uninstall the packaged version of setuptools. I run ez_setup.py
again, and what do I see?
“Because this distribution was installed –multi-version or –install-dir,”
I’m sure you know the rest. Well, I didn’t specify either of those,
but yes, I do have a custom installation target set up for distutils.
And indeed, setuptools is unimportable. Even though I told it to
install to a site directory. Minus a point.
Now what? I have no idea. I check out svn and try setup.py. Does
that work? No, not at all.
“error: bad install directory or PYTHONPATH”
Once again, trying to install to a site directory.. Only this time it
fails before it even starts, setup.py won’t even try to install
setuptools. It tells me some stuff I already know about my
environment and gives me an URL to look at. Oh, well, that’s nice of
it. There’s a lot of text on this page and I’m pretty upset already.
None of it really addresses the issue I’m having, either. Minus a
point.
Finally, I give up on trying to do things setuptools’ way and add my
own pth file pointing to the egg ez_setup.py installed. Why couldn’t
ez_setup.py do this? I have no idea. If it had this would have taken
2 minutes instead of an hour. Blech.
So here I am, that wasted hour later. Five ways removed from what I
actually want to be doing. Not that I’m really surprised. This is
about how every interaction I’ve had with setuptools has gone. It
never works right. Sometimes it works eventually, but usually it
doesn’t.
What am I supposed to think?
“””Woops, it won’t even try as long
as my current setuptools installation exists.”””
It sounds to me like it should’ve warned you of conflicting packages, and asked you to rerun with -D or –ignore-conflicts-at-my-risk. Did it not do this?
“””Even though I told it to
install to a site directory. Minus a point.”””
I’ll bet it was the /usr/local one, which is a Debian-ism, not a Python-ism — meaning that EasyInstall is currently unable to tell that it’s a valid site directory unless specified via –site-dirs or a config file.
“”””error: bad install directory or PYTHONPATH”
Once again, trying to install to a site directory.. Only this time it
fails before it even starts, setup.py won’t even try to install
setuptools. It tells me some stuff I already know about my
environment and gives me an URL to look at.”””
The issue here is presumably that you’re using a non-standard “site” directory, by which I mean one that’s defined by Ubuntu but not by Python. If you read the URL pointed to, you’d see lots of nice ways to work around this issue, such as –site-dirs, and the option to configure that permanently.
“””Finally, I give up on trying to do things setuptools’ way and add my
own pth file pointing to the egg ez_setup.py installed. Why couldn’t
ez_setup.py do this? I have no idea.”””
From the error message:
“You are attempting to install a package to a directory that is not on PYTHONPATH and is not registered as supporting Python “.pth” files by default.
…
You can set up the installation directory to support “.pth” files, and configure EasyInstall to recognize this, by using one of the approaches described here
…”
Do you have any suggestions as to how that text could’ve been made clearer?
It sounds to me as though you had already ruled out your choice of directory as the culprit, even though the error message explained this and said what you needed to do to fix it, i.e., put it on PYTHONPATH or register it with EasyInstall as a “site” directory.
Your workaround, meanwhile, has not fixed the actual problem, so the next thing you do with setuptools will give you the same error message. If it’s a genuine site directory, add it to the site-dirs setting like the error message said.
What would you have me do in this circumstance? Blindly install to a directory that won’t work, then have the user’s packages possibly end up mysteriously unusable, with *no* error message? Or make people with weird setups add an extra option or edit a config file? Bug Ubuntu to pre-configure their non-standard site dirs that are implemented by *patching Python*?
I suppose the other alternative is that if I don’t know whether a directory is truly a “site” directory or not, I could write a test .pth file and spawn another Python process to see if it gets processed. On subsequent runs I could check to see if easy-install.pth is there and skip the shell-out overhead. But I can’t skip the overhead unless -m is in effect. Hm. I’ll give it some thought, but I worry that this will introduce new possible failure modes into stuff that was already working. Ah well. In for a penny, in for a pound. How many more ways can there be to have a forked up Python? (By which I mean distros’ patching.)
But I can see that if I do this, I can at least simplify the error message and even get rid of the –site-dirs option, since whether a directory is “site” or not will not be subject to configuration, thus eliminating another source of installation failure. In fact, it’ll be a cleaner fix for the /usr/lib64 woes some users encountered last week.
I think I like it. Finally, a way to crush the forces of widespread distro-patching! Thanks for the idea. 🙂
“””Usability: Please, please, please: rename ez_setup or easy_install.”””
ez_setup is a front-end to easy_install; it just happens to download the setuptools .egg and put it on sys.path before running easy_install. Thus, it takes exactly the same options as easy_install – it *is* easy_install, only bootstrapped so if you have net access, all you need is ez_setup.py.
Btw, Paul Graham (of viaweb fame) also said “A 10% improvement in ease of use doesn’t just increase your sales 10%. It’s more likely to double your sales.”
From here.
“””
It sounds to me like it should’ve warned you of conflicting packages, and asked you to rerun with -D or –ignore-conflicts-at-my-risk. Did it not do this?
“””
Nope, nothing like this. –ignore-conflicts-at-my-risk sounds kind of like what I was hoping for (although I would have guessed it to be spelled “–force”).
“””
I’ll bet it was the /usr/local one, which is a Debian-ism, not a Python-ism — meaning that EasyInstall is currently unable to tell that it’s a valid site directory unless specified via –site-dirs or a config file.
“””
It was one beneath ~/.local/ – not a Debian-ism or a Python-ism. I did notice –site-dirs, but the prospect of having to remember to pass an extra command line argument every time I used … what? setuptools? anything installed with setuptools? anyway, not an exciting prospect. I saw the documentation about ~/.pydistutils.cfg and tried to configure an extra site directory using that, but to no avail; whatever I put in that file was ignored and I eventually gave up on it.
“””
Do you have any suggestions as to how that text could’ve been made clearer?
“””
Basically, the message is wrong. As you pointed out, EasyInstall can’t currently determine if a directory is a site directory or not. I was going to suggest a –force option, so that users who know what they’re about to do will actually work don’t have to jump through so many additional hoops. The thing you mentioned at the end of your comment sounds like it might address this particular issue, but –force might still be useful, for when the next guy runs into the next unexpected problem (it’s a really hard problem, there will probably be unexpected problems for a long time to come).
“””
It sounds to me as though you had already ruled out your choice of directory as the culprit, even though the error message explained this and said what you needed to do to fix it, i.e., put it on PYTHONPATH or register it with EasyInstall as a “site” directory.
“””
It was pretty clear what was causing the problem, but the particular changes suggested by the error messages and the documentation just weren’t an option. I have a bunch of other tools that are more important that depend on features of my current configuration.
“””
Your workaround, meanwhile, has not fixed the actual problem, so the next thing you do with setuptools will give you the same error message. If it’s a genuine site directory, add it to the site-dirs setting like the error message said.
“””
Hmm, are you sure? I went ahead and /used/ setuptools to build an egg using my work-around. So that works, at least. Is there some other feature which will not work in this configuration?
“””
What would you have me do in this circumstance?
…
I think I like it. Finally, a way to crush the forces of widespread distro-patching! “””
Sounds nice, although wouldn’t it be even nicer if Python just kept track of what directories were site directories? How hard could maintaining one more list of strings be? Granted, this won’t be of any use until Python 2.5 (assuming someone gets to work on this and gets it into 2.5) is in wide-spread use.
Thanks for the detailed response, and good luck with the changes you’re planning. 🙂
Setuptools seem nice for pure python, and RPMs or Autopackage is nice for pure C, but I’m kind of flummoxed when I mix them into one app because some Linux distributions follow Python upstream and use UCS2 but Fedora uses UCS4. Any suggestions?
Here’s one –
I’ve been fighting with installing pysqlite2, and specifically with its use of setuptools.
When I launch the app (using sudo), it blows up with a permission denied error while trying to access my home directory!:
File “/usr/local/lib/python2.4/site-packages/setuptools-0.6a10-py2.4.egg/pkg_resources.py”, line 2301, in ensure_directory
os.makedirs(dirname)
File “/usr/local/lib/python2.4/os.py”, line 156, in makedirs
makedirs(head, mode)
File “/usr/local/lib/python2.4/os.py”, line 156, in makedirs
makedirs(head, mode)
File “/usr/local/lib/python2.4/os.py”, line 159, in makedirs
mkdir(name, mode)
OSError: [Errno 13] Permission denied: ‘/home/ronwalf/.python-eggs’
Not exactly fun! How in the world do I get this thing to launch as a user without its own writable home directory?
I don’t understand what you mean by “launch the app”. Do you mean you’re trying to use pysqlite2?
Probably the reason it’s trying to write to your home directory is that there’s a C extension involved, and it was installed in zipped form. If you use the –always-unzip option to easy_install, however, it will be installed unzipped and won’t need to extract the extension module at runtime.
So users without a writable home directory (apache, mysql, etc) can’t import any module that uses a C extension that was installed with the default arguments?
It should at least fail with an comprehensible message (and instructions to install it unzipped).
“””It should at least fail with an comprehensible message (and instructions to install it unzipped).”””
Yep, that’d be good alright. I’m adding it to my “fix before beta” list.
In the meantime, there’s an –always-unzip option you can pass to easy_install that will fix your immediate issue.