I’m beginning to think that predicate-dispatch generic functions capture something really fundamental about computation, or that at least they’re much more fundamental than interfaces or adaptation. As useful as interfaces and adaptation are, they’re actually pretty cumbersome compared to generic functions.
For example, in peak.web I’d put together a framework with many Zope 3-like capabilities, that was nonetheless a lot more lightweight. It incoprorates dozens of interfaces, but far fewer than are used in Zope X3 for the same features. Today, I took a look at several aspects of the framework, and it’s looking to me like I could encapsulate something like 80% of its non-XML-related functionality in maybe half a dozen generic functions: one to respond to HTTP requests, one to traverse names in URL paths, one to get an object’s URL, one to do any special handling needed upon arriving at a particular traversal location (like requiring a login or starting a session), and one to handle formatting and/or reporting of exceptions.
In the process, I’d be doing away with at least as many interfaces, and probably that many more concrete classes. And, it would be a lot more obvious if you wanted to change some behavior, where to change it. That is, just add a method to the right generic function. No need to figure out what interface to implement or base class to subclass, just add code that says, “if this happens, do that”. So simple, so basic. Yet at the same time, more flexible!
Like peak.security before it, peak.web was built on an interlocking collection of interfaces and adaptations in order to make it flexible. But, it’s only flexible at the places where I figured out that “joints” were needed. Generic functions are more like octopus tentacles, they bend wherever you want them to bend, because you can add “joints” at runtime. So, instead of having to bend and stretch oddly to deal with the limitations of a particular set of joints, you just do your thing and move on.
So far, in fact, there is only one thing that I’ve found really sucks about generic functions, and that’s the name. Not only is it too “generic”, but in the vast majority of languages that have generic functions currently, they don’t do all the things that predicate-dispatched generic functions do. But even saying “generic functions” is bad enough without having to say “symmetrical predicate-dispatched generic functions” in order to explain how they’re different from say, CLOS or Dylan generic functions.
Maybe I should start calling them “dispatch functions” or “extensible functions” or “rule functions” or “modular functions” or something else like that. Anybody got any ideas?
I feel the same way about predicate-dispatch generics. Although I don’t think I had the enlightenment experience yet. It kind of feels like Calc 101 after studying derivatives for half a semester, right before they break out the Fundamental Theorem. So naturally, I’m looking forward to seeing the more involved functions you end up implementing in p.web.
Im stumped wrt. a name too. “generic” has more name recognition, “dispatch” and “extensible” are OK, I guess. “Composite” maybe? I’ve always thought naming things was the hardest part of software…
I beg of you… please throw together some pseudo-code, 20 lines or so. My Spidey-sense tells me you are talking about something interesting, but without code I cannot be sure.
“””I beg of you… please throw together some pseudo-code, 20 lines or so.”””
Take a look at the peak.security docs and implementation.
Specifically, look at the ‘Context’ class in the implementation, a mere 30 or so lines, containing two generic functions and their default behaviors. They won’t be very illuminating until you look at the docs, though, which contain several examples of adding new methods to the generic functions to create sophisticated access-control rules.
This actually isn’t a particularly sophisticated example of generic function use; indeed it doesn’t even use any features that aren’t found in CLOS’ generic functions. But I think it provides a very good example of the benefits of generic functions’ modularity and extensibility.
The most surprising thing about the code is how little “there” is “there”. If you look at the implementation, it’s like, “is that all there is?” Especially if you had seen the previous version with twice the code and seven more interfaces needed to do the same darn thing with a bit less flexibility.
“””I don’t think I had the enlightenment experience yet.”””
The tricky part is that there’s nothing to be enlightened about. Or at least, if there is, I don’t know what it is myself. 🙂
If anything I would call it an “UNlightenment” experience, because you sort of have to forget how hard it is to program extensible, maintainable frameworks, and just look at the simple aspects of a problem.
Interfaces were a big step in that direction for me, or more precisely, adaptation was. Adaptation lets you say, “What’s the *ideal* interface for this operation to depend on?” And then you worry about *getting* that ideal interface later.
With generics, you get to take one more step in that direction. Just skip the interface and define a function for the ideal operation. Then add implementations to it for any specific situation you care about.
With predicate dispatch, you go another step, and just define ideal operations that represent what your program *does*. So, the peak.security rewrite went like, “Okay, what does this framework do? Well, it lets you decide when a user has a permission for an object in some context, and it lets you decide what permission is needed to access an attribute in a given context. Okay, so that’s two generic functions: hasPermission(context,user,perm,subject) and permissionFor(context,subject,attrname).” And boom, that’s more or less the whole thing, except for writing ten times as much additional text to document and test it. 🙂
Yeah, there are a few lines of other glue code, to hook it into the PEAK metadata framework and to implement “Denial” messages, but like 80% of the framework’s functionality is just, “hey, call one of these functions, or add a case to it if you need to customize it.” (That’s why this post is titled, “Like Acid for Frameworks”; this approach seems to dissolve away anything that looks like a “framework” into “well, just use a generic function and get on with it.”)
peak.web is going to be more complex than peak.security, though, because there are a lot more “built-in” cases that will have to be included. However, nearly all of that code already exists and can plug in largely unchanged; I’ll just be sticking decorators on the functions and possibly yanking them out of classes. The more interesting part (to me, anyway) is all the degrees of freedom that peak.web will gain, and the decrease in concepts that you have to know about in order to implement sophisticated apps with it.
For example, instead of needing to know to implement the IWebTraversable interface and include a beforeHTTP() method if you want to do something like require a login or session cookie in some part of your app, you would just add a “before” method to the doHTTP() generic function, that says something like “when you’re rendering an object of this kind (or in this location) and there’s no session established, set one up”. In other words, just say what you mean. 🙂
So, just by learning a few core objects (traversal context, interaction policy, and environment), and a few generic functions, it should be possible for someone to implement very sophisticated application rules. And, the rules themselves are modular, so you can have a basic application and then plug in rules to customize it for a specific installation, without modifying the base code.
Ah well, I’m ranting (raving?) again, so I’ll stop now. 🙂
You said: “you would just add a “before” method to the doHTTP() generic function”
How would this call the base doHTTP functionality, or would I have to replicate the code? (Apologies for the OO-terminology, but I’m not sure how to refer to the concepts in generic-function-land. A more OO way of asking the question would be “how do I call super.doHttp()?) Do all the variants of the function with satisfied predicates get run? If so, in which order? What if one of the variants modifies a predicate that another depends on?
I can certainly see the coolness of generic functions, and where (or at least one place where) they would be useful. They’re like overloading on steroids. Have you noticed any drawbacks to using them? (Hey, how about “dynamically-overloaded functions” as a name, or “runtime-overloaded functions”?)
Later,
Blake.
The security example is quite compelling, if a little frustrating that it does indeed take much more effort to explain than to implement 😉 But, unlike a lot of other times when I’ve tried to understand pieces of PEAK, it’s easy to understand without understanding other basic concepts in the framework. I guess it mentions bindings a bit, but that’s it.
Definitely keep throwing up code snippets, especially how to do things we’re familiar with in new ways.
“””If anything I would call it an “UNlightenment” experience, because you sort of have to forget how hard it is to program extensible, maintainable frameworks, and just look at the simple aspects of a problem.”””
How about “enlightened functions” 🙂
I hear you though – in the places where I’m currently phasing in generic functions to replace now-superflous subclassing, I have to force myself to think simply, and try not to make things complex out of habit. It’s actually *is* kinda zen… letting go of attachment to form and all.. 🙂
– Andy Gross (original anonymous commenter)
“””Do all the variants of the function with satisfied predicates get run? If so, in which order?”””
PJE can probably give a more complete answer, but you can define whatever behavior you want in the presence of multiple statisfied predicates. You write a “method combiner” which takes a list of the matching functions and executes/returns however/whatever it wants… from the PEAK list (although the interface for this may change soon, I’m told.
”’
Just make your own combiner, and use that as the argument to dispatch.generic:
def my_combiner(funcs):
def combined(*__args,*__kw):
# Call each of the applicable methods
return [f(*__args,*__kw) for f in funcs]
return combined
@dispatch.generic(my_combiner)
def gfunc_that_returns_list_of_results(…):
…
”’
“””How would this call the base doHTTP functionality, or would I have to replicate the code?”””
It depends on what method combiner is used, but if you used the things that are available right now in CVS, there’s a ‘call_next_method(…)’ function that you use.
I’m thinking about changing the method combiner machinery a bit, to support some advanced use cases like pattern matching, and in the process I may change it so that chainable method combiners use an extra “next_method” argument instead of a global “call_next_method” function, because I think I can squeeze out more performance that way. (Because if it’s the first argument, I can chain instancemethod wrappers to do it, effectively doing the chaining in C code. But that’s an implementation detail.)
“””Do all the variants of the function with satisfied predicates get run? If so, in which order?”””
This is determined by the method combiner for that generic function. The current default method combiner invokes only the method with the most-specific predicate. There is an alternate method combiner that allows you to use “call_next_method”. And you can write custom combiners that do whatever you want. The combiner is supplied to the ‘dispatch.generic()’ decorator on the function prototype. That is, ‘dispatch.generic(foo)’ as the decorator means that ‘foo’ is the method combiner that will be called to create a combined method.
The combined method must be a callable with the same signature as the original function, and it should be reusable, as it will be cached for use any time the exact same set of methods applies to an invocation.
So, obviously, the trivial default method combiner just returns the most-specific method. The chaining combiner returns a function that wraps a call to the most-specific method, with a frame that holds an iterator over the remaining less-specific methods, where “call_next_method” can get at it.
“””What if one of the variants modifies a predicate that another depends on?”””
Well, you’re screwed, just like if you call super() after breaking a superclass invariant. 🙂
More seriously, all the predicates are tested simultaneously, before *any* of the methods are invoked. By the time the first method is invoked, the generic function has already chosen the set of “applicable methods” – i.e., methods that were applicable when it picked them. You can change whatever you want, but the methods are already selected.
Of course, you can always *not* call the “next” method, and instead recursively invoke the function so that it selects methods according to the *new* state of things, or according to different arguments or whatever.
Finally, note that for some custom method combinations, you can’t stop the other methods from executing, because you’re not responsible for calling them. For example, a set of pricing rules might sum up the various prices involved, and you’d have a ‘computeCost’ generic function with a ‘sum’ method combiner that returns a wrapper function that calls all the applicable methods and adds their return values together.
So, really, the answer is, “it all depends on the method combiner you use”, which of course you’ll need to select (or create) based on the desired semantics of the operation you’re performing.
I’m hoping that by PyCon time I’ll have a nice suite of simple method combiners and a lightweight framework for creating fancy ones for various purposes.
“””Have you noticed any drawbacks to using them?”””
As compared to….?
“””Definitely keep throwing up code snippets, especially how to do things we’re familiar with in new ways.”””
Have you looked at the PEAK option parsing docs? That subsystem is also generic-function based, but not in a way that’s exposed to the user. And, it doesn’t rely on knowledge of the rest of PEAK; it just depends tangentially on the binding.metadata API, in much the same way as the security stuff does. It’s just that instead of declaring permissions for attributes, you use it to declare command-line options. You can actually use both at the same time (i.e. options and permissions) for the same attributes, although I can’t think of a use case for doing so. 🙂
Also, there’s some similar documentation/tests for the attribute metadata system, which is also (single-dispatch) generic-function based.
Forget finding a new, better name for “generic functions”. The answer is just do the last 20% of “dispatch”, doing what is needed to get it ready for “prime-time”. Of course, the last 20% will take 80% of the effort…
Because of “itertools”, you don’t have to explain generators to a working programmer. “itertools” is the best way to work with a whole class of problems, and its scope and quality motivates a working programmer to learn generators inside and out.
By “working programmer” I mean “any programmer motivated enough to cast off superstitions, fetishes, and security blankets when they stand in the way of an economic solution”.
The opposite of a “non-working programmer” or “broken programmer”.
“””Of course, the last 20% will take 80% of the effort…”””
Heh, it’s more like the last 80%… Things on my to-do list for the dispatch package currently include:
* Documentation
* Predicate functions (shortcuts for reusing similar conditions)
* Easy framework for custom method combiners, with CLOS-style qualifiers
* optimization for idioms like .startswith(), issubclass(), etc. (These consist mostly of adding some rules to the generic function that’s used to optimize generic functions, and adding some code contributed by Radek Kanovsky for an IsSubclass index extension.)
* C speedups for certain inner loops in the decision tree generation code and the expression interpreter
* Pattern matching and variable binding (very useful for compilers and other tree transformations)
* Totally-ordered classifiers
* More documentation
Of course, the implementation as it sits today is pretty close to production quality anyway, but the above “20%” should make it world-class.
There will probably be a couple of API changes to get the combiners framework and the variable binding stuff to work right, though. Currently, method combiners don’t get access to the generic function itself, and there’s no way to pass extra computed arguments to individual methods (which is needed for pattern matching – sort of like regular expressions for arbitrary data structures instead of strings).