Proposal: Inheritance for TAL
(Note that I started writing this in 2004, and I figured I'm not getting any younger, so I'm just gonna unleash this idea and see what the reaction is. In my present, non-involvement with Zope, I probably won't be implementing it anytime soon — and doing so would be decidely non-trivial — but who knows, maybe the idea has merit and someone will do it or convince me to.)
The other day [more like the other year -ed] I was writing a custom extension to an existing CMF application. In this extension I needed to modify a folder view in a small way (basically just add some new links). In order to make this small change, I needed to override the template in it's entirety, which meant copying the entire template to another skin layer and then making this change. This, of course worked, but it bugged me. If the application changed, or this template was enhanced in some way, I would need to synchronize the custom version if I wanted or needed that change in the customization. In short, that sucks.
It occurred to me that if the underlying template had contemplated such a modification, it could have provided macros and slots that I could use from the custom extension thus sparing the code duplication. Since the application does not do this, however, I am out of luck. This got me thinking. From an OO perspective, this is ass-backwards. Think about classes: although you may give a bit of thought about how a class might be extended later, usually you don't (and shouldn't). You rely upon the ability of classes to be subclassed arbitrarily. The subclass can override methods, add new methods and change attributes to create a new class. The whole idea is to avoid code duplication and therefore achieve some measure of separation whilst still providing for reuse.
ZPT doesn't work like that. ZPT bits can be reused only through direct insertion (one template including another) or through macros. The former case requires factoring pages into many fragments, with the base knowing how to assemble them into a whole. The latter requires the base(s) to provide macros and slots which must already fully contemplate whatever extensibility is necessary at run-time. The higher-level templates use the macros and fill the slots. In both cases the level of extensibility is predetermined and fixed. In the real world this often means no extensibility is available or it is extremely course-grained.
So the obvious answer to this problem is make ZPT work like classes
— The End. —
Ok, so now that the crack's worn off, how the hell are we going to do that? I'm glad you asked.
Let's start with a simple example. Suppose you have a template like this (forget about macros for a moment):
<html>
<body>
<h1>Document List</h1>
<table>
<tr>
<th>Title</th>
<th>Description</th>
</tr>
<tr tal:repeat="item here/listFolderContents">
<td tal:content="item/Title">Document Title</td>
<td tal:content="item/Description">Document Description</td>
</tr>
</table>
</body>
</html>
You need to customize this template in the following ways:
- Change the <h1> to include the folder title
- Add the description of the folder under the title
Let's think about this in terms of inheritance.
- means we we must override an existing tag.
- Means we must add a tag.
Assuming the above template is called doc_list, let's see how a minimal "subtemplate" would look to provide the desired changes:
<html intal:base="here/doc_list">
<body>
<h1 intal:override="xpath:self"
tal:content="string:Document List - ${here/Title}">
Document List - Folder Title
</h1>
<p intal:insert="xpath:self"
class="description"
tal:condition="here/Description|nothing"
tal:content="here/Description">
Folder Description
</p>
</body>
</html>
Let's examine this a bit more closely. On the first line of the subtemplate we encounter this new attribute:
intal:base="here/doc_list"
This is conceptually similar to metal:use-macro in many respects. The first difference is the namepace. The intal namespace is used for all inheritance operations. The operation of intal is (with due respect to Ken) orthogonal to the existing tal and metal namespaces. It supercedes both however, and intal attributes are always evaluated before metal and tal attributes.
intal:base means that we want to create a subtemplate from another template. The base template is located using a standard path expression. As in metal:use-macro the effect is to replace the entire <html> tag and contents with the code of the doc_list template. Also, as with metal we will have the opportunity to make changes to this base by using other intal statements inside.
The next intal attribute we encounter is:
intal:override="xpath:self"
As you may suspect, this directive allows us to override a base template tag in the subtemplate. The attribute value introduces something else new to ZPT: xpath expressions. In this case we have about the simplest possible xpath expression, "self". Unsurprisingly, this expression means that the cooresponding tag in the same position in the base template (the first <h1>) is replaced by this one. So this:
<h1>Document List</h1>
Gets replaced by this:
<h1 tal:content="string:Document List - ${here/Title}">
Document List - Folder Title
</h1>
The latter contains tal attributes that will be further evaluated once the intal attributes are processed. So we have successfully overridden a static <h1> tag with a dynamic one.
The next intal attribute:
intal:insert="xpath:self"
Allows a new tag to be inserted by the subtemplate. As above, we use an xpath expression for the value. This time the expression points to the position in the DOM where the new tag should appear. In this case a new <p> tag gets inserted after the <h1>.
This should give you a bit of a taste for the idea of TAL inheritance. In this simple example we used really simple xpath expressions, but in the real-world more complex expressions would allow for more complex manipulations. I think trying to mirror the exact DOM structure in the subtemplate, as we did above, often won't be practical.
Admittedly there are some murky issues to iron out here, especially in the intal:insert case where the DOM node referred to in the subtemplate won't yet exist in the base. But those are topics for later discussion...
Here are some questions I have come up with so far to ponder:
- Why are you using xpath?
Why not? it's an important piece that is already invented for us. Notice that you must explicitly state you are using xpath though, this is intentional in case we want to add other ways to traverse the DOM (e.g., CSS selectors or some other simplified notation). Probably the biggest problem with xpath is that it is so general. xpath makes it easy to select multiple elements or other types of DOM nodes. We should aim to make use of that power where we can, but not at the expense of code clarity
- Is multiple inheritance possible?
I think so, in a manner of speaking. There is nothing that says that the outer-most tag is the only one that can have an intal:base attribute. I could imagine creating base templates that are simple components (e.g., nav links, calendars, etc) that are meant to be assembled as "mixins" — much like macros are used now. I haven't thought this through entirely yet though, so it may be a YAGNI given that we already have macros
- What if I want to inherit just part of a base template?
We could contemplate adding a second optional argument to intal:base that specifies the tag to use. Then you could write things like:
<div intal:base="here/some_template xpath:/html/body/div[@id=foo]">...</div>
There are some questions about encapsulation to ponder here. Is it really a good idea to reuse arbitrary parts of other templates? It may be too ad-hoc.
- What if I want to override just the content of a tag or the attributes of a tag?
I can envision additional intal directives like intal:override-content or intal:override-attribute, but I'm not sure yet when the complexity is justified. Also, since xpath can refer to attributes and multiple elements at once, they may not be necessary
- I think writing
xpath:selfeverywhere will be tedious, can't we come up with a shorthand for that?
sure, but only once we've demonstrated that it gets used a whole lot
- How much wood would a wood chuck chuck if a wood chuck could chuck wood?
see: http://www.getodd.com/stuf/stupid/woodchuck.html
So that's it for now. I'm certainly very open to further ideas, criticisms and suggestions.
Macro Extension
http://www.zope.org/Wikis/DevSite/Projects/ZPT/MacroExtension
Another proposal: "Pymeld 2"
metal:extends is great for existing ZPT projects, but I still want something simpler and more pythonic than ZPT. So I'm trying to sort out what I don't like about PyMeld and see if it can be fixed. Turns out to be only a couple of things.
Comments welcome, but I haven't set up comments over there, so we can continue to abuse Chris's blog if he doesn't mind ;-)
go on go on...
pymeld2
It's not the cleanest or smartest code; it inherits from PyMeldLite.Meld
and I think I could improve things somewhat if Meld were refactored a bit first.
I've posted it at http://www.slinkp.com/code/zopestuff/templates/Meld2.py
You also need PyMeldLite.py which you can get by downloading Spambayes.
Now I need to play around with it some more and see what needs improving.
btw, I've corresponded with Meld's author and he's interested in this stuff.
But it's not inheritance per se that makes your proposal interesting. Consider normal class inheritance.
You can't override some inline code in the middle of a method. If you want to do that, you have to factor it out somehow. The equivalent refactoring in METAL would be to add another slot or macro. Which means modifying
a copy of the original template, as you ended up doing. Kid, for example, allows inherited templates; I've just been looking at the examples, and here's one that shows inheritance can easily suffer from the same wrong-way-round inflexibility as METAL: http://lesscode.org/projects/kid/wiki/IncludeSectionFromOtherTemplateRecipe (see example 2, using "py:extends".) So, inheritance alone doesn't help much.
Your proposal is interesting because it *can* override things the original author didn't anticipate - and this isn't because of inheritance, it's because you give us xpath to grab arbitrary chunks of the DOM.
But, I don't want to add xpath to ZPT. ZPT is already much too complicated! No, really. We have three kinds of TALES expressions, and that's two too many. Plus a macro system. I really really do not want to have to deal with adding another syntax (xpath) into the mix; think of the poor newbies.
The reason I was looking at Kid was because it reminded me of a proposal I halfheartedly made on one of the zope lists a long time ago, and quickly abandoned: What if we ditch string and path expressions and have ONLY python expressions in TALES? That way we don't ever have to add syntax to TALES, e.g. the proposal that was floating around at the time to deal with indeces and keys in path expressions; instead, we could just use plain old Python. No new language for newbies to learn, just a very small handful of xml attributes. I didn't get far with the idea, but this is exactly what Kid does, while preserving some nice things from zpt: valid xhtml, no custom tags.
But looking at Kid has pretty much convinced me that this is not really what I want. Embedded python still smells bad to me. I'd rather move in the direction of simplifying ZPT (or whatever) as much as possible and moving all the work out of the template entirely.
We had a discussion a while back on this same blog, in the "HTML Blows" thread; in that discussion Chris quoted you saying much the same thing as I just did: "Casey Duncan had an interesting idea a while back which sort of turned ZPT "inside out". I'm sure other non-Zope folks have had the same idea over and over again too.. Basically, you would grab an XML document (the "template") and modify it within Python using a simple API. When you were done, you'd tell it to render itself. There is just never any logic in the XML itself, it's just a starting point with maybe a few event hints."
There are some interesting attempts out there, as we discussed in that thread. Nevow takes a step in that direction, but there's still too much work going on in the template itself, and I find it hard to wrap my head around (as ZPT was when I first learned it). HTMLTemplate is similar. And both still suffer from the inside-out problem: you can only customize the things that the template author has decided to mark.
The closest thing to that quote above that I've seen is PyMeld, but as an experiment I started rewriting a complex ZPT in PyMeld to see what it felt like, and I think I know why it hasn't caught on:
1) The python API for it is *too* simple, it has almost no features, which is nice in theory but it leaves you to reinvent every wheel that we take for granted in zpt. This could be fixed with some higher-level python classes on top of pymeld, but those will take time to discover and IMO should really be part of the library.
2) It uses python attributes for both sub-node IDs and for tag attributes. DTML taught us that flattening everything into one namespace is not necessarily good :-) Things will break when somebody adds a node id that clashes with your attribute id, or vice-versa.
3) The only real-world example I know of (and I asked the author) is SpamBayes, which uses PyMeldLite for its UI. And while the template (one huuuuge template) has pretty clean markup, the python code that drives it is hardly a model of clarity. It's full of repetitive ad-hoc gunk, and for sheer lines-of-code-per-method, it can compete with the hoariest corners of Zope. So there are simply no good examples to look at.
4) You have to be careful if you want to make multiple passes, because it's easy to end up with duplicate ids.
5) the "id" attribute is already overloaded: it's used both by CSS and by javascript.
This could maybe be solved by using a custom namespace with a single name, e.g. <p meld:id="foo" />.
That doesn't seem in danger of sprouting ZPT-like complexity, since it has no behavior at all ;-)
Having said all that, PyMeld still doesn't do what you (Casey) want either: You *still* can't get at arbitrary parts of the template. If the template author didn't slap an ID attribute on some tag, tough luck, you can't touch it.
And you know, I'm not so sure that's bad, on balance. I'm actually not totally convinced that this ability to reach in and grab any bit of template would be good for maintainability. Why not? Because once you depend on walking the DOM, you become more and more tightly coupled to the template's precise structure. What if that H1 gets changed to an H2? What if you're looking for "the first <p> after the <h1>" and either or both of the tags changes or moves?
That may still be preferable to maintaining a copy of the entire template, maybe even a whole lot preferable; but it hardly makes for maintenance-free upgrades.
Replies to this comment