Casey, I think you hit on a real problem: We want to override things that the original author didn't anticipate.

But it's not inheritance per se that makes your proposal interesting. Consider normal class inheritance.
You can't override some inline code in the middle of a method. If you want to do that, you have to factor it out somehow. The equivalent refactoring in METAL would be to add another slot or macro. Which means modifying
a copy of the original template, as you ended up doing. Kid, for example, allows inherited templates; I've just been looking at the examples, and here's one that shows inheritance can easily suffer from the same wrong-way-round inflexibility as METAL: http://lesscode.org/projects/kid/wiki/IncludeSectionFromOtherTemplateRecipe (see example 2, using "py:extends".) So, inheritance alone doesn't help much.

Your proposal is interesting because it *can* override things the original author didn't anticipate - and this isn't because of inheritance, it's because you give us xpath to grab arbitrary chunks of the DOM.

But, I don't want to add xpath to ZPT. ZPT is already much too complicated! No, really. We have three kinds of TALES expressions, and that's two too many. Plus a macro system. I really really do not want to have to deal with adding another syntax (xpath) into the mix; think of the poor newbies.

The reason I was looking at Kid was because it reminded me of a proposal I halfheartedly made on one of the zope lists a long time ago, and quickly abandoned: What if we ditch string and path expressions and have ONLY python expressions in TALES? That way we don't ever have to add syntax to TALES, e.g. the proposal that was floating around at the time to deal with indeces and keys in path expressions; instead, we could just use plain old Python. No new language for newbies to learn, just a very small handful of xml attributes. I didn't get far with the idea, but this is exactly what Kid does, while preserving some nice things from zpt: valid xhtml, no custom tags.

But looking at Kid has pretty much convinced me that this is not really what I want. Embedded python still smells bad to me. I'd rather move in the direction of simplifying ZPT (or whatever) as much as possible and moving all the work out of the template entirely.

We had a discussion a while back on this same blog, in the "HTML Blows" thread; in that discussion Chris quoted you saying much the same thing as I just did: "Casey Duncan had an interesting idea a while back which sort of turned ZPT "inside out". I'm sure other non-Zope folks have had the same idea over and over again too.. Basically, you would grab an XML document (the "template") and modify it within Python using a simple API. When you were done, you'd tell it to render itself. There is just never any logic in the XML itself, it's just a starting point with maybe a few event hints."

There are some interesting attempts out there, as we discussed in that thread. Nevow takes a step in that direction, but there's still too much work going on in the template itself, and I find it hard to wrap my head around (as ZPT was when I first learned it). HTMLTemplate is similar. And both still suffer from the inside-out problem: you can only customize the things that the template author has decided to mark.

The closest thing to that quote above that I've seen is PyMeld, but as an experiment I started rewriting a complex ZPT in PyMeld to see what it felt like, and I think I know why it hasn't caught on:

1) The python API for it is *too* simple, it has almost no features, which is nice in theory but it leaves you to reinvent every wheel that we take for granted in zpt. This could be fixed with some higher-level python classes on top of pymeld, but those will take time to discover and IMO should really be part of the library.

2) It uses python attributes for both sub-node IDs and for tag attributes. DTML taught us that flattening everything into one namespace is not necessarily good :-) Things will break when somebody adds a node id that clashes with your attribute id, or vice-versa.

3) The only real-world example I know of (and I asked the author) is SpamBayes, which uses PyMeldLite for its UI. And while the template (one huuuuge template) has pretty clean markup, the python code that drives it is hardly a model of clarity. It's full of repetitive ad-hoc gunk, and for sheer lines-of-code-per-method, it can compete with the hoariest corners of Zope. So there are simply no good examples to look at.

4) You have to be careful if you want to make multiple passes, because it's easy to end up with duplicate ids.

5) the "id" attribute is already overloaded: it's used both by CSS and by javascript.
This could maybe be solved by using a custom namespace with a single name, e.g. <p meld:id="foo" />.
That doesn't seem in danger of sprouting ZPT-like complexity, since it has no behavior at all ;-)

Having said all that, PyMeld still doesn't do what you (Casey) want either: You *still* can't get at arbitrary parts of the template. If the template author didn't slap an ID attribute on some tag, tough luck, you can't touch it.

And you know, I'm not so sure that's bad, on balance. I'm actually not totally convinced that this ability to reach in and grab any bit of template would be good for maintainability. Why not? Because once you depend on walking the DOM, you become more and more tightly coupled to the template's precise structure. What if that H1 gets changed to an H2? What if you're looking for "the first <p> after the <h1>" and either or both of the tags changes or moves?

That may still be preferable to maintaining a copy of the entire template, maybe even a whole lot preferable; but it hardly makes for maintenance-free upgrades.