PyMeld2: A Proposal for Next Generation Templates for Python Web Apps
Author: Paul Winkler, http://www.slinkp.com
This proposal grew out of a discussion on plope.com
I have removed links to the experimental Meld2 code. It has been supplanted by Chris McDonough's Meld3
Disclaimer
This page is a proposal only. It is not documentation for any specific implementation such as Meld3.
Rationale
Every web framework on the planet (and some standalone renegades) has a template system. These generally take one of two approaches: Implement every feature you could want in the templating language (a la DTML, ZPT), and/or require you to embed Python in the template (Kid etc).
IMHO these approaches lead to much complexity and too many new concepts for newbies to learn; and the templates tend to be unfriendly for round-trip collaboration with designers (ranging from "impossible" for DTML to "possible but still a bit scary" for ZPT).
The third approach is to radically simplify the template language and offload the work into some associated python code. (You can of course take this approach in any template language, just by using as few of its features as possible and relying on python code to do the heavy lifting) but in practice the temptation to take the easy / sloppy route means that you will have to work with other peoples' nasty, unreadable, poorly-factored crap a lot of the time.)
There are three examples I know of of the third approach.
- Nevow templates are pretty clean, but the python API for working with them introduces a bunch of new concepts and the docs are very hard for me to grok. I think we can do better.
- HtmlTemplate is a pretty decent balance, pretty clean but there's still a tiny bit of control on the template side and I think we can get something even cleaner, more general, and easier to learn by eliminating that. But I think we can probably glean some ideas from it.
- PyMeld is radically simple - it isn't a template language at all really, the templates are pure (X)HTML, you only rely on the "id" attribute and all work happens in python. This smells like the right direction to me. Unfortunately, as provided it's just a bit too primitive for real use, and has a couple shortcomings I'll mention below, all of which are fixable IMO.
So, I propose to start with a PyMeld-like approach, but fix its shortcomings.
For purposes of this document I'm calling the project "Meld2". I thought of calling it "Teng", for "Template Engine Next Generation", but there's already a template system at teng.sourceforge.net! (It looks to be a minor extension to existing systems, e.g. PHP.)
Goals:
- Templates can be (required to be?) valid XML / XHTML.
- Templates are dumb and indicate no behavior whatsoever.
- No embedded code of any sort.
- Absolutely minimal extensions to html - just attributes to identify elements for the python api.
- All heavy lifting done in Python.
- Highly portable: nothing specific to Zope or any other platform.
- Easy to document.
- Simple implementation. (No bytecode compiler!)
- It should be possible to process templates in multiple passes, for those who prefer a "pipeline" approach to page generation. In other words, the output should always be valid input.
So far, PyMeld already satisfies all of the above. Where PyMeld falls down:
- Python API should provide the important common features of other templating systems, so you don't have to reinvent the common case. This could be done as a layer on top of PyMeld. And all these features would be optional; they're just convenience methods, you can ignore them if you wish.
- Python API must explicitly distinguish between sub-nodes and attributes. IMO this is a foolish DWIM in PyMeld, its first small-but-serious flaw.
- The attributes we use for finding nodes should live in an XML namespace so as not to further overload html IDs (which are already used by css & javascript). This fixes PyMeld's second small-but-serious flaw.
Possible example:
<p meld:id="foo">...</p>For flexibility, and backward compatibility with existing Meld templates, the attribute to use could be configurable (HtmlTemplate does this, I think it's a nice idea).
- The repeat implementation should generate new ids for repeated elements, so each one is still (locally) unique; so you can process your template in multiple passes if you like (PyMeld makes you handle id-uniqueness explicitly.) My first idea for this: If you repeat an element like this one:
<p meld:id="foo">...</p> ...you could end up with elements like these:: <p meld:id="foo/0">...</p> <p meld:id="foo/1">...</p> We should provide a simple API for finding these later, or for getting all elements with ids like foo/N.- Maybe IDs must be unique at local scope (i.e. within the parent element), but should not need to be globally unique within the template? Unfortunately, global uniqueness is a wart in HTML ids that doesn't fly when you're recomposing fragments of XML. For reasons that remain unclear to me (ease of implementation?), XML has the same stupid requirement on all elements of the ID datatype.
What I would like is if this were legal in Meld:
... <div meld:id="foo/0"> <div meld:id="bar" /> </div> <div meld:id="foo/1"> <div meld:id="bar" /> </div> ...But this would NOT be legal, since there are two identical IDs in the same scope:
... <div meld:id="foo/0"> <div meld:id="bar" /> </div> <div meld:id="foo/1"> <div meld:id="bar" /> <div meld:id="bar"> This is an illegal duplicate </div> </div> ...The downside is that finding a specific "bar" in that fragment requires you to know what its parent is, which increases our dependence on a particular document structure and this tends to increase brittleness (see comments on Casey's tal_inheritance proposal).
There's an implementation wrinkle. In the meld implementation, which presumably uses an existing XML library, meld:id cannot be implemented as the XML ID type.
The reason is twofold, or rather, there are two aspects to it: The ID datatype mandates uniqueness in the scope of the document, and an element can have only one attribute of the ID type, even if they are from different namespaces, because all ID attributes effectively share one namespace.
That smells awful to me. It seems shortsighted (what's so special about documents? don't we combine documents all the time? how does xslt handle this?), but that's what XML spec says. Sigh.
So, meld:id IS NOT AN ID in the XML sense.
The obvious workaround is to simply say fine, meld ids are not IDs and we don't care :-)
- Casey Duncan's extension idea: It should be easy to twiddle stuff the template author didn't anticipate. Proposed as TAL inheritance
I have not used xpath, but it (perhaps unfairly) makes me nervous. I tend to think that walking and arbitrarily twiddling the DOM will be fragile, so I'm skeptical that this proposal as written is really practical. It's as if a subclass attempted to extend its parent by saying "I want to replace the third line of method foo() with this line, and insert some other lines at the top of the fourth "if" block." You become coupled to the structure of the base code and things become brittle. This is roughly analogous to the problems identified by the "Law of Demeter", e.g. at http://www.ccs.neu.edu/home/lieber/LoD.html But maybe appropriate use of relative xpath statements would help.
Worse, I already think ZPT is too complicated, and adding yet another syntax (xpath) only makes that worse.
The METAL extension in Zope 3.1, which Shane pointed out at http://www.zope.org/Wikis/DevSite/Projects/ZPT/MacroExtension provides a more limited and controlled kind of extension - you can quickly and cleanly decorate a slot with new slots before and after it, but you can't arbitrarily muck around with the middle of it. And the syntax is a lot simpler, no xpath, just one more metal attribute (metal:extends). This is a win for existing ZPT projects, but again, ZPT is already too complex.
In PyMeld or Meld2, hopefully a combination of factors will go a long way toward encouraging a higher level of granularity for extension and reuse:
- It's really easy to mark nodes!
- Exactly the same markup is used for all templating features, so once a node is marked you can do anything with it.
- It's really easy to recompose nodes into a new template object.
Risk factor: ID noise - when you look at the template, you have no idea which IDs matter, and for what purposes. On balance I think this is OK if we can make the python code sufficiently self-evident (i.e. not like Nevow).
Note: In Nevow's favor, it appears to use exactly the same "stan" DOM to represent both parsed templates and python-generated pages, which means you can start a view as a python-generated stan object and later replace it with a template if you need more control over the markup. Or vice versa. AFAICT any code that works with that page, e.g. render and data methods, doesn't change at all when you do this. This is a very nice feature.
Common Features of Templating Languages
This is a sort of catalog of things that should be taken care of by the python API for meld2. XXX put HTMLTemplate in here too.
- Reuse and extension.
- ZPT: metal, or structure insertion
- DTML: dtml-var (which means each template is an indivisible unit)
- Kid: py:def and py:extends
- Nevow: not sure, but see Fragments and Macros at http://divmod.org/trac/browser/trunk/Nevow/examples
AFAICT it's easier in PyMeld: You can just recompose nodes arbitrarily for reuse, modify a clone of a node for extension, etc. The total generality of named nodes means that if I mark an element because e.g. I want to repeat it, other people can come along later and decide to use it for extension, or as dummy content to be stripped, or whatever.
- Repetition.
- ZPT: tal:repeat
- DTML: dtml-in
- Kid: py:for
- Nevow: marked with nevow:pattern, repeated in python code.
Note, a wrinkle worth thinking about is the need to wrap two or more elements that you want to repeat as a unit. It's annoying to end up with all those extra "span" tags wrapping things. In ZPT you can remove it with tal:omit-tag; in Kid, use py:strip. In DTML you don't need anything because the dtml-in tag is not delivered to the client. But of course that means the template is invalid (X)HTML.
This is a minor gripe, I probably won't invest much thought in it.
- Conditionals.
- ZPT: tal:condition can be pretty clunky to use, you're often better off doing all the work externally and just rendering whatever you get.
- DTML: dtml-if
- Kid: py:if
- Nevow: all on the python side AFAICT.
We get this for free by using python.
- Replacing element content, or the whole element.
- ZPT: tal:content, tal:replace
- DTML: dtml-var
- Kid: py:content, py:replace
- Nevow: nevow:render, nevow:data, nevow:slot
In PyMeld, assigning to a node replaces its content. del() on a node removes it from the document. By combining those you could effectively do replacement.
I may prefer to make these more explicit using method calls rather than operators.
- Replacing/inserting/removing attributes.
- ZPT: tal:attributes
- DTML: &dtml-var;
- Kid: py:attrs
- Nevow: nevow:attr
Need to define an API for this. In PyMeld, attributes and named sub-nodes live in the same namespace and I want to disambiguate that.
- Falling back to original content.
- ZPT: the "default" binding
- DTML: N/A
- Kid: ?
- Nevow: ?
This is just not a problem in PyMeld / Meld2. E.g. if you need to mutate something and then under some conditions restore the original, you just use trivial python techniques for that.
- Error handling.
- ZPT: tal:on-error
- DTML: dtml-try/dtml-except
- Kid: ?
- Nevow: ?
I think we get this for free too. A typical approach would be to have a node in the template named e.g. "error_some_descriptive_blahblah", and then some trivial exception handling in your python code decides whether to remove this node.
- Custom tags.
- JSP: taglibs.
- Kid: py:match
- ZPT: N/A
- DTML: N/A
- Nevow: ?
Custom tags are antithetical to the PyMeld approach, only listed here for completeness.
- Inserting in free text without tags.
- ZPT: N/A, you need to add a tag and use tal:replace.
- DTML: dtml-var foo and &dtml-foo; can be placed anywhere.
- Kid: String expressions can go anywhere, I like that.
- Nevow: N/A
I'm not sure yet how much I care about this, so it would probably be left out of version 1.
One approach is that the parser could recognize text like ${meld:foo}. That could be treated identically to but the tag itself would have no text. It wouldn't have child nodes either (unless you add some programmatically.)
As a bonus, it would make the Meld2 API useful for processing non-XML text documents, e.g. all the things that DTML is still used for in Zope: ZSQL, email, stylesheets, etc.
- Local variable definition.
- ZPT: tal:define
- DTML: dtml-let
- Kid: ? I know you can define things in python code blocks, but I'm not sure what their scope is. o Nevow: ?
We obviously don't need this.
Summary of Feature List
After looking through that list, it may be that the only thing that PyMeld doesn't provide enough help for is repetition. But repetition is really important.
Repetition: Casey Duncan's proposal
(from http://plope.com/Members/casey/tal_inheritance/talkback/1131264921)
Casey writes:
...each element in the template has a method 'zip_repeat(iterable)'. When called this method returns a generator which first removes the template element from the document, then yields tuples of '(element.clone(), iterable.next())'. Each time a cloned element is yielded, it is inserted in the document right after the last element (the first element is inserted where the original element was). The result is similar to "tal:repeat", especially if you iterate it in a for-loop. Here's an example. Here is a snippet of the document (for now we continue overloading the tag "id" for simplicity):: <table> <tr> <th>Title</th> <th>Description</th> </tr> <tr id="doc_item"> <td id="title">Document Title</td> <td id="description">Document Description</td> </tr> </table> Here is the python "transform" to populate it, imagine that we have loaded the above into a Template instance named "template". __getitem__() on Template objects returns the element with that id:: for tr, doc in template['doc_item'].zip_repeat(context.listFolderContents()): tr['title'].content = doc.getTitle() tr['description'].content = doc.getDescription() So iterating zip_repeat() yields clones of the original table row preplaced in the DOM where you want it, along with each document in the folder so you can modify the row content or anything else you want. When rendered, it might look something like this:: <table> <tr> <th>Title</th> <th>Description</th> </tr> <tr id="doc_item"> <td id="title">Foo Document</td> <td id="description">The Document of Foo</td> </tr> <tr id="doc_item"> <td id="title">Bar Document</td> <td id="description">The Document of Bar</td> </tr> <tr id="doc_item"> <td id="title">Baz Document</td> <td id="description">The Document of Baz baby!</td> </tr> </table> Probably the biggest problem I see with this is that we have multiple tags with the same "id" value which is technically not allowed. So that's probably a vote for using a custom namespace for identifying elements used by templates. Either way the ids will probably need to be munged to keep thing unabiguous in cases where a pipeline of template "transforms" like above are employed.(End of quote from Casey.)
The last paragraph matches my conclusions too. If we do things as I proposed above in the notes on repeating, the input would look like:
<table> <tr> <th>Title</th> <th>Description</th> </tr> <tr meld:id="doc_item"> <td meld:id="title">Document Title</td> <td meld:id="description">Document Description</td> </tr> </table>and the output of Casey's code would look like:
<table> <tr> <th>Title</th> <th>Description</th> </tr> <tr meld:id="doc_item/0"> <td meld:id="title">Foo Document</td> <td meld:id="description">The Document of Foo</td> </tr> <tr meld:id="doc_item/1"> <td meld:id="title">Bar Document</td> <td meld:id="description">The Document of Bar</td> </tr> <tr meld:id="doc_item/2"> <td meld:id="title">Baz Document</td> <td meld:id="description">The Document of Baz baby!</td> </tr> </table>Implementation Status
Meld3 implements pretty much everything here, although the API has changed a bit, so it doesn't look quite like Casey's example anymore, and it does not implement the auto-fixing of repeated ids.
Send me some mail slinkP home page ![]()