Skip to content.

plope

Personal tools
You are here: Home » Members » chrism's Home » In Defense of Zope Libraries
 
 

In Defense of Zope Libraries

A much too long defense of Pyramid's use of Zope libraries.

I spend more time than I'd like responding to this sort of question in the #pyramid IRC channel on Freenode:

  <someguy> i heard that pyramid is based on zope.  zope is bad.
            why is pyramid based on zope?

Pyramid uses libraries that have "zope" in their names. It uses exactly two of them, in fact: zope.interface and zope.deprecation. But as evidenced by IRC questions like the above, some people think "zope==bad" and therefore, transitively they feel like they should be able to reach the conclusion "pyramid==bad", or at least that "removing zope" might make Pyramid better. I understand that folks naturally very badly want things to be black and white ("zope==bad, pyramid==zope, pyramid==bad"), but real life is more complicated. I'd also much prefer it if the story really was that simple, it would be a lot easier to explain. But it isn't, and I'm asked questions like the above so often that it's clear I need to have something to point people at. Yet, "if you're explaining, you're losing." So I'll tell a story instead.

The year: 1999. The place: A software consultancy in Philadelphia PA, US. A twenty-something software developer with not much programming experience needs to write an intranet application for a local power company. He is very green. He is in way over his head. He knows enough Perl to be incredibly dangerous. He starts to write the system in Perl using mysql as a backend. Every function in the system looks something like this:

  sub get_domains {
      my $r = Apache::Request->new(shift);
      my %cookiejar = Apache::Cookie->new($r)->parse;
      # ... check the cookies for a user id ...
      # ... check the userid against a table in the database that
      # ... says whether the calling user is allowed to get domains
      # ... if not, redirect to login page.
      # ... if so, do some inscrutable SQL statement to get the domains
      # ... and render them into HTML.
      }

This does not go well. On the internet on a Friday afternoon, in the throes of developing what is turning out to be a mod_perl clusterfuck of epic proportions, the developer sees on Slashdot that a new system named Zope 1.10, written in a competing scripting language named Python has been open sourced. It advertises itself as an "application server", and the screenshots show pretty administrative pages with security-related stuff on them. He decides to try it out. He downloads an installer for it on his Windows NT computer, installs it by double-clicking, and follows some very basic instructions to see its administrative interface in his Netscape Navigator 4 browser. He is immediately blown away by three things: a) you can write code using your web browser, b) its security model is very advanced, so you can even let untrusted people write code using a web browser and c) the security stuff in Zope is declarative, which means all the boilerplate security-checking he has been doing in his mod_perl app stuff can go away if he buys into its security and data model. It doesn't even need to use a separate relational database; it has its own object database built in. The UI lets him see a graphical representation of both the data in the database and the code arranged as a tree. It's an amazing revelation to him, and he knows he can write this system a lot faster and a lot easier if he takes advantage of the features it's giving him. There is just nothing comparable to it available that could make someone at his skill level as productive.

So he takes a chance. He dumps all his mod_perl code. Without knowing any Python at all, picking up Zope information from the source, and from random maillist discussions, he cobbles together a very basic intranet system using Zope. He doesn't know anything at all about writing tests, and no one at his company values testing very much, so he doesn't write any. And despite the through-the-web coding model, and despite not knowing any Python before setting out to do the job, what he comes up with is cleaner than what he might have come up with in mod_perl, if only because a) he can rely on the declarative security system in Zope, b) a UI exists that he can just point the customer at if they need to change security and other settings and c) the system has a much more granular security model than anything he might have come up with himself; the security stuff is context-senstive, so if someone adds a document to the intranet, the document itself can carry its own authorization settings without much trouble. He presents the result to the customer; they are happy and they pay. When a new version of Zope, version 2.0 is released a very short time later, he ports the code to it with relative ease.

He does a few more Zope jobs, but he falls out of favor with the Philadelphia company because they want to concentrate on doing Microsoft stuff. He subsequently through some cosmic miracle winds up with a job at Digital Creations, the creators of Zope itself. Life is good for a few years. He writes Zope 2 applications for a variety of customers and learns from his (much, much more experienced) peers and managers at Digital Creations what real software engineering means.

Over those years, however, the Philadelphia developer notices a few things about the software development industry. He notices that if it behooves the industry temporarily, it will tolerate things that are slightly different, but it always holds its nose while doing so and will always retreat back to the familiar at first opportunity. The industry is suspicious of all databases that are not relational. In particular, it is deeply suspicious of object databases due to what proves to be over-hyping of Smalltalk in the late 80s and early 90s. It also does not like things with poor documentation, he notes.

Some suboptimal things about Zope also become obvious to him. It becomes clear that Zope itself needs some automated testing, because people with varying skill levels (including him) are adding questionable stuff to it at a dizzying rate without much quality control, and it is suffering as a result. It also becomes clear that the "through the web" development model that so fascinated the Philadelphia developer has a lot of flaws. It's very difficult to write automated tests for code written this way. And because the TTW model is so deeply ingrained in the system, it actually makes writing code that doesn't want to care about the through the web model much more difficult. The mere existence of a TTW security model and the expectations of users about the administrative interface makes writing some types of more advanced code a chore. It has painted itself somewhat into a corner, because the very things that attract new Zope users are the things that tend to repel them after they've had initial success with the system. In order to write testable code, the Digital Creations engineers cannot use the TTW development model, but writing non-TTW code in Zope is kind of a pain in the ass. It becomes obvious that either the TTW coding model needs to die or that much, much better tooling needs to be developed to support it. Nobody has the time to write a significantly better toolchain than ones already possessed by UNIX, however. Disillusionment about the programming model is felt within and outside the company.

Despite all this, Zope is quite successful in 2001. It wins the "Jolt Product Excellence Award" from Software Development Magazine in 2001 (from the award text... "products that win "have "jolted" the industry with their significance and made the difficult task of creating corporate software faster, easier and more efficient.") and is featured on that magazine's cover. Major media corporations launch sites built on the platform; it is perfect for them, and Digital Creation enjoys a good deal of consulting success, becoming more profitable. The main Zope maillist is getting about 100 messages sent to it per day. The joke around the office is that the company should use the reception of an email to the maillist as an event trigger for critical services: it's like clockwork that new messages come in at all times of day and night. New systems pop up all the time based on Zope, including the Content Management Framework (on which Plone was, and is, to this day, based).

Somewhere around this time, the Philadelphia developer watches as work starts on the next version of Zope, named Zope 3. Zope 3 will be entirely incompatible with the older Zope 2. The promise is that some sort of compatibility layer will be developed later, but Zope is redesigned and reengineered from the ground up. It will fix everything. Its engineering is taken very seriously within Digital Creations. The original primary author of Zope 2 dedicates himself to it more or less full-time, and he is granted the services of some of the company's "A Team" developers to help. Zope 3 will be based on a "component architecture", which will allow independently developed and tested components to snap into it, to morph Zope into whatever the developer needs, giving him the possibility to swap off-the-shelf third-party components in and out as necessary, in contrast to the relative brittleness and inflexibility of Zope 2. It will prefer composition over subclassing. It even sprouts its own XML-based configuration language for these components named ZCML. Instead of being a largely monolithic system, like Zope 2, it will be comprised of very small parts that interoperate. The Philadelphia developer is sure it will be great, because very smart people he respects are working on it.

The Philadelphia developer watches on as hundreds of thousands of lines of Zope 3 code are written. The development team learns how to write good tests (although it never does learn how to write good docs). The very best Digital Creations engineers architect the shit out of Zope 3. It slowly turns into a complex system that has a core of beauty, but it's very different than Zope 2, and much, much different than anything else in the web development world. It is supposed to take a year or maybe two. It ends up taking four years to see its first non-beta release. Zope 2 continues to be popular during this time, but many folks are nervous about its future, given the spectre of a Zope 3. Much to the dismay of the Philadelphia developer (who is tasked mostly with Zope 2 consulting projects by Digital Creations during this time), when Zope 3 it sees its first release, it is so different, so hard to approach as a whole, and so poorly documented, that many people get impatient, and start to look for alternatives to Zope entirely, and people with very simple requirements find more suitable alternatives quickly.

Meanwhile, Digital Creations changes its name to Zope Corporation, and slowly begins in fits and starts to cultivate a much more venture-capital-focused business model. It attempts to create products for sale ($$$$$$$$$$) rather than doing straight consulting ($). The company becomes less engineering-focused (and more sales-focused) as a result, and eventually the bottom drops out of the Zope 3 project because the company just cannot justify continuing to carry the expenditure of creating a new Zope when all of its revenue-driving projects still use older Zope 2 technology. Both the Zope 2 and the Zope 3 project founder as their main benefactor benefacts a little less as their attention is consumed by creating products for sale. Open source contributions from third parties to the Zope 3 project are nontrivial, but without the backing of Zope Corporation, they cannot seem to make up the difference.

Zope 3 never sees mass takeup. Some of its technologies, such as its component architecture, are grafted back in to Zope 2. But this only tends to make an already-very-complicated Zope 2 even more complicated. Projects such as Bluebream and Grok use Zope 3 as a base to build higher-level frameworks, but these don't see much takeup either. People are confused about what "Zope" is as the result of five years of brand-muddying. "What is Zope? Is it Zope 2? Is it Zope 3? Is it Zope Corporation? Is it a package on PyPI? Is it a collection of packages?" All of these answers are true. The brand name, as a result, becomes largely meaningless, signifying both everything and nothing at all at the same time. Horrifyingly, as other well-documented and well-engineered systems such as Django become available, the brand rapidly becomes the target of disdain. The triple-whammy of a second-system-syndrome-based failure, a lack of continual support by its largest benefactor, and the muddying the brand allows the word "Zope" to be used, even by those who have never used anything Zope-related, as a stand-in for "overengineering" or "developer hubris" in the larger Python community. The popularity of Zope 2, the original Zope application server, largely blameless except for being still relatively monolithic and crufty suffers badly. The brand has lost most of its credibility, and anything with the word "Zope" in its title starts to become treated with suspicion and it is afforded little respect, even things that are only marginally related to web development at all.

However, despite the branding disaster, the Philadelphia developer knows that some of the technical benefits of reengineering Zope are quite successful. Zope's object database, ZODB, is factored out of Zope entirely and made useful independently. The "component architecture" (which is effectively a fancy superdictionary with a pretentious name) proves useful for doing very fast lookups based on complex inputs. Zope 3's interface system allows for structured duck-typing of objects on an instance (rather than class) basis, and proves somewhat useful for API documentation. Zope's original raison d'etre for the Philadelpha developer, its declarative context-sensitive security system, is muddled completely in the move, and cannot be reused, but the idea is a keeper. Zope's concept of "traversal" over a graph of objects to find a security context proves extremely popular in the Zope 2 and Plone worlds, and has no analogue in competing systems of the time.

So, as you've surely guessed, I'm of course that Philadelphia developer. And I've lived through all that. And I can't say I'm completely happy about how things turned out. It pains me to see Zope get kicked around like this. But a few things were made clear to me during the journey:

  • Most developers are very, very risk-averse. They like taking small steps, or no steps at all. You have to allow them to consume familiar technologies and allow them to disuse things that get in their way.
  • The allure of a completely integrated, monolithic system that effectively prevents the use of alternate development techniques and technologies eventually wears off. And when it does, it wears off with a vengeance.
  • The natural reaction to the problems caused by monolithic systems, which is to make anything and everything uniformly "pluggable" is also a net lose.
  • Tests are very important.
  • Documentation is even more important than tests.
  • When trying to replace a complex system, make it simpler, don't just push the complexity around.
  • Brands are important. You, ideally, must make sure that the question "what is 'noun'" have one answer. Sublety and branding do not mix; they produce a toxic haze of confusion.

Pyramid is heavily influenced by Zope; in many areas, it is a shameless derivative. Without a Zope, there would surely be no Pyramid. But at the same time, Pyramid is a reaction to Zope. We created Pyramid because we came to understand what was suboptimal in Zope and what was great, not because we didn't. While doing so, we happened to use some packages with "zope" in their names to do so, because they were factored out of the Zope whole well enough to reuse. Pyramid is also influenced heavily by Pylons and by Django, but ironically we could not reuse anything from either of those systems very easily, because they weren't factored as quite as well.

Pyramid uses the zope.interface package to provide:

  • Context lookups for declarative context-sensitive security (the ACL authorization system).
  • View lookups based on a context (e.g. exception views).

Pyramid uses the zope.deprecation package to warn framework users of pending API deprecations. It has no dependencies and is only about 100 lines of code that really has nothing to do with "Zope-the-application-server".

Users needn't know or care about either to use Pyramid. The developers of the framework use them because they are answers to actual requirements. We could fork them and change their names, and a Zope-skeptic would just never know. We could even re-release them under a different name and they'd likely be instantly very popular! But forking them just to change their names to mollify those who just want the binary simplicy of "zope==bad" would also be incredibly disrespectful to hundreds of folks who worked very hard on them. Those folks deserve respect and admiration despite Zope's branding tragedies. They've contributed more than I ever have (or ever will) to the Python web world, and possibly more than you too.

So to folks who want to very badly to pre-make the "zope==bad, pyramid==zope, pyramid==bad" argument, I'd suggest that you be brave. Within your first ten minutes of evaluation, considering the stuff I've written above, please try to momentarily entertain the notion that the history of Pyramid's Zope dependencies isn't written entirely by hapless dunces. If you need a concrete positive replacement, try to think of something like this: Mac OS X today ships with zope.interface out of the box and that sort of deployment effects more users than you and me put together plus all of our friends (and maybe even all of their friends) will likely ever influence. Consider that the common wisdom you've gained from Reddit or that guy who used Zope-the-application-server for one day in 2002 may be irrelevant. Please lend us this suspension of disbelief until after you write your first nontrivial Pyramid program. After that, specific critiques are welcome.

Created by chrism
Last modified 2011-12-19 10:46 PM

+1

Awesome

fo' sho'

exactly the right number of words.

+1

Great summary of so many things I have wanted to say to others that attempt to lay on the zope "hate".

First rate

Really excellent post, Chris. You have learned from history, so Pyramid is not condemned to repeat it.

Good history of "Zope"

So much of that is vivid in my memories of the past decade+ ... I never worked at ZC/DC but otherwise I lived it on #zope and the mailing lists.

Thanks for putting that all down, Chris. Great summary of where we are and how we got there.

It's interesting that Django hasn't accrued the same kind of dirty-word status given that it shares a couple important problems with Zope 2.
Some of its continued success is surely due to a much stronger focus on good docs, combined with never losing sight of their original sweet spot.

Obligatory Reddit link...

http://www.reddit.com/r/Python/comments/njgk1/plope_in_defense_of_zope_libraries/

+1

Bravo mcdonc, thanks for telling that story so well.