On one slide you lump in DirectoryStorage as a way to boost ZODB performance. Now I know Toby is a genius in his own right (anyone developing a full ZODB data storage backend is), but DirectoryStorage is still woefully slow compared to good old FileStorage. They were using DirectoryStorage at the place where I work right now, and one of the first things we did was to get rid of it. Not only did we boost the application speed, we also finally got around all the typical other slowdowns you get when you store millions of little objects on the file system. Such as super-slow backups, buildouts, sandbox removal, etc. Everyone is much happier now.
reliability booster
Posted byslinkpat
2005-07-22 03:02 PM
I haven't noticed any slowdown due to directorystorage. But our usage patterns are read-heavy and write-light.
For that usage, as far as I can tell the impact of DirectoryStorage is dwarfed by the impact of ZEO across a network. Once stuff is in the ZEO client cache, it doesn't really matter what the underlying storage is anyway -
unless you are doing heavy writes.
But performance is not why we switched to DS. We frequently had filestorage corruption (some of which were AFAICT caused by since-fixed bugs). Nightly runs of fsrefs and fstest frequently mailed me problems that would mysteriously go away after a pack. Many of these were never explained. Frankly after a couple years of sporadic hell, i.e. every few months wasting a day searching for causes and solutions to nasty POSKeyErrors, we just had little confidence in the system. Since moving to DS on the same hardware, we have had exactly zero storage issues in the past year and half. Not one ever.
Also I don't think repozo existed back then, and I was longing for an incremental backup feature, which DS has out of the box.
However, your other comments are spot-on. If you have a big (several GB) DirectoryStorage and don't pack it for a while, it can take all night to make a full backup, and longer to pack. Even with frequent (weekly in our case) packing, a 3 GB storage takes about an hour to back up on our (somewhat old) hardware.
DS also has some other nice features. The replication feature can be handy. You can't do warm failover, but you can pretty cheaply maintain a ready-to-go backup storage on a remote system and use that for cold failover. I'm not aware of any free way to do that with Filestorage.
For that usage, as far as I can tell the impact of DirectoryStorage is dwarfed by the impact of ZEO across a network. Once stuff is in the ZEO client cache, it doesn't really matter what the underlying storage is anyway -
unless you are doing heavy writes.
But performance is not why we switched to DS. We frequently had filestorage corruption (some of which were AFAICT caused by since-fixed bugs). Nightly runs of fsrefs and fstest frequently mailed me problems that would mysteriously go away after a pack. Many of these were never explained. Frankly after a couple years of sporadic hell, i.e. every few months wasting a day searching for causes and solutions to nasty POSKeyErrors, we just had little confidence in the system. Since moving to DS on the same hardware, we have had exactly zero storage issues in the past year and half. Not one ever.
Also I don't think repozo existed back then, and I was longing for an incremental backup feature, which DS has out of the box.
However, your other comments are spot-on. If you have a big (several GB) DirectoryStorage and don't pack it for a while, it can take all night to make a full backup, and longer to pack. Even with frequent (weekly in our case) packing, a 3 GB storage takes about an hour to back up on our (somewhat old) hardware.
DS also has some other nice features. The replication feature can be handy. You can't do warm failover, but you can pretty cheaply maintain a ready-to-go backup storage on a remote system and use that for cold failover. I'm not aware of any free way to do that with Filestorage.
Replies to this comment