README for Filesystem Migration Tools

Download

FilesystemMigration 0.1

View CVS

http://cvs.zope.org/Products/FilesystemMigration/

Overview

This Zope product allows you to create "migratable" File objects (or subclasses thereof). A "migratable" File is one that may move its body from ZODB storage to a disk file.

Rationale

Many resources are consumed dealing with large File objects as ZODB data. RAM is consumed unpickling and repickling File body, CPU cycles are wasted doing the same. All of this work adds absolutely no benefit. File-like data can much more quickly be accessed from a real filesystem than from ZODB. Databases are poor filesystems.

Dependencies

• the FileCacheManager product (version >= 1.1)
• Zope 2.7.3 or better (> 2.7.1 might work too, but untested. 2.7.1+ definitely required, however due to a requirement for ZPublisher filestream iterator support).
• An operating system platform with a "/dev/urandom" device. This leaves out Windows, but Linux definitely has it in most recent releases. Unsure about BSD, Solaris, AIX, and so forth.

Installation (single-Zope-non-ZEO setup)

• Install the FileCacheManager product.
• Install the FilesystemMigration product.
• Restart your Zope instance.
• From the ZMI, in the root of your Zope instance, use the "Add..." dialog to create a "Filesystem Migration Tool". It will be named portal_fsmigration.
• Edit its properties as necessary.

  Particularly, on the File System tab, change the "Data Path" and "Tempfile Path" to an empty directory to which the Zope process has write access. You can leave it as "/tmp" but most operating systems clear out the /tmp directory on a reboot, which would be very bad if you'd migrated a bunch of content (it would be lost).

  If you want your content to be migrated to the filesystem as soon as it is added, choose the "Immediate Migration" option (recommended).

Installation (multiple-Zope-ZEO setup)

• do the same thing as for a single-Zope-non-ZEO setup above.
• Ensure you have a filesystem available shared between each ZEO client (NFS, GFS, SMB, whatever) and ensure that filesystem is accessible via the same filesystem mount on every ZEO client system.
• Change the Data Path and Tempfile Path to a directory within this filesystem.

NOTE: This has not been tested; I'd like to hear about how it works for you if you try it!

Usage

USERS

• Create some number of Migratable File instances via the ZMI.

DEVELOPERS

• Subclass MigratableFile as if it were OFS.Image.File in your own Products. Take care to make sure the manage_after and manage_before methods of the MigratableFile objects are called during the course of normal operations (if you override these, call them explicitly from your code).

USERS AND DEVELOPERS

• If you are using the "Immediate Migration" option, your file data will be moved to the filesystem and will never exist in the ZODB.
• If you are not using the "Immediate Migration" option, your file data will exist in ZODB until it is migrated explicitly. You may migrate content to the filesystem explicitly by calling the "fstool_migrateBody" method of each MigratableFile object you've created. For more info, see the source code.
• When migrated objects are removed from the ZODB, their file data is not immediately removed. Use the Invalidate tab of the migration tool to delete invalidated file data.

Features

• Due to the use of FileCacheManager as a basis for the filesystem migration features of this product, transactions are respected (if a transaction involving a migration is committed, the file is migrated; if a transaction involving a migration is aborted, it is not).
• Also due to the use of FileCacheManager, even if you don't immediately migrate content, using a migratable file improves the speed at which you can serve content because the Filesystem Migration Tool is also a cache manager and behaves as a "mandatory" cache manager, saving files to disk and returning them, even for content that has not been permanently migrated to disk.
• File bodies are stored on disk within directories underneath the "Data Path". Each MigratableFile instance has a "filesystem id" and the file path in which it is stored is a function of that filesystem id.
• After a file has been migrated to disk, deleting its corresponding disk file (from a command line, say) will cause the respective Zope object to not have a body. This is usually bad.
• When migratable file objects are copied using the ZMI or via the cut/paste API, the "right thing" happens. For migratable files that are moved, if the body has already been migrated, the file data is not put into ZODB. For migratable files that are copied, if the body has already been migrated, the file data is conditionally put into the ZODB iff the tool's "immediate migration" option is turned off.

Q & A

• Why wouldn't I always want "immediate migration"? That's the whole point, no?

  Yes, it is. The ability to have non-immediate migration allows you to use Zope with many ZEO clients normally and migrate all content once in a while via a scheduled process. This is thought to solve some class of concurrency problems (ZEO handles concurrency well, whereas the code to deal with concurrency in File Cache Manager is not battle-tested).

• There are products like ExtFile and LocalFS. Why bother?

  These products are venerable but they are not "drop in" replacements for OFS.File, nor, to my knowledge, are they aware of transactions. If you delete a MigratableFile during a transaction, only when the transaction commits will the disk data actually be marked as invalid. Likewise, when you add a MigratableFile during a transaction, only when the transaction commits will the file body be stored permanently.

Bug Reporting

• Use the bug tracker at http://www.plope.com/software/collector to report bugs.

Enjoy!

• Chris McDonough (chrism@plope.com)