README for FilesystemMigration.
README for Filesystem Migration Tools
This Zope product allows you to create "migratable" File objects
(or subclasses thereof). A "migratable" File is one that may move
its body from ZODB storage to a disk file.
Many resources are consumed dealing with large File objects as
ZODB data. RAM is consumed unpickling and repickling File body,
CPU cycles are wasted doing the same. All of this work adds
absolutely no benefit. File-like data can much more quickly be
accessed from a real filesystem than from ZODB. Databases are
- the FileCacheManager
product (version >= 1.1)
- Zope 2.7.3 or better (> 2.7.1 might work too, but untested.
2.7.1+ definitely required, however due to a requirement for
filestream iterator support).
- An operating system platform with a "/dev/urandom" device. This
leaves out Windows, but Linux definitely has it in most recent
releases. Unsure about BSD, Solaris, AIX, and so forth.
Installation (single-Zope-non-ZEO setup)
- Install the FileCacheManager product.
- Install the FilesystemMigration product.
- Restart your Zope instance.
- From the ZMI, in the root of your Zope instance, use the "Add..."
dialog to create a "Filesystem Migration Tool". It will be named
- Edit its properties as necessary.
Particularly, on the File System tab, change the "Data Path"
and "Tempfile Path" to an empty directory to which the Zope
process has write access. You can leave it as "/tmp" but most
operating systems clear out the /tmp directory on a reboot,
which would be very bad if you'd migrated a bunch of content
(it would be lost).
If you want your content to be migrated to the filesystem as
soon as it is added, choose the "Immediate Migration" option
Installation (multiple-Zope-ZEO setup)
- do the same thing as for a single-Zope-non-ZEO setup above.
- Ensure you have a filesystem available shared between each ZEO
client (NFS, GFS, SMB, whatever) and ensure that filesystem
is accessible via the same filesystem mount on every ZEO client
- Change the Data Path and Tempfile Path to a directory within
NOTE: This has not been tested; I'd like to hear about how it
works for you if you try it!
- Create some number of Migratable File instances via the ZMI.
- Subclass MigratableFile as if it were OFS.Image.File in your
own Products. Take care to make sure the manage_after and
manage_before methods of the MigratableFile objects are called
during the course of normal operations (if you override these,
call them explicitly from your code).
USERS AND DEVELOPERS
- If you are using the "Immediate Migration" option, your file
data will be moved to the filesystem and will never exist in
- If you are not using the "Immediate Migration" option, your
file data will exist in ZODB until it is migrated explicitly.
You may migrate content to the filesystem explicitly by calling
the "fstool_migrateBody" method of each MigratableFile object
you've created. For more info, see the source code.
- When migrated objects are removed from the ZODB, their file
data is not immediately removed. Use the Invalidate tab of the
migration tool to delete invalidated file data.
- Due to the use of FileCacheManager as a basis for the
filesystem migration features of this product, transactions are
respected (if a transaction involving a migration is committed,
the file is migrated; if a transaction involving a migration is
aborted, it is not).
- Also due to the use of FileCacheManager, even if you don't
immediately migrate content, using a migratable file improves
the speed at which you can serve content because the Filesystem
Migration Tool is also a cache manager and behaves as a
"mandatory" cache manager, saving files to disk and returning
them, even for content that has not been permanently migrated
- File bodies are stored on disk within directories underneath
the "Data Path". Each MigratableFile instance has a
"filesystem id" and the file path in which it is stored is a
function of that filesystem id.
- After a file has been migrated to disk, deleting its
corresponding disk file (from a command line, say) will cause
the respective Zope object to not have a body. This is usually
- When migratable file objects are copied using the ZMI or via
the cut/paste API, the "right thing" happens. For migratable
files that are moved, if the body has already been migrated,
the file data is not put into ZODB. For migratable files that
are copied, if the body has already been migrated, the file
data is conditionally put into the ZODB iff the tool's
"immediate migration" option is turned off.
Q & A
- Why wouldn't I always want "immediate migration"? That's the
whole point, no?
Yes, it is. The ability to have non-immediate migration allows
you to use Zope with many ZEO clients normally and migrate all
content once in a while via a scheduled process. This is
thought to solve some class of concurrency problems (ZEO
handles concurrency well, whereas the code to deal with
concurrency in File Cache Manager is not battle-tested).
- There are products like ExtFile and LocalFS. Why bother?
These products are venerable but they are not "drop in"
replacements for OFS.File, nor, to my knowledge, are they aware
of transactions. If you delete a MigratableFile during a
transaction, only when the transaction commits will the disk
data actually be marked as invalid. Likewise, when you add a
MigratableFile during a transaction, only when the transaction
commits will the file body be stored permanently.
- Use the bug tracker at http://www.plope.com/software/collector
to report bugs.
- Chris McDonough (email@example.com)