Personal tools
Skip to content. | Skip to navigation
The zc.async package provides an easy-to-use Python tool that schedules work persistently and reliably across multiple processes and machines. For instance... * Web apps: maybe your web application lets users request the creation of a large PDF, or some other expensive task. * Postponed work: maybe you have a job that needs to be done at a certain time, not right now. * Parallel processing: maybe you have a long-running problem that can be made to complete faster by splitting it up into discrete parts, each performed in parallel, across multiple machines. * Serial processing: maybe you want to decompose and serialize a job. High-level features include the following: * easy to use; * flexible configuration, changeable dynamically in production; * reliable; * supports high availability; * good debugging tools; * well-tested; and * friendly to testing. While developed as part of the Zope project, zc.async can be used stand-alone. How does it work? The system uses the Zope Object Database (ZODB), a transactional, pickle-based Python object database, for communication and coordination among participating processes. zc.async participants can each run in their own process, or share a process (run in threads) with other code. The Twisted framework supplies some code (failures and reactor implementations, primarily) and some concepts to the package.
The sequence in this package has a list-like API, but stores its values in individual buckets. This means that, for small changes in large sequences, the sequence could be a big win. For instance, an ordered BTree-based container might want to store order in a sequence, so that moves only cause a bucket or two--around 50 strings or less--to be rewritten in the database, rather than the entire contents (which might be thousands of strings, for instance). If the sequence is most often completely rearranged, the complexity of the code in this package is not desirable. It only makes sense if changes most frequently are fairly small. One downside is that reading and writing is more work than with a normal list. If this were to actually gain traction, perhaps writing some or all of it in C would be helpful. However, it still seems pretty snappy. Another downside is the corollary of the bucket advantage listed initially: with more persistent objects, iterating over it will fill a lot of ZODB's object cache (which is based on the number of objects cached, rather than the size). Consider specifying a big object cache if you are using these to store a lot of data and are frequently iterating or changing. These sequences return slices as iterators, and add some helpful iteration methods. It adds a copy method that provides a cheap copy of the blist that shares all buckets and indexes until a write happens, at which point it copies and mutates the affected indexes and buckets. We'll take a glance at how these differences work, and then describe the implementation's basic mechanism, and close with a brief discussion of performance characteristics in the abstract.
Buildout is a project designed to solve 2 problems: 1. Application-centric assembly and deployment Assembly runs the gamut from stitching together libraries to create a running program, to production deployment configuration of applications, and associated systems and tools (e.g. run-control scripts, cron jobs, logs, service registration, etc.). Buildout might be confused with build tools like make or ant, but it is a little higher level and might invoke systems like make or ant to get it's work done. Buildout might be confused with systems like puppet or chef, but it is more application focused. Systems like puppet or chef might use buildout to get their work done. Buildout is also somewhat Python-centric, even though it can be used to assemble and deploy non-python applications. It has some special features for assembling Python programs. It's scripted with Python, unlike, say puppet or chef, which are scripted with Ruby. 2. Repeatable assembly of programs from Python software distributions Buildout puts great effort toward making program assembly a highly repeatable process, whether in a very open-ended development mode, where dependency versions aren't locked down, or in a deployment environment where dependency versions are fully specified. You should be able to check buildout into a VCS and later check it out. Two checkouts built at the same time in the same environment should always give the same result, regardless of their history. Among other things, after a buildout, all dependencies should be at the most recent version consistent with any version specifications expressed in the buildout. Buildout supports applications consisting of multiple programs, with different programs in an application free to use different versions of Python distributions. This is in contrast with a Python installation (real or virtual), where, for any given distribution, there can only be one installed. To learn more about buildout, including how to use it, see http://buildout.org/.
BTree-based persistent dict-like objects (regular dict and ordered) that can be used as base classes. This is a bit of a heavyweight solution, as every zc.dict.Dict (and zc.dict.OrderedDict) is at least 3 persistent objects. Keep this in mind if you intend to create lots and lots of these. To build, run ``python bootstrap/bootstrap.py`` and then ``bin/buildout`` from this directory. A clean, non-system Python is strongly recommended. An efficient, persistent and subclassable dict PersistentDict is very inefficient if it contains more than a couple of values, and BTrees are not recommended to inherit from. This class is a simple wrapper over a BTree. It retains the efficiency of BTrees and is safe to use as a base class. Also, it implements the full Python dict interface.