Overview

Part of the work I did was to reverse-engineer at least a minimal changelog for each of the packages (given the ZopeX3.0.0 version as "first release"). Please have a look:

• 3.0.0 packages: http://download.zope.org/distribution/3.0.0/
• 3.2.0 packages: http://download.zope.org/distribution/

A couple of notes on the process (summarized from my reply to Martijn Faassen earlier today):

Kickstarting a Package

There is some support for "kickstarting" an eggifying project. E.g.:

    $ export ZSVN=svn+ssh://svn.zope.org/repos/main
    $ svn cp -m "Eggify" $ZSVN/productsupport/trunk/project-template \
                         $ZSVN/zope.foopackage

The top-level zope.foopackage will have a skeleton trunk, plus empty branches and 'tags':

    $ svn co $ZSVN/zope.foopackage/trunk zope.foopackage-trunk
    $ cd zope.foopackage-trunk
    $ ls -1
    CHANGES.txt
    develop.py
    INSTALL.txt
    README.txt
    setup.cfg.in
    setup.py
    src
    test.py
    workspace

The text files are templates, and should be edited to fit.

The develop.py script will set up a local bin and lib directory inside your checkout, and try to install dependencies as specified in setup.py and setup.cfg (which it copies from setup.cfg.in). It also jams the Zope egg URL into setup.cfg.

I usually run setup.py develop afterwards, as it is better at finding all the dependencies:

    $ PYTHONPATH=lib /path/to/your/python setup.py develop

The zope.testing package is supposed to be pulled in during that bootstrapping, but often seems missed (I haven't yet figured out why). Running the local easy_install does the trick:

    $ PYTHONPATH=lib /path/to/your/python bin/easy_install \
      zope.testing

The src/zope directory is intended to contain a svn:externals pointer to the "canonical" location of the source (we may eventually reverse this, and make the main Zope tree point out at the top-level package directories). E.g.:

    $ svn propedit svn:externals src/zope
    #... add your external here
    $ svn up
    # ... fetches your external.

That src/zope directory also contains "setuptools / pkgutil" boilerplat namespace __init__.py:

    $ cat src/zope/__init__.py
    # namespace package boilerplate
    try:
        __import__('pkg_resources').declare_namespace(__name__)
    except ImportError, e:
        from pkgutil import extend_path
        __path__ = extend_path(__path__, __name__)

The test.py needs to be edited to point to your package. At that point, you should have all your package's dependencies installed (assuming that you named them in setup.py), and should be able to run the tests using your local eggs:

    $ PYTHONPATH=lib /path/to/your/python test.py

Notes from the Road

Observations I jotted down while running through this process a dozen or so times:

• If we are going to go down this road, we need to extend the "fine-grained changelog" stuff, either by moving the "canonical" location of the changelog out to the packaging diretories, or else by tagging the "main" changelog more carefully (with package names) to allow mechanical extraction.
• The process of using a C header defined in one package to build an extension in another is tortured: the zpkg based solution was weird, but the egg-based one isn't much better.

  I was trying to handle those cases without actually installing the dependencies anywhere but the local sandbox, which led me to create an include directory inside the packaging directory, and add the package with the header as an svn:externals entry there.

• I made heavy use of "tagged" svn:externals to ensure that I was pulling in the correct versions for 3.0.0 and 3.2.0. Please let me know if I fat-fingered any of them (I wish viewcvs would expose the externals!).

Test Driving It

Give a dependency-driven install of a pile of packages a spin. E.g.:

    $ mkdir aside
    $ PYTHONPATH=aside \
      /path/to/setuptools/enabled/python /path/to/easy_install \
      --install-dir=aside \
      --find-links=http://download.zope.org/distribution/ \
      zope.documenttemplate[untrusted]

That command kicks off an interesting process:

• It finds the zope.documenttemplate package (the 3.2.0 version, given the URL we passed), and tries to install it, using the added untrusted feature.
• It transitively installs each of the dependencies, including particularly zope.security and its dependencies, based on the feature.
• All the packages are installed as "non-zip safe" eggs in the aside directory, along with the setuptools bookkeeping stuff needed to stitch them all together.

To-Do

1. Chris McDonough made a start on the ZODB-related packages. Investigate and merge:
   • BTrees
   • persistent
   • transaction
   • ThreadedAsync
   • ZEO
   • ZODB
   • ZConfig
   • zdaemon
2. Build a meta-package which depends on all the ZODB and related stuff, and supplies the scripts to create storage server instances.
3. Eggify packages which were not part of zope.app in 3.0.0 / 3.2.0 (avoidg zope.app because detangling the dependencies inside that beast is too painful to think about):
   • docutils (we should be able to use a stock one from the Cheese Shop, I think)
   • zope.cachedescriptors
   • zope.configuration
   • zope.dependencytool
   • zope.hookable
   • zope.importtool
   • zope.modulealias
   • zope.publisher
   • zope.server
4. Build a meta-package which depends on all that and contains scripts for setting up a minimal Zope3 appserver instance.
5. Look at packaging zope.app, prefereably using some kind of automated tool (maybe the zpkgegg stuff which Nathan prototyped?)
6. Play with packaging some of the "extras" not present in those releases:
   • bugtracker
   • zwiki
   • zope.formlib
   • zope.contentprovider

Resources

• Phillip Eby's setuptools page
• Phillip's Python Eggs page
• The Cheese Shop