Solving the Environment Problem at Channel 4

Since I started at Channel 4 a year ago, I’ve had a pet project to solve some of the headaches inherent in TeamSite development.  Here I’d like to share the ideas behind what I’ve done so far, using the Big Brother 2009 project as an example.

This article covers how we’ve got TeamSite project code to behave more like a deployable atomic application.  This has been achieved by two distinct initiatives:

1. Getting project code to run in the same TeamSite branch as the project content
2. Developing a mechanism to promote this code between virtual environments

The remainder of this article relates to the first of these.

Summary

We wanted all the TeamSite code for the Big Brother site to be located in the project branch, including templates, workflow code, modules, CGI callouts, and DNR scripts.  This allows us to have more than one set of Big Brother code running independently on the same server, or located arbitrarily on different TeamSite servers.

Developers develop in a development branch (crikey, that’s a mouthful).  They use standard TeamSite functionality to copy or merge their completed work into a stable Integration branch.  A ‘Code Promotion’ workflow deploys this code from Integration to all other environments: Test, Uat, Stage and Production.  Using this method, all of the TeamSite code and configuration is managed and deployed atomically.

There is a simple relationship between TeamSite code, TeamSite content and the target application – there’s only one of each for each environment, and no interdependencies.   This keeps the Build Manager happy.  If you know a Build Manager, you’ll know this is quite an achievement.

The benefits for Content Management developers are:

• Single-click deployment.  There are no fiddly release notes to follow when deploying a project to an environment, very handy for those 4am red-eye release slots.
• All environments are stable.  The exception is Dev, but that’s what it’s for.
• Several instances of a project can exist on the same TeamSite box, running code completely independent of each other.

The Problems

Cost

In practice most organisations have two or three TeamSite servers.  One production server for the editorial team, a non-production server for development and testing, and sometimes also a DR server for production fail-over if you’re lucky.

In a perfect world, every environment would have its own physical TeamSite server, or perhaps running in a Virtual Machine (this is another interest of mine, I’ll write about this one day).   For both of these options however, the license cost is prohibitive.  It’s hard to justify this expenditure on a back-office system like a CMS.  So we have to make do, we try to maintain multiple versions of the same project from the same server.

iw-home

One of the best things about TeamSite is the virtual file system it uses for versioning and organising content.  Code is a different story, the arrangement is a bit shoddy.  When you work with TeamSite as a developer, one of the first things you notice is how reliant you are on putting code in the file system.  iw-home is the installation directory for the product, and by default you’re required to put your custom code right alongside it.  Not ideal.

Making changes on the file system directly are a really bad idea for many reasons.  Primarily there is the problem of source control, the details of which I won’t go into here.  TeamSite is essentially a file versioning system, so the solution is obvious after you’ve worked with TeamSite for a bit.  Indeed every customer installation I have ever seen gets around  problem in the same way: pull the file system code into a TeamSite branch and manage it from there.  At Channel 4 this is called “TeamSite_Admin”.  Fidelity and Camelot had something similar.  Developers edit their code in this branch, changes are versioned, and a submit workflow sends the files out to the file system.

Sharing iw-home

Now consider having two project branches, Dev and Test, both on the same TeamSite box using the same iw-home.  Developers and Testers need their environment for very different things.

• Developers are in a constant cycle of breaking things and fixing them.
• Testers require a stable project in a known state.

These are clearly diametric needs, and this can only be solved by finding a way of splitting the code base into isolated environments.

Isolated Environments

A little Perl module called “AreaLib” is at the centre of the solution we’ve developed at Channel 4.  I have to give credit for this work to two highly skilled independent consultants, Steve Martina and Lee Ablett, who wrote this for us… albeit from a proof of concept that I put together

If you include AreaLib in a piece of code, it attempts to work out which branch you’re in.  It does this using a variety of methods, such that it will work within a template, workflow task, CGI, a ‘call server’ routine, or wherever.  When you include this module at the top of your code, it then looks locally in the branch for any further dependencies, falling back to the default locations in iw-perl if they’re not found.

It does this by amending Perl’s @INC path.  For example, if you use the following code in a template, in the branch /default/main/Channel4/dev/BigBrother/WORKAREA/editorial:

use C4::AreaLib;
use C4::EvictRacistHousemate;

The “EvictRacistHousemate” module will be picked up from one of the following locations:

/default/main/Channel4/dev/BigBrother/WORKAREA/editorial/iw_admin/perl/lib/C4/EvictRacistHousemate.pm
/default/main/Channel4/dev/BigBrother/STAGING/iw_admin/perl/lib/C4/EvictRacistHousemate.pm
/default/main/Channel4/dev/STAGING/iw_admin/perl/lib/C4/EvictRacistHousemate.pm
/default/main/Channel4/STAGING/iw_admin/perl/lib/C4/EvictRacistHousemate.pm
iw-home/iw-perl/lib/C4/EvictRacistHousemate.pm
iw-home/iw-perl/site/lib/C4/EvictRacistHousemate.pm
iw-home/iw-perl/vendor/lib/C4/EvictRacistHousemate.pm

This is easy to use from a Template.  However other TeamSite code needs a bit more thought, but it boils down to passing all requests through a wrapper within iw-home.  The wrapper uses AreaLib, and then forwards the request off to the right module for the job.  For example, an external task in a workflow uses the wrapper script as its command, passing the name of the required module in a workflow variable.  Control passes from the wrapper script to the module, which takes over the external task processing and calls back to the workflow.

The end result: all code can run, via a small wrapper and AreaLib, from modules located within the branch.

Soon, I’ll describe the Code Promotion workflow, the way we move all this stuff about in one operation.  I promise some fancy diagrams.