A Guide to Simplified Automated Drupal Testing

This post is a followup to a presentation I gave at the Pacific North West Drupal Summit in Vancouver held back in October.

I posted the slides for that presentation on Slideshare, Continuous integration with Git & CI Joe.

Background

We've discussed using the Simpletest module in a previous post, Drupal Simpletest Module Abridged. Simpletest is a powerful way to know that the code powering your Drupal site is operating correctly and that new functionality is not breaking what has already been implemented.

Running tests in the browser is time consuming. In this post we'll look at ways to automate the testing process. By the time you're finished reading, you'll know how to receive notifications with test results for your custom code each time you make a commit to your remote version control repository. And you'll also know how to configure daily test runs that cover all available tests including core and contrib.

Along with explaining how to test projects where the entire codebase is stored in version control, I will also describe testing a distribution. The examples given can be used for both Drupal 6.x and Drupal 7.x.

Assumptions

• Your code is stored in a Git repository
• Your Git repository supports POST callbacks (more on that to come)
• You are using the latest version of drush, and for distributions drush_make
• You will install a testing environment that is a functional copy of your Drupal project. For the following examples this site is http://test.example.com

Introducing automated testing

To get a good understanding of the workflow we will be creating, here is an overview of a common development workflow when running tests in the browser.

1. Code new functionality
2. Write a test for that functionality
3. Run the test in the browser
4. Wait for test results
5. Possibly make fixes and return to item 3
6. Eventually run all the available tests
7. Wait a long time

In an ideal world you would run all available tests after you are satisfied with your new code and test. But then you'd end up staring at the browser for an hour or so.

Here is the development workflow we will create using automated testing.

1. Code new functionality
2. Write a test for that functionality
3. Run the test using drush
4. Wait for test results
5. Possibly make fixes and return to item 3
6. Commit code into version control and move to next task
7. Receive a notification with the results of all your custom tests

Lets take a look at how we'll use drush to run tests from the command line.

Installing a drush test runner

UPDATE: Nov. 29, 2010: Drush HEAD now contains an updated version of the test.drush.inc file mentioned below. The drush command is now "test run" not "test". If you are using Drush HEAD >= Nov 29, 2010 or a Drush version > 3.3 please review the updated usage.

A test runner is a command used by a human or a continuous integration server to run tests. We're going to install a drush command that can be used both for test automation and for running tests on the command line. Drush commands live in your ~/.drush directory. If you do not already have one, create it and download the following drush command:

$ git clone git://github.com/sprice/drush_test.git ~/.drush/drush_test

This version of test.drush.inc is based on the Drupal 6.x command created by Young Hahn. It has been updated to work with both Drupal 6.x and Drupal 7.x.

Running tests with Drush

To get a list of all the tests available cd to your webroot and type:

$ drush test --uri=http://test.example.com

Here is a look at a number of tests available:

Using drush you can run a single test, a class of tests, or all tests.

$ drush test --uri=http://test.example.com BlockAdminThemeTestCase will test a single test.

$ drush test --uri=http://test.example.com Block will test all six of the block tests.

$ drush test --uri=http://test.example.com all will test all tests. This includes all the core and contrib tests and all the custom tests you have written for your project code.

When creating custom tests, it is a good idea to use the same test class for each one. That way drush can run all your custom tests with a single command.

Note that there currently seems to be an issue testing the included D7.x Standard profile with this drush test runner. There is no problem testing the D7.x Minimal profile or the custom profile I include later in this post.

Choosing a continuous integration server

A continuous integration server is the primary tool that makes test automation possible. There are many options available though the choice I have seen most often used with Drupal projects is Hudson.

For this post we will be using the small and easy to configure CI Joe. I have enjoyed using CI Joe on projects because it is simple to install and configure and because it does the things that I want and nothing more.

Installing CI Joe

OSX 10.6

It can be helpful to have a basic test setup on your local development environment. However this won't be an automated system.

$ gem install cijoe

$ echo "export PATH=/Users/username/.gem/ruby/1.8/bin:\$PATH" >> ~/.profile

$ source .profile

Linux

The following has been tested on Ubuntu 10.10. Similar instructions will probably work for other distributions though package versions may vary.

$ sudo apt-get install ruby

$ sudo apt-get install rubygems

$ sudo gem install cijoe

Add /var/lib/gems/1.8/bin to your PATH.

$ sudo nano ~/.bashrc

PATH=$PATH:/var/lib/gems/1.8/bin

$ source ~/.bashrc

Installing a test environment

It's important to note that you want to have a Drupal site running that is for testing only. Never test your production site (Simpletest should not be installed on production). And it's a good idea not to test your development or staging sites either. Automated testing will slow the sites down and the purpose of introducing the practices is to increase efficiency.

As outlined in the assumptions of this post I expect you are familiar with installing and configuring a Drupal site. To begin, create a new duplicate of your project site with it's own codebase and own database.

The following examples will assume you will install your site to /var/www/test.example.com and that the site is available at http://test.example.com.

Configuring CI Joe

We need to add some configuration options to CI Joe. All configuration is done via Git config options. We're going to define a test runner command, select a branch of our version control repository to test, and protect the CI server with HTTP Basic Auth. We're also going to configure CI Joe to queue multiple test runs. For more information about CI Joe configuration see the documentation.

For this example I'm going to clone my project code and configure CI Joe to run tests within the CustomTestCase.

$ cd /var/www

$ git clone git://github.com/user/example.git test.example.com

$ cd test.example.com

$ git config --add cijoe.runner "drush test --uri=http://test.example.com/ CustomTestCase"

$ git config --add cijoe.branch develop

$ git config --add cijoe.user username

$ git config --add cijoe.pass secret

$ git config --add cijoe.buildallfile tmp/cijoe.txt

Start CI Joe on port 4567

$ cijoe -p 4567 .

Visit CI Joe at http://test.example.com:4567 using your username and password for access.

Click "Build". You'll need to refresh your browser to see when CI Joe has finished building.

When you click "Build", CI Joe checks out the latest code from the develop branch and executes the test runner command. CI Joe is agnostic when it comes to what runner to use. As long is it returns a non-zero exit status when it fails and a zero exit status when it passes, it just works. This also means that the run-tests.sh PHP script included in Drupal 7 core won't work as it doesn't exit in a way that CI Joe and some other continuous integration servers expect.

Getting notified

Two git hooks handle notifications. Depending on the test outcome, .git/hooks/build-failed or .git/hooks/build-worked will be executed after a build run. These are shell scripts and can do whatever you want them to. For our example we will simply have them send an email. Make sure they are executable.

$ nano /var/www/test.example.com/.git/hooks/build-failed

$ chmod +x /var/www/test.example.com/.git/hooks/build-failed

$ nano /var/www/test.example.com/.git/hooks/build-worked

$ chmod +x /var/www/test.example.com/.git/hooks/build-worked

The examples included in the CI Joe project include scripts that use the mailutils package.

Now when a build runs you'll receive an email letting you know whether your tests passed or failed. Go ahead and click "Build" again to see.

Automation

We now have the testing system configured so that clicking the "Build" button is all that is needed in order to be notified of test results. The last step is to automate the build process so that every time commits are pushed to the remote repository, a build will be triggered.

Most hosted Git repositories support POST URL's. GitHub calls them Post-Receive URL's, Unfuddle calls them Repository Callbacks. When configured, any time the git repository receives a commit, it will POST information about the commit to the URL provided.

Luckily for us, CI Joe doesn't care how the POST is formatted. If CI Joe receives any POST at all a build will be triggered. That means that this will work:

$ curl -d "This is a POST" http://username:secret@test.example.com:4567

Enter the URL of your test server as described in the above command, including the username, password, and port into your Git repositories POST URL option. In GitHub that is found in Admin -> Service Hooks -> Post-Receive URL's.

That's it! You've now configured your project to include automated continuous integration. Push some commits to your remote repository, wait for an email and then pat yourself on the back.

Testing a Drupal distribution

I'll include a note about testing distributions since that's how many are building their Drupal projects these days. A distribution contains a manifest of the code used in the project, an installation profile and likely custom code as well. When you make a change to your project you're sometimes only updating a module version in the .make file. So if you click "Build" in CI Joe it will check out the new .make file but the related code in your project will still be the same.

To solve this problem we will rebuild the codebase on each test run. This may sound crazy to some, but it's important to remember that it's best to commit locally as often as you can and only push to your remote repository a few times a day. If you really want to speed things up you can always use Squid as a caching server for drush module downloads.

Here is a demonstration Drupal 7 distribution so you can get a good understanding of how it works.

Copy the simple_distro.make file to your system and use drush_make to install your project codebase.

$ drush make --working-copy simple_distro.make /var/www/test.example.com

Now one more Git hook needs to be configured that will trigger the project rebuild before running the tests, .git/hooks/after-reset. Make sure the script is executable.

$ nano .git/hooks/after-reset

$ chmod +x .git/hooks/after-reset

This script will cd to your profile directory and run the rebuild.sh script to trigger a rebuild. I've added the -y flag as I'm passing that to my rebuild.sh script.

Configure your git hooks noting that your project repository is in /var/www/test.example.com/profiles/profilename

Complete daily test builds

Earlier I discussed that running all available tests is time consuming. Given that Drupal core and many contrib modules have good test coverage we're not going to worry about running those tests on every remote commit. We'll use cron to run all available tests once a day. Install a second testing platform for this purpose.

Configure the second test environment at: /var/www/daily.test.example.com Make changes to your notification scripts so that it's clear that these are daily test runs. Finally change the test runner git config option to be:

$ git config --add cijoe.runner "drush test --uri=http://daily.test.example.com/ all"

You can run many CI Joe apps on a single machine as long as they are on separate ports. Start CI Joe as described above using port 4568.

Now use cron to enable the tests to run at 1:00am.

Setup the cron job:

$ sudo crontab -e

* 1 * * * curl -d "POST" http://username:pass@daily.test.example.com:4568

Each morning you'll know whether your entire test suite is passing of not. Get ready to start sleeping easier each night.

Conclusion

This post has described how to configure your development environment to include automated continuous integration. By writing tests for the important functionality of your project you'll ensure the integrity of your code over time. By automating the testing process you reduce to zero the individual effort to run tests, and ensure your team is notified if there is a problem. Happy testing.