MS RFC 12: C code Unit tests

Date:2006/02/28 Author:Umberto Nicoletti Contact:umberto.nicoletti at gmail.com Status:Draft Id:$Id$

Overview

Unit tests are a simple but effective way of checking progress and code correctness while programming. MapServer already has unit tests for python MapScript and a functional testing suite but lacks unit tests for the C code which represents the core of its functionality.

This rfc outlines how unit tests can be (gradually) added to the usual development cycle of MapServer, without pretending that developers switch to extreme programming or adopt test driven development.

Issue #1664 on bugzilla and mapserver-dev will be used to track code samples and discussion.

Example

A patch has been attached to issue #1664 on bugzilla which implements the basic infrastructure and some unit tests to demonstrate functionality. The patch was created in under 6 hours by a single person, who is not a frequent C programmer. This is meant to prove that unit tests do not take away much time from developers and that in general the benefit is worth the cost.

The example already supports all the usual preprocessor flags (USE_FLAGS like #ifdef PROJ) so only the relevant tests will be compiled and run. This is necessary to avoid test failures due to a missing feature (as happens with Python unit tests).

Unit testing software

This rfc recommends the use of the CUnit unit testing library for the C language, version 2.0.x. The configure script must be updated to reflect this new dependency and the following or a similar text will be displayed when cunit is not found: “cunit not found, C unit testing disabled.”.

When cunit is available unit tests could be run by default at the end of the compile phase: this will give users some more confidence in the software they just compiled (most of them are linux newbies and have no experience with programming) and could help in catching obvious bugs (like a typo while committing to cvs).

Unit tests are placed in the cunit subdirectory of the MapServer source and distributed with all future releases. Unit tests files should be named with respect to the original MapServer source file they test plus the addition of the _tests suffix: for instance unit tests for the mappool.c file should be placed in a file called mappool_tests.c. The files can have an alphanumeric code added to the suffix to keep them conveniently short (like mappool_tests_1.c, mappool_tests_1b.c or mappool_tests_umberto.c).

Usage recommendations

Developers are not required to write unit tests but when they do we suggest to follow this checklist.

Developer implementing a new feature

• she will try to reproduce the bug with a unit test; if this is not possible or seems difficult she should consider asking on mapserver-dev. The unit test must fail at this stage (if written)
• she fixes the bug and verifies that the test written at the previous item now passes (if written)
• she verifies that also all other tests pass and only after she commits

Developer adding functionality

• she writes a test that verifies the new functionality (this can also be done afterward, as we have relaxed requirements)
• she writes the code implementing the desired functionality
• she now verifies that the test passes
• she verifies that all other tests pass and only after she commits

General

She will also use the usual preprocessor flags to ensure only the relevant tests for the current configuration context will be compiled and run. The example attached to issue #1664 already implements this feature.

Testing specific functionalities

Some aspects of MapServer like database connections and gd rendering are inherently difficult to test. This section provides guidance on how to deal with them in unit tests and will be expanded as new or better solutions are devised.

Database connections

Database connections generally require a specific setup so that expected tables, data and possibly other structures are in place. While this should be a long term goal in the short term unit tests could be limited to comparing the sql query against the expected one.

This quite likely requires a refactoring of current PostGIS code.

In the future a minimal setup script should be provided to create and populate the database for the user (if she desires to do so).

GD (and others) rendering engines

Taking inspiration from the perl GD module a set of images must be produced and then compared against those produced by MapServer. This kind of operation is already performed by the msautotest suite, so effort should rather go toward improving that instead of implementing yet another gd test suite.

WFS/WMS support

Note: The author is not an expert here

In this case unit tests can be used to verify that for certain requests (possibly using the test data supplied with MapServer) the string returned by MapServer equals the expected one.

A refactoring of existing code could be necessary.

Mapscript

Unit tests should be developed by the MapScript mantainers by following the guidelines given here.

Running unit tests and functional tests ( Continuos integration )

Not part of unit tests for now, but useful in perspective

Effort should be put toward developing a build system capable of

1. testing the overall build/test of MapServer and of the various MapServers with different configure options
2. and integrating those results with the msautotest suite.