Login | Register
My pages Projects Community openCollabNet

SVNChecker Manual

Heinrich Wendel



SVNChecker

SVNChecker is a framework for Subversion pre-commit and post-commit hook scripts.

The SVNChecker allows tightly integration Subversion into existing work and development environments and to perform a variety of checks of the committed source code. The major use cases are the integration of Subversion with issue and bug tracking systems and the check of source code against coding standards. But, the SVNChecker can be used for any other use case where a tight integration with the source code management is necessary.

SVNChecker is designed and developed as an open extensible framework. By writing appropriate plug-ins, it can be extended to almost any kind of "check" (for example, to check source code or to query external databases). This can be used to enforce a variety of regulations of the software development process. By developing "handler" plug-ins, the result of checks can be handled in almost all needed ways (for example, for sending emails, updating databases, or running any given external software).


Chapter 1. Architecture

The basic architecture of SVNChecker is displayed in the following graphic. It provides a flexible framework which is easily extendable by Checks and Handlers:

The entry point for SVNChecker is the capability of Subversion to support hook scripts which are called when specific events are triggered by the repository. There are nine different kinds of hook scripts, three deal with the commit process, two with the change of revision properties and four with the handling of locks. For a complete reference of the hook scripts take a look at the "Repository Hooks" chapter in the Subversion manual. For now it is enough to know about the pre-commit and post-commit hook.

The pre-commit script is called whenever a user wants to commit his changes to the repository. A complete snapshot of how the repository would look after this commit, a so called transaction, is made. The transaction also includes information about the author of the commit, the commit message and a list of changed files. You can perform checks on this transaction and reject it if doesn't conform to your rules.

The post-commit script is called whenever a transaction was completed and a new revision exists. A revision contains the same information as a transaction with the minor difference that the revision was already committed and can't be undone anymore. But you still might want to send a log message containing a list of all changed files to a mailing list for example.

In the following the term transaction will be used for both, a real transaction and a revision, since the SVNChecker provides an abstraction layer which handles both cases uniformly.

Based on this transaction a list of checks, e.g. code convention examination, verification of access rights or validation of the commit message, can be performed. Each check returns an exit code, which is zero for success and bigger than zero for failure, and a message including further details.

After each check was performed, the message, the exit code and the transaction are passed to the configured handlers. A handler is an extension which processes the output of the checks, e.g. it can send mails, create a log file or print the output to the console.

The configuration of the SVNChecker is very flexible. You can define the checks for the pre-commit and post-commit cases and a list of handlers for each check. Furthermore each check and handler has access to the configuration for its own purpose. If you use SVNChecker for more than one repository you can define a global system configuration with fixed values which cannot be overwritten by the local configuration in each repository. For bigger repositories you can configure different profiles affecting different parts of the repository. Each profile can have its own configuration and share some default values with the other profiles. One use-case may be stricter checks for the trunk and laxer rules for branches.


Chapter 2. Installation

2.1. Requirements

SVNChecker requires at least python 2.4 and Subversion 1.2. It runs on both, Windows and *nix. Some checks may depend on external tools. For the requirements of the individual checks take a look at the "Available Modules" section.


2.2. Download

SVNChecker is available from the project side at tigris .


2.3. Hooks

After extracting the SVNChecker package to an arbitrary directory, you have to create two files in your Subversion hooks directory. On *nix systems those files are named pre-commit and post-commit and must be set executable. They contain the following content, where %pathtosvnchecker% is replaced by the directory to which you unpacked the SVNChecker and %hook% is either PreCommit or PostCommit (case-sensitive):


#!/bin/sh
/usr/bin/env python %pathtosvnchecker%/Main.py %hook% $1 $2 || exit 1
				
On Windows systems those files are named pre-commit.bat and post-commit.bat with the following contents, %pathtopython% replaced by the path to the python executable:

%pathtopython% %pathtosvnchecker%/Main.py %hook% $1 $2 || exit 1
				


Chapter 3. Configuration

3.1. Configuration Files

The SVNChecker can use any combination of the following two configuration files. The first one is a global configuration file which affects all repositories installed on the same system and must be located in the SVNChecker directory, just aside of "Main.py", named "svncheckerconfig.ini". The second one is a repository local configuration file which must be located in the "hooks" directory of a repository and names "svncheckerconfig.ini" as well. If only one of those configuration files exist, this only one will be used. If both exist, the options from the global file will override the options from the local one. The file format is a simple ini-like format with categories, which are used for profiles, and key=value pairs.


3.2. Basic configuration

The basic configuration of the SVNChecker is very simple. It starts with a category, named "Default". After that you need to specify which checks to activate for the pre-commit and post-commit hooks. For each check you then activate the desired handlers for the success and failure cases. An example is shown below:


[Default]
Main.PreCommitChecks=Pylint, XMLValidator
Main.PostCommitChecks=Log

Pylint.SuccessHandlers=Console, Mail
Pylint.FailureHandlers=Console, Mail

XMLValidator.SuccessHandlers=Console, Mail
XMLValidator.FailureHandlers=Console, Mail

Log.SuccessHandlers=Console, Mail
Log.FailureHandlers=Console, Mail
				


3.3. Resolving of Variables

You might have seen in the previous example that the flexibility is very high, you can configure a success and failure handlers for each check. But this can make the configuration file very big and unclear, typing the same code every time although you want to use the same handlers for all checks. Furthermore one configuration option may be required by more then one check, e.g. the path to the java binary. To support those cases the SVNChecker does not only look for the real name of the specified configuration option. If it can't find the complete name it will substitute the first part of the check, the one before the "." with "Main". So the previous example could have also been written as:


[Default]
Main.PreCommitChecks=Pylint, XMLValidator
Main.PostCommitChecks=Log

Main.SuccessHandlers=Console, Mail
Main.FailureHandlers=Console, Mail
				


3.4. Profiles

If your repository grows or you manage different trunks and branches, you might want to configure specific settings for different parts of the repository. In order to do that the SVNChecker support profiles. Profiles allow you to configure different settings of the SVNChecker for different files or paths in the repository. Profiles are defined by categories in the configuration file and must have a "Main.Regex" configuration option which specifies a regular expression which matches all files this profile is valid for. All files that didn't match a profile will be checked against the "Default" profile. Here an example profile that disables all PreCommitChecks for branches:


[Default]
Main.PreCommitChecks=Pylint, XMLValidator
Main.PostCommitChecks=Log

Main.SuccessHandlers=Console, Mail
Main.FailureHandlers=Console, Mail

[Branches]
Main.Regex=^branches/.*$
Main.PreCommitChecks=
				
If a specified variable can't be found in the given profile the one from the default profile will be checked. In this example Main.PostCommitChecks will be taken from the default profile.


3.5. Variable Substitution

Sometimes it might be useful to know the path of the hooks directory of the repository in your configuration file, e.g. when writing log files. For this purpose you can use the string "%HOOKS%", which will be replaced by the real directory during runtime.


Chapter 4. Available Modules

4.1. Checks

4.1.1. AccessRights

With the help of this check you can restrict or grant access for all or selected files only to specified users. The configuration options are:

AccessRights.Rules

List of rules that should be checked. Those are just identifiers for the following options. (Default="")

AccessRights.%RULE%.CheckFiles

List of regular expressions that match files that should be checked by this rule. (Default=".*")

AccessRights.%RULE%.IgnoreFiles

List of regular expressions that match files that should not be checked by this rule. (Default="")

AccessRights.%RULE%.AllowUsers

List of users that may commit changes to the files. (Default="")

AccessRights.%RULE%.DenyUsers

List of users that may not commit changes to the files. (Default="")


4.1.2. CaseInsensitiveFilenameClash

This check rejects commits which contain files which already exist in the repository with the same name but one or more characters in a different case.


4.1.3. Checkout

This check checks-out files from the repository to the local filesystem. SECURITY: Please be careful to not accidently overwrite any file. The configuration options are:

Checkout.Entries

List of entries that should be checked-out. Those are just identifiers for the following options. (Default="")

Checkout.%ENTRY%.Source

Name of the file in the repository.

Checkout.%ENTRY%.Destination

Destination, including name, where to put the file on the local filesystem.


4.1.4. Checkstyle

This check runs checkstyle over a given set of files. The configuration options are:

Checkstyle.Java

Path to the java binary.

Checkstyle.Classpath

Path to the checkstyle-all.jar

Checkstyle.ConfigFile

Path to the checkstyle configuration file.

Checkstyle.CheckFiles

List of regular expressions that match files that should be checked by this rule. (Default=".*\.java")

Checkstyle.IgnoreFiles

List of regular expressions that match files that should not be checked by this rule. (Default="")


4.1.5. Keywords

This check tests if a specified list of keywords is set for a specified set of files. The configuration options are:

Keywords.Keywords

Keywords which should be checked.

Keywords.Keywords.CheckFiles

List of regular expressions that match files that should be checked by this rule. (Default=".*")

Keywords.Keywords.IgnoreFiles

List of regular expressions that match files that should not be checked by this rule. (Default="")


4.1.6. Log

Generates a log message. The configuration options are:

Log.ViewVcUrl

Url to viewvc to include in the log message. (Default="")


4.1.7. Mantis

Checks if the given mantis id(s) exists, is set to the status "in_progress" and is assigned to the currently committing user. The Mantis ID must be specified in the log message and match the pattern 'MANTIS([:#]|[\s\-_]ID) ([0-9]+)'. The configuration options are:

Mantis.CheckInProgress

Whether to check the "in_progress" status. (Default=True)

Mantis.CheckHandler

Whether to check if the issue is assigned to the currently committing user. (Default=True)

Mantis.URL

URL to the mantisconnect script.

Mantis.User

Username to login in mantis.

Mantis.Password

Password to login in mantis.


4.1.8. Pylint

This check runs pylint over a given set of files. The configuration options are:

Pylint.ConfigFile

Path to the pylint configuration file. (Default="")

Pylint.CheckFiles

List of regular expressions that match files that should be checked by this rule. (Default=".*\.py")

Pylint.IgnoreFiles

List of regular expressions that match files that should not be checked by this rule. (Default="")


4.1.9. UnitTests

This check tests if a unit test exists for a given java class. The java class must follow the pattern /main/.../Class.java and the unit test must follow the pattern /main/.../TestClass.java. Interfaces are omitted. The configuration options are:

UnitTests.CheckFiles

List of regular expressions that match files that should be checked by this rule. (Default=".*\.java")

UnitTests.IgnoreFiles

List of regular expressions that match files that should not be checked by this rule. (Default="")


4.1.10. XMLValidator

This check checks if a file is a valid xml file.

XMLValidator.CheckFiles

List of regular expressions that match files that should be checked by this rule. (Default=".*\.xml")

XMLValidator.IgnoreFiles

List of regular expressions that match files that should not be checked by this rule. (Default="")


4.2. Handlers

4.2.1. Console

Prints the output either to sys.stderr or sys.stdout.


4.2.2. File

Print the output to a given file. Configuration options are:

%Check%.SuccessFile

Path to the file to print the output in the case of success.

%Check%.FailureFile

Path to the file to print the output in the case of failure.


4.2.3. Mail

Mails the output to the given addresses. The configuration options are:

%Check%.SuccessAddresses

Mail addresses to send the output in the case of success.

%Check%.FailureAddresses

Mail addresses to send the output in the case of failure.


4.2.4. Mantis

Appends the message as bug note to the given mantis id(s). The Mantis ID must be specified in the logmessage and match the pattern 'MANTIS([:#]|[\s\-_]ID) ([0-9]+)'. Available configuration options are:

Mantis.URL

URL to the mantisconnect script.

Mantis.User

Username to login in mantis.

Mantis.Password

Password to login in mantis.


Chapter 5. Developers Guide

Developing Checks and Handlers to extend the SVNChecker is a very easy process. Each of them must only implement one method which does the actual work. As helper classes the current Transaction and Configuration is provided to them.


5.1. The Transaction class

The Transaction class gives you all information you need to know about the commit and the repository. It has the following methods:

getUserID()

Returns a string with the username of the current transaction.

getFiles(checkList, ignoreList)

Returns a map of all modified files. The keys of the map are the filenames. The value of each key of the map is the associated attribute, which can be one of the default svnlook changed attributes.

CheckList must be a list of regular expressions for files which should be included, it defaults to [".*"].

IgnoreList must be a list of regular expressions for files which should be ignored, it defaults to [].

getFile(filename)

Returns the path to a temporary copy of a file in the repository.

fileExists(filename, ignoreCase=False)

Returns whether a file exists in the current transaction or revision of the repository, optionally case-insensitive.

getCommitMsg()

Returns the commit message.

getRevision()

Return the id of the revision or transaction.

getProperty(keyword, filename)

Returns a specified property of a file.

hasProperty(keyword, filename)

Checks if a given file has the given property.

listProperties(filename)

Returns a list of names of the properties for a file.


5.2. The Config class

The Configuration class provides you with all required methods to access configuration variables:

getString(var, default=None)

Returns a variable as string. If the variable does not exist and default is set default will be returned, otherwise a NoSuchConfigurationValueError will be raised.

getArray(var, default=None)

Returns a variable as array. The configuration string is split by the ',' character. If the variable does not exist and default is set default will be returned, otherwise a NoSuchConfigurationValueError will be raised.

getBoolean(var, default=None)

Returns a variable as boolean. True, true, 1 will return True. False, false, 0 will return False. If the variable does not match the pattern a ValueError will be raised. If the variable does not exist and default is set default will be returned, otherwise a NoSuchConfigurationValueError will be raised.

getInteger(var, default=None)

Returns a variable as integer. If the variable cannot be converted to an integer a ValueError will be raised. If the variable does not exist and default is set default will be returned, otherwise a NoSuchConfigurationValueError will be raised.


5.3. Implementing a Check

To implement a new Check you only need to write a python module containing a method named "run", with two parameters, "transaction" and "config". The return value must be a tuple containing the return message and the exit code. You have to put the check into the "checks" package of the SVNChecker. Here an example of a simple check, testing if a xml file is parseable:


from xml.dom import minidom
from xml.parsers import expat

def run(transaction, config):

    check = config.getArray("XMLValidator.CheckFiles", [".*\.xml"])
    ignore = config.getArray("XMLValidator.IgnoreFiles", [])
    files = transaction.getFiles(check, ignore)

    msg= ""
    for filename, attribute in files.iteritems():
        if attribute in ["A", "U"]:
            try:
                minidom.parse(transaction.getFile(filename))
            except expat.ExpatError, e:
                msg += "XML Validation error in file %r: %s" % (filename, e)

    if msg:
        return (msg, 1)
    else:
        return ("", 0)
				


5.4. Implementing a Handler

To implement a new Handler you only need to write a python module containing a method named "run", with five parameters, "transaction", "config", "check", "msg" and "exitCode". "check" is the name of the just executed check. Here an example of a simple handler, printing the output to the console:


import sys

separator = "\n" + "=" * 80 + "\n"

def run(transaction, config, check, msg, exitCode):

    if (exitCode == 1):
        out = sys.stderr
    else:
        out = sys.stdout

    out.write(separator)
    out.write(msg)
    out.write(separator)
				


Chapter 6. Changelog

6.1. Version 0.2.1

  • Fixed bug in the pylint check that made the check pass although it failed.

  • Made the "Pylint.ConfigFile" configuration option optional, a default pylintrc will be used if you don't specify it.

  • Fixed a bug in the transaction module that returned an empty file when calling getFile() more than once.

  • Fixed a bug in the transaction module to avoid a deadlock when receiving big files.

  • Make the SVNChecker compatible with python 2.4.


6.2. Version 0.2

This is the first big revision of the SVNChecker. The interfaces to implement a Check or Handler were changed. Furthermore the Transaction and Config classes have been improved. If you develop your own modules take a look at the updated "Developers Guide". A lot of checks have been enhanced, e.g. Checkstyle and Pylint are faster now, and new Checks were added, including Checkout and CaseInsensitiveFilenameClash. For a complete reference of the available Checks and Handlers and their configuration options take a look at the "Available Modules" section. The configuration now supports a system-wide file which can define configuration options which are fixed for all installed repositories. If you want to use the hooks directory in the configuration file you can now use %HOOKS%, which will automatically be replaced by the real directory. With the help of a new test-environment all modules have undergone a better testing for improved stability. A complete list of fixed issues is available via the issue tracker at the tigris project page.