Integrating PEP8 & PyLint Tests with Nose & Xunit

Motivation

Most continuous integration platforms support the xunit/junit formats for test results. They allow a greater granularity of test management above a simple pass/fail. Nose supports xunit reports by default, but PEP8 and PyLint do not. By integrating PEP8 & PyLint into nose tests, we can include their results in our xunit reports. This allows easy tracking of lint and pep failures and linking them to issues.

Technique & Limitations

In both cases I used the technique of collecting results, then using a test generator pattern with the PEP or lint output as the error message. This has the benefit of easily integrating with nose's xunit output plugin. However, it does come with the disadvantage that tests do not persist across runs. This is due to the test only existing when it fails and disappearing when it does not. The consequence is that if your CI tracks 'fixed' bugs, it'll lose that metric since the test never technically succeeds ... it just disappears.

Providing persistance would require a bit of footwork to track past failures and generate a test, with the same name, with a 'pass' flag in the xunit.

PEP8

I couldn't find a method of extracting test data directly from the pep8 reports. To work around that, I subclassed the StandardReport and overrode the get_file_results method so data is collected in an array of results.

class CustomReport(pep8.StandardReport):
    """
    Collect report into an array of results.
    """
    results = []

    def get_file_results(self):
        if self._deferred_print:
            self._deferred_print.sort()
            for line_number, offset, code, text, _ in self._deferred_print:
                self.results.append({
                    'path': self.filename,
                    'row': self.line_offset + line_number,
                    'col': offset + 1,
                    'code': code,
                    'text': text,
                })
        return self.file_errors

Next, a generator requires a test method to yield a function and its args which in turn produces a test result. I created a small wrapper method to always fail.

def fail(msg):
    """
    Fails with message.
    """
    assert_true(False, msg)

Finally, I created a test method that programmatically calls pep8 on the project and generates a failure for each error found.

def test_pep8_conformance():
    """
    Test for pep8 conformance
    """
    # Here I'm matching the project path so I can remove the prefix path
    # in the output. This strips the often obtuse absolute paths in CI
    # environments. Yours will obviously differ.
    pattern = re.compile(r'.*(roadrage/roadrage.*\.py)')
    pep8style = pep8.StyleGuide(reporter=CustomReport)
    base = os.path.dirname(os.path.abspath(__file__))
    dirname = os.path.abspath(os.path.join(base, '..'))
    sources = [
        os.path.join(root, pyfile) for root, _, files in os.walk(dirname)
        for pyfile in files
        if pyfile.endswith('.py')]
    report = pep8style.init_report()
    pep8style.check_files(sources)

    for error in report.results:
        msg = "{path}: {code} {row}, {col} - {text}"
        match = pattern.match(error['path'])
        if match:
            rel_path = match.group(1)
        else:
            rel_path = error['path']

        yield fail, msg.format(
            path=rel_path,
            code=error['code'],
            row=error['row'],
            col=error['col'],
            text=error['text']
        )

These are placed where nose can find them, in my case tests/test_pep8.py. Here's the full source.

"""
Provides code conformance testing.

We do it here so we can collect errors into the xunit report for CI
integration.
"""
import os
import pep8
import re

from nose.tools import assert_true  # pylint: disable=E0611

PROJ_ROOT = "roadrage/roadrage"

def fail(msg):
    """
    Fails with message.
    """
    assert_true(False, msg)


class CustomReport(pep8.StandardReport):
    """
    Collect report into an array of results.
    """
    results = []

    def get_file_results(self):
        if self._deferred_print:
            self._deferred_print.sort()
            for line_number, offset, code, text, _ in self._deferred_print:
                self.results.append({
                    'path': self.filename,
                    'row': self.line_offset + line_number,
                    'col': offset + 1,
                    'code': code,
                    'text': text,
                })
        return self.file_errors


def test_pep8_conformance():
    """
    Test for pep8 conformance
    """
    pattern = re.compile(r'.*({0}.*\.py)'.format(PROJ_ROOT)
    pep8style = pep8.StyleGuide(reporter=CustomReport)
    base = os.path.dirname(os.path.abspath(__file__))
    dirname = os.path.abspath(os.path.join(base, '..'))
    sources = [
        os.path.join(root, pyfile) for root, _, files in os.walk(dirname)
        for pyfile in files
        if pyfile.endswith('.py')]
    report = pep8style.init_report()
    pep8style.check_files(sources)

    for error in report.results:
        msg = "{path}: {code} {row}, {col} - {text}"
        match = pattern.match(error['path'])
        if match:
            rel_path = match.group(1)
        else:
            rel_path = error['path']

        yield fail, msg.format(
            path=rel_path,
            code=error['code'],
            row=error['row'],
            col=error['col'],
            text=error['text']
        )

PyLint

I used a very similar method with PyLint, with one exception. I couldn't figure out how to do it purely programmatically. The failure was primarily due to a bug in the current version where the py_run method ignored the pylintrc options passed to it. I seriously can't find the bug report now, but it was being tracked. My workaround involved configuring a custom output template, shelling out, and capturing then parsing the output.

The method requires pylintrc settings for the msg template that separates each field by a '|' and disabling of the verbose reporting options.

msg-template={path}|{msg_id}|{line},{column}|{msg}
reports=no

"""
Provides lint testing for the roadrage project.
"""
import os

from subprocess import Popen, PIPE
from nose.tools import assert_true  # pylint: disable=E0611


def fail(msg):
    """
    Fails with message.
    """
    assert_true(False, msg)


def test_lint_conformance():
    """
    Collects all lint tests and creates nose errors.
    """
    base = os.path.dirname(os.path.abspath(__file__))
    root = os.path.abspath(os.path.join(base, '..', '..'))
    rcfile = os.path.join(root, '.pylintrc')
    cmd = ['pylint', 'roadrage', '--rcfile={0}'.format(rcfile)]
    proc = Popen(cmd, stdout=PIPE)
    proc.wait()

    errors = [line
              for line in proc.stdout.readlines()
              if not line.startswith('*')]

    for err in errors:
        fields = err.split('|')
        msg = "{path}: {code} ({position}) - {msg}"
        yield fail, msg.format(
            path=fields[0].strip(),
            code=fields[1].strip(),
            position=fields[2].strip(),
            msg=fields[3].strip()
        )

External Links

• PEP8
• PyLint
• Nose