This document lists the requirements for a statement coverage tool
for Python, describes some issues in design and implementation, and
coverage.py with other statement coverage
You can run many tests and perform coverage analysis based on all the tests.
You can get a summary report showing coverage for a set of modules and the total.
You can annotate Python source code to show which statements are covered.
The coverage analysis deals correctly with all Python features.
You can do coverage analysis while testing interactively.
Recording of coverage information doesn’t slow down a test case more than necessary.
You can do coverage analysis for multi-threaded programs.
The coverage tool is portable to all Python versions from 1.5.2 onwards and all operating systems where Python runs.
Requirement 1 means that coverage information needs to be accumulated in a file during a sequence of tests. See [GDR 2001-12-04, 2] for the command-line interface which achieves this. Requirement 5 means that there needs to be a documented programmatic interface. See [GDR 2001-12-04, 3] for the documented interface.
To meet requirement 4 we need to know which source lines represent
statements. Looking for non-blank, non-comment lines isn’t good enough
because some statements extend across many lines, but only the first
line will appear in Python as a source line number. So we use the
Python parser [van Rossum
2001-07-20, 17.1] to parse the module sources and walk the source
tree looking for the first line of each statement (when
symbol.stmt we descend
tree until we get to a
terminal token, whose line we record).
Code on the second and subsequent lines of multi-line simple
statements is reported by the tracing interface as appearing on the
first line of the statement, so recording the first line of each
statement captures all the executed lines except for
finally lines in compound
statements. So we record the lines with these tokens separately. See the
No execution takes place on a line containing only the
else token of
try compound statements, so we don’t record lines
else. But when we annotate a listing, such
a line should be marked as covered if and only if the following
statement is covered. So the
annotate() method has special
logic for this case.
The filename in a Python code object rarely matches the
__file__ attribute of the module to which the code belongs.
There are three kinds of difference:
module.__file__ is the compiled byte code
.pyc), but the filename for the code is the source file.
We work around this by turning
when we find it.
The file names differ as to directory, for example we may have:
module.__file__ == '/dev/project/foo/module.py'but:
code.co_filename == 'module.py'We work around this by searching
sys.pathto find the file.
The code might have been compiled somewhere temporary, for example:
module.__file__ == '/usr/lib/python1.5/getopt.py'but:
code.co_filename == '/var/tmp/python/usr/lib/python1.5/getopt.py'We work around this as a last resort by stripping the directory part and then looking for the file in
How should the coverage module handle programs with multiple threads (requirement 7)? When run from the command line as
python coverage.py -x script.pywe must collect coverage data in all threads. When testing interactively, (at least) three approaches make sense:
Each thread can turn collection on and off independently.
All threads started from a thread inherit the collection status of that thread at the point where the thread is started.
Any thread can turn collection on or off for all threads.
I think it’s unlikely that testers will need fine control of when collection takes place. Most of the time you just turn collection on to begin with and collect everything thereafter. The second and third approaches are suitable for this. Any implementation of the third approach penalises execution speed when collection isn’t taking place. So the second approach is best of these three.
I looked at two other statement coverage testing tools for Python:
trace.py [Dalke 1999]
pycover 0.2 [Csillag
1999-07-01]. Neither met all my requirements. Both have these problems:
No summary report (requirement 2).
Annotations aren’t accurate (requirement 4):
Second and subsequent lines of multi-line statements are marked as not executed.
Lines containing only
else: are marked as not
Comment lines and blank lines are incorrectly recognized and marked as blank if they appear in multi-line strings.
No support for multi-threaded programs (requirement 7).
trace.py also has these problems:
Complicated programmatic interface (requirement 5).
Substantially slower than other coverage testing tools (requirement 6): see table 1.
|Test||Execution time (s)|
|[Csillag 1999-07-01]||“pycover 0.2”; Andrew Csillag; .|
|[Dalke 1999]||“trace.py”; Andrew Dalke; 1999.|
|[GDR 2001-12-04]||“Statement coverage for Python”; Gareth Rees; Ravenbrook Limited; .|
|[van Rossum 2001-07-20]||“Python Reference Manual (release 2.1.1)”; Guido van Rossum; .|