Statement coverage for Python

Gareth Rees, 2001-12-04

1. Introduction

The coverage.py Python module provides statement coverage for Python. It accumulates coverage data over many runs; generates coverage reports; and annotates Python source showing which statements have been covered.

To use it, download coverage.py and add it to your Python library (for example, in /usr/local/lib/python-version/lib/site-packages/) and your execution path.

See [GDR 2001-12-04] for design and analysis.

2. Command-line interface

Use the -x option (--execute) to execute a Python module with command-line arguments, and record coverage information:

$ coverage.py -x module.py [ARG1 ARG2 ...]

Coverage information accumulates over a sequence of executions, in the file .coverage (set the COVERAGE_FILE environment variable if you want to use a different file). To erase the recorded coverage information, use the -e option (--erase):

$ coverage.py -e

To report on the statement coverage for a set of files, use the -r option (--report):

$ coverage.py -r [-m] FILE1 FILE2 ...

Specify the -m option (--show-missing) to show the the line numbers of statements that weren’t excecuted. For example:

$ coverage.py -e
$ coverage.py -x test_foo.py
Test errors (test_foo.error) ... ok
Test options (test_foo.options) ... ok
--------------------------------------
Ran 1 test in 0.966s

OK
$ coverage.py -r -m foo.py bar.py
Name    Stmts   Exec  Cover  Missing
------------------------------------
foo        64     56    87%  23, 57, 85, 119, 125, 133, 137, 152
bar       105     90    86%  78-86, 237-246
------------------------------------
TOTAL     169    146    86%
$

To make a copy of source code annotated with > for statements that are executed, and ! for statements that are not, use the -a option (--annotate):

$ coverage.py -a FILE1 FILE2 ...

The annotated copy of file is written as file,cover. Normally each annotated copy is written in the same directory as the original. Add the -d DIRECTORY option (--directory=DIRECTORY) to create the annotated copies in specified directory. For example:

$ coverage.py -a -d . /project/src/foo.py
$ cat foo.py,cover
...
>          length = self.msg.getheader('content-length')
>          if length:
>              try:
>                  self.length = int(length)
!              except ValueError:
!                  self.length = 0
>              self.length = max(self.length, 0)
!          else:
!              self.length = 0
...
$

3. Programmatic interface

When testing interactively, you’ll need to use the programmatic interface.

Call start() to start recording coverage:

>>> import coverage
>>> coverage.start()

Stop recording by calling stop(). Call erase() to erase all results (including the .coverage file).

To calculate the coverage of a module or file, call analysis(), passing a module object or a filename. It returns a 4-tuple consisting of

1. The module’s filename.

2. A list of line numbers of statements in the module.

3. A list of line numbers of statements that weren’t executed.

4. A human-readable string describing the line numbers of statements that weren’t executed, coalescing groups of adjacent statements.

For example:

>>> import coverage
>>> coverage.erase()
>>> coverage.start()
>>> import spong
>>> spong.run_tests()
>>> coverage.stop()
>>> coverage.analysis(spong)
('/project/foo/tests/foo.py',
 [10, 11, 12, 13, 17, 18, 20, 21, 23],
 [12, 13, 17, 21, 23],
 '12-17, 21-23')

report() prints a coverage report. Pass it a list of modules or filenames, and give optional argument ignore_errors = 1 to ignore errors in analyzing modules; and show_missing = 0 to hide the missing statements.

For example:

>>> coverage.report([foo, bar])
Name    Stmts   Exec  Cover  Missing
------------------------------------
foo        64     56    87%  23, 57, 85, 119, 125, 133, 137, 152
bar       105     90    86%  78-86, 237-246
------------------------------------
TOTAL     169    146    86%
>>> m = sys.modules.values()
>>> coverage.report(m, ignore_errors=1, show_missing=0)
Name       Stmts   Exec  Cover
------------------------------
types         47      0     0%
foo           64     56    87%
token        111      0     0%
whrandom      69     21    30%
bar          105     90    86%
coverage     141      2     1%
unittest     400     94    23%
string       121     15    12%
...

4. Limitations

• You can’t record coverage data while tracing or profiling.

• Statement coverage is the weakest measure of code coverage. It can’t tell you when an if statement is missing an else clause ("branch coverage"); when a condition is only tested in one direction ("condition coverage"); when a loop is always taken and never skipped ("loop coverage"); and so on. See [Kaner 2000-10-17] for a summary of test coverage measures.

  Also, just because a statement is covered by a test case doesn’t mean that the test case checks that the statement does the right thing. So it’s not a good idea to organize your testing around statement coverage or to set targets for statement coverage. However, coverage testing is a good tool for discovering features that haven’t been tested and suggesting profitable new areas to test. See [Marick 1997] for more discussion of the use of coverage measures.

  Here are some particular cases in Python where statement coverage is inadequate:

  • Short-circuit evaluation. The expression 0 and foo() is considered to be executed when the statement it belongs to is executed, but foo() is not called. Similarly for 1 or foo().

  • Lambda expressions. lambda: foo() is considered to be executed when the lambda form is evaluated. But foo() is not called.

  • Assertions. assert 1, foo() is considered to have been executed if the statement is reached. But foo() is not called.

• Top-level statements are executed when a module is loaded. So if you import a module before starting to record coverage, those top-level statements are missed. To get a better idea, start recording before importing the module, or reload() the module.

• The coverage module can’t tell when coverage information is out of date with respect to the source code. So if you edit Python source code while recording coverage information, reports and annotations might be wrong. You should erase coverage information after you edit source code.

A. References

B. Document History