Update 2013-05-15. coverage.py has been maintained by Ned Batchelder for some years now: please use his version and take advantage of the many improvements he has made.
The coverage.py Python
module provides statement coverage for Python. It accumulates coverage
data over many runs; generates coverage reports; and annotates Python
source showing which statements have been covered. To use it,
download coverage.py and add it
to your Python library (for example,
in /usr/local/lib/python-version/lib/site-packages/)
and your execution path. See [GDR
2001-12-04] for design and analysis.
Use the -x option (--execute) to execute a
Python module with command-line arguments, and record coverage
information:
$ coverage.py -x module.py [ARG1 ARG2 ...] Coverage information accumulates over a sequence of executions, in
the file .coverage (set the COVERAGE_FILE
environment variable if you want to use a different file). To erase the
recorded coverage information, use the -e option
(--erase):
$ coverage.py -e To report on the statement coverage for a set of files, use the
-r option (--report):
$ coverage.py -r [-m] FILE1 FILE2 ... Specify the -m option (--show-missing) to
show the the line numbers of statements that weren’t excecuted. For
example:
$ coverage.py -e
$ coverage.py -x test_foo.py
Test errors (test_foo.error) ... ok
Test options (test_foo.options) ... ok
--------------------------------------
Ran 1 test in 0.966s
OK
$ coverage.py -r -m foo.py bar.py
Name Stmts Exec Cover Missing
------------------------------------
foo 64 56 87% 23, 57, 85, 119, 125, 133, 137, 152
bar 105 90 86% 78-86, 237-246
------------------------------------
TOTAL 169 146 86%
$
To make a copy of source code annotated with > for
statements that are executed, and ! for statements that are
not, use the -a option (--annotate):
$ coverage.py -a FILE1 FILE2 ... The annotated copy of file is written as
file,cover. Normally each annotated copy is written in the
same directory as the original. Add the -d
DIRECTORY option
(--directory=DIRECTORY) to create the annotated
copies in specified directory. For example:
$ coverage.py -a -d . /project/src/foo.py
$ cat foo.py,cover
...
> length = self.msg.getheader('content-length')
> if length:
> try:
> self.length = int(length)
! except ValueError:
! self.length = 0
> self.length = max(self.length, 0)
! else:
! self.length = 0
...
$
When testing interactively, you’ll need to use the programmatic interface.
Call start() to start recording coverage:
>>> import coverage
>>> coverage.start()
Stop recording by calling stop(). Call
erase() to erase all results (including the
.coverage file). To calculate the coverage of a module or
file, call
analysis(), passing a module object or a filename. It
returns a 4-tuple consisting of
The module’s filename.
A list of line numbers of statements in the module.
A list of line numbers of statements that weren’t executed.
A human-readable string describing the line numbers of statements that weren’t executed, coalescing groups of adjacent statements.
For example:
>>> import coverage
>>> coverage.erase()
>>> coverage.start()
>>> import spong
>>> spong.run_tests()
>>> coverage.stop()
>>> coverage.analysis(spong)
('/project/foo/tests/foo.py',
[10, 11, 12, 13, 17, 18, 20, 21, 23],
[12, 13, 17, 21, 23],
'12-17, 21-23')
report() prints a coverage report. Pass it a list of
modules or filenames, and give optional argument ignore_errors =
1 to ignore errors in analyzing modules; and
show_missing = 0 to hide the missing statements. For
example:
>>> coverage.report([foo, bar])
Name Stmts Exec Cover Missing
------------------------------------
foo 64 56 87% 23, 57, 85, 119, 125, 133, 137, 152
bar 105 90 86% 78-86, 237-246
------------------------------------
TOTAL 169 146 86%
>>> m = sys.modules.values()
>>> coverage.report(m, ignore_errors=1, show_missing=0)
Name Stmts Exec Cover
------------------------------
types 47 0 0%
foo 64 56 87%
token 111 0 0%
whrandom 69 21 30%
bar 105 90 86%
coverage 141 2 1%
unittest 400 94 23%
string 121 15 12%
...
You can’t record coverage data while tracing or profiling.
Statement coverage is the weakest measure of code coverage.
It can’t tell you when an if statement is missing an
else clause ("branch coverage"); when a condition is only
tested in one direction ("condition coverage"); when a loop is always
taken and never skipped ("loop coverage"); and so on. See [Kaner 2000-10-17] for a
summary of test coverage measures.
Also, just because a statement is covered by a test case doesn’t mean that the test case checks that the statement does the right thing. So it’s not a good idea to organize your testing around statement coverage or to set targets for statement coverage. However, coverage testing is a good tool for discovering features that haven’t been tested and suggesting profitable new areas to test. See [Marick 1997] for more discussion of the use of coverage measures.
Here are some particular cases in Python where statement coverage is inadequate:
Short-circuit evaluation. The expression 0 and
foo() is considered to be executed when the statement it
belongs to is executed, but foo() is not called.
Similarly for 1 or foo().
Lambda expressions. lambda: foo() is
considered to be executed when the lambda form is
evaluated. But foo() is not called.
Assertions. assert 1, foo() is considered to
have been executed if the statement is reached. But
foo() is not called.
Top-level statements are executed when a module is loaded. So
if you import a module before starting to record coverage, those
top-level statements are missed. To get a better idea, start
recording before importing the module, or reload() the
module.
The coverage module can’t tell when coverage information is out of date with respect to the source code. So if you edit Python source code while recording coverage information, reports and annotations might be wrong. You should erase coverage information after you edit source code.
| GDR | Created. | |
| GDR | Document -d option for annotation. |