coveralls-python¶
coveralls.io is a service for publishing your coverage stats online. This package provides seamless integration with coverage.py (and thus py.test
, nosetests
, etc…) in your Python projects.
Getting Started¶
Usage¶
This package works with any CI environment. Special handling has been included for some CI service providers, but coveralls-python can run anywhere.
To get started with coveralls-python, make sure to add your repo on the coveralls.io website. If you will be using coveralls-python on TravisCI, you’re done here – otherwise, take note of the “repo token” in the coveralls.io dashboard.
After that, its as simple as installing coveralls-python, collecting coverage results, and sending them to coveralls.io.
For example:
pip install coveralls
coverage run --source=my_package setup.py test
COVERALLS_REPO_TOKEN=tGSdG5Qcd2dcQa2oQN9GlJkL50wFZPv1j coveralls
coveralls-python can be configured with several environment variables, as seen above. See Configuration for more details.
Configuration¶
coveralls-python often works without any outside configuration by examining the environment it is being run in. Special handling has been added for AppVeyor, BuildKite, CircleCI, Github Actions, Jenkins, and TravisCI to make coveralls-python as close to “plug and play” as possible. It should be useable in any other CI system as well, but may need some configuration!
In cases where you do need to modify the configuration, we obey a very strict precedence order where the latest value is used:
- first, the CI environment will be loaded
- second, any environment variables will be loaded (eg. those which begin with
COVERALLS_
- third, the config file is loaded (eg.
./..coveralls.yml
) - finally, any command line flags are evaluated
Most often, you will simply need to run coveralls-python with no additional options after you have run your coverage suite:
coveralls
If you have placed your .coveragerc
in a non-standard location, you can run:
coveralls --rcfile=/path/to/coveragerc
If you would like to override the service name (auto-discovered on most CI systems, set to coveralls-python
otherwise):
coveralls --service=travis-pro
# or, via env var:
COVERALLS_SERVICE_NAME=travis-pro coveralls
If you are interested in merging the coverage results between multiple languages/projects, see our multi-language documentation.
If coveralls-python is being run on TravisCI or on GitHub Actions, it will automatically set the token for communication with coveralls.io. Otherwise, you should set the environment variable COVERALLS_REPO_TOKEN
, which can be found on the dashboard for your project in coveralls.io:
COVERALLS_REPO_TOKEN=mV2Jajb8y3c6AFlcVNagHO20fiZNkXPVy coveralls
If you are running multiple jobs in parallel and want coveralls.io to merge those results, you should set COVERALLS_PARALLEL
to true
in your environment:
COVERALLS_PARALLEL=true coveralls
Later on, you can use coveralls --finish
to let the Coveralls service know you have completed all your parallel runs:
coveralls --finish
If you are using a non-public coveralls.io instance (for example: self-hosted Coveralls Enterprise), you can set COVERALLS_HOST
to the base URL of that insance:
COVERALLS_HOST="https://coveralls.aperture.com" coveralls
In that case, you may also be interested in disabling SSL verification:
COVERALLS_SKIP_SSL_VERIFY='1' coveralls
If you are using named jobs, you can set:
COVERALLS_FLAG_NAME="insert-name-here"
You can also set any of these values in a .coveralls.yml
file in the root of your project repository. If you are planning to use this method, please ensure you install coveralls[yaml]
instead of just the base coveralls
package.
Sample .coveralls.yml
file:
service_name: travis-pro
repo_token: mV2Jajb8y3c6AFlcVNagHO20fiZNkXPVy
parallel: true
coveralls_host: https://coveralls.aperture.com
Github Actions support¶
Coveralls natively supports jobs running on Github Actions. You can directly pass the default-provided secret GITHUB_TOKEN:
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
coveralls --service=github
Passing a coveralls.io token via the COVERALLS_REPO_TOKEN
environment variable
(or via the repo_token
parameter in the config file) is not needed for
Github Actions.
Sometimes Github Actions gets a little picky about the service name which needs
to be used in various cases. If you run into issues, try setting the
COVERALLS_SERVICE_NAME
explicitly to either github
or
github-actions
. It seems to be the case that you should use the
--service=github
value if you are also planning to use the GITHUB_TOKEN
env var, and github-actions
(which is the default) in any other case, but
we’ve have conflicting reports on this: YMMV! See
#452 for more
info.
For parallel builds, you have to add a final step to let coveralls.io know the parallel build is finished:
jobs:
test:
strategy:
matrix:
test-name:
- test1
- test2
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Test
run: ./run_tests.sh ${{ matrix.test-name }}
- name: Upload coverage data to coveralls.io
run: coveralls --service=github
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
COVERALLS_FLAG_NAME: ${{ matrix.test-name }}
COVERALLS_PARALLEL: true
coveralls:
name: Indicate completion to coveralls.io
needs: test
runs-on: ubuntu-latest
container: python:3-slim
steps:
- name: Finished
run: |
pip3 install --upgrade coveralls
coveralls --service=github --finish
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
The COVERALLS_FLAG_NAME
environment variable (or the flag_name
parameter
in the config file) is optional and can be used to better identify each job
on coveralls.io. It does not need to be unique across the parallel jobs.
Azure Pipelines support¶
Coveralls does not yet support Azure Pipelines, but you can make things work by impersonating another CI system such as CircleCI. For example, you can set this up by using the following script at the end of your test pipeline:
- script: |
pip install coveralls
export CIRCLE_BRANCH=$BUILD_SOURCEBRANCH
coveralls
displayName: 'coveralls'
env:
CIRCLECI: 1
CIRCLE_BUILD_NUM: $(Build.BuildNumber)
COVERALLS_REPO_TOKEN: $(coveralls_repo_token)
Note that you will also need to use the Azure Pipelines web UI to add the
coveralls_repo_token
variable to this pipeline with your repo token (which
you can copy from the coveralls.io website).
As per #245, our users suggest leaving “keep this value secret” unchecked – this may be secure enough as-is, in that a user making a PR cannot access this variable.
Other CI systems¶
As specified in the Coveralls official docs <https://docs.coveralls.io/supported-ci-services> other CI systems can be supported if the following environment variables are defined:
CI_NAME
# Name of the CI service being used.
CI_BUILD_NUMBER
# The number assigned to the build by your CI service.
CI_BUILD_URL
# URL to a webpage showing the build information/logs.
CI_BRANCH
# For pull requests this is the name of the branch being targeted,
# otherwise it corresponds to the name of the current branch or tag.
CI_JOB_ID (optional)
# For parallel builds, the number assigned to each job comprising the build.
# When missing, Coveralls will assign an incrementing integer (1, 2, 3 ...).
# This value should not change between multiple runs of the build.
CI_PULL_REQUEST (optional)
# If given, corresponds to the number of the pull request, as specified
# in the supported repository hosting service (GitHub, GitLab, etc).
# This variable expects a value defined as an integer, e.g.:
# CI_PULL_REQUEST=42 (recommended)
# However, for flexibility, any single line string ending with the same
# integer value can also be used (such as the pull request URL or
# relative path), e.g.:
# CI_PULL_REQUEST='myuser/myrepo/pull/42'
# CI_PULL_REQUEST='https://github.com/myuser/myrepo/pull/42'
VCS Configuration¶
coveralls-python
supports git
by default and will run the necessary git
commands to collect the required information without any intervention.
As describe in the coveralls docs, you may also configure these values by setting environment variables. These will be used in the fallback case, eg. if git
is not available or your project is not a git
repository.
As described in the linked documentation, you can also use this method to support non- git
projects:
GIT_ID=$(hg tip --template '{node}\n')
GIT_AUTHOR_NAME=$(hg tip --template '{author|person}\n')
GIT_AUTHOR_EMAIL=$(hg tip --template '{author|email}\n')
GIT_COMMITTER_NAME=$(hg tip --template '{author|person}\n')
GIT_COMMITTER_EMAIL=$(hg tip --template '{author|email}\n')
GIT_MESSAGE=$(hg tip --template '{desc}\n')
GIT_BRANCH=$(hg branch)
Usage Within Tox¶
Running coveralls-python from within a tox environment (v2.0 and above) requires an additional step; since coveralls-python relies on environment variables to function, you’ll need to configure tox to capture those variables using the passenv
configuration option in your tox.ini
.
For example, on TravisCI:
[tox]
envlist = py34,py35,py36,py37,py38
[testenv]
passenv = TRAVIS TRAVIS_*
deps =
coveralls
commands =
coverage run --source=yourpackagename setup.py test
coveralls
If you are configuring coveralls-python with environment variables, you should also pass those. See Configuration for more details.
AppVeyor¶
passenv = APPVEYOR APPVEYOR_*
All variables:
APPVEYOR
APPVEYOR_BUILD_ID
APPVEYOR_REPO_BRANCH
APPVEYOR_PULL_REQUEST_NUMBER
BuildKite¶
passenv = BUILDKITE BUILDKITE_*
All variables:
BUILDKITE
BUILDKITE_JOB_ID
BUILDKITE_BRANCH
CircleCI¶
passenv = CIRCLECI CIRCLE_* CI_PULL_REQUEST
All variables:
CIRCLECI
CIRCLE_BUILD_NUM
CIRCLE_BRANCH
CI_PULL_REQUEST
Github Actions¶
passenv = GITHUB_*
All variables:
GITHUB_ACTIONS
GITHUB_REF
GITHUB_SHA
GITHUB_HEAD_REF
GITHUB_REPOSITORY
GITHUB_RUN_ID
GITHUB_TOKEN
Jenkins¶
passenv = JENKINS_HOME BUILD_NUMBER GIT_BRANCH CI_PULL_REQUEST
All variables:
JENKINS_HOME
BUILD_NUMBER
GIT_BRANCH
CI_PULL_REQUEST
TravisCI¶
passenv = TRAVIS TRAVIS_*
All variables:
TRAVIS
TRAVIS_JOB_ID
TRAVIS_BRANCH
TRAVIS_PULL_REQUEST
SemaphoreCI¶
Classic¶
passenv = SEMAPHORE SEMAPHORE_EXECUTABLE_UUID SEMAPHORE_JOB_UUID SEMAPHORE_BRANCH_ID BRANCH_NAME
All variables:
SEMAPHORE
SEMAPHORE_EXECUTABLE_UUID
SEMAPHORE_JOB_UUID
SEMAPHORE_BRANCH_ID
BRANCH_NAME
2.0¶
passenv = SEMAPHORE SEMAPHORE_WORKFLOW_ID SEMAPHORE_JOB_ID SEMAPHORE_GIT_PR_NUMBER BRANCH_NAME
All variables:
SEMAPHORE
SEMAPHORE_WORKFLOW_ID
SEMAPHORE_JOB_ID
SEMAPHORE_GIT_PR_NUMBER
BRANCH_NAME
Multiple Language Support¶
Tracking multi-language repo coverage requires an extra setup of merging coverage data for submission.
To send coveralls.io merged data, you must use each of your coverage reporting tools in sequence, then merge the JSON data in the last step.
For example, to submit coverage for a project using both mocha
and py.test
, you could use the coveralls-lcov library and run:
# generate mocha coverage data
mocha --reporter mocha-lcov-reporter */tests/static/js/* > coverage.info
# convert data with coveralls-lcov
coveralls-lcov -v -n coverage.info > coverage.json
# merge mocha coverage with python coverage and send to coveralls
coveralls --merge=coverage.json
If you want to use this library to create a JSON blob for usage elsewhere, you can run:
coveralls --output=coverage.json
Technical Details¶
The JSON file to be merged must be of “coveralls-style” and contain thus a source_files
key. The Coveralls API has more information.
Tips for .coveragerc¶
This section is a list of most common options for coverage.py
, which collects all the coverage information. Coveralls is populated from this data, so it’s good to know how to to configure coverage.py.
To limit the report to only your packages, specify their names (or directories):
[run]
source = pkgname,your_otherpackage
To exclude parts of your source from coverage, for example migrations folders:
[report]
omit = */migrations/*
Some lines are never executed in your tests, but that can be ok. To mark those lines use inline comments right in your source code:
if debug: # pragma: no cover
msg = "blah blah"
log_message(msg, a)
Sometimes it can be tedious to mark them in code, so you can specify whole lines in .coveragerc:
[report]
exclude_lines =
pragma: no cover
def __repr__
raise AssertionError
raise NotImplementedError
if __name__ == .__main__.:
Finally, if you’re using non-default configuration file, you can specify it in the coveralls command:
$ coveralls --rcfile=<file>
Nosetests¶
Nosetests provide a plugin for coverage measurement of your code:
$ nosetests --with-coverage --cover-package=<your_package_name>
However, nosetests gathers coverage for all executed code, ignoring the source
config option in .coveragerc
.
This well make coveralls
report unnecessary files, which can be inconvenient. To workaround this issue, you can use the omit
option in your .coveragerc
to specify a list of filename patterns to leave out of reporting.
For example:
[report]
omit =
*/venv/*
*/my_project/ignorable_file.py
*/test_script.py
Note, that native coverage.py
and py.test
are not affected by this problem and do not require this workaround.
Troubleshooting¶
If you are having difficulties submitting your coverage to coveralls.io, debug mode may help you figure out the problem:
$ coveralls debug
Debug mode doesn’t send anything, it just outputs prepared json and reported files list to stdout.
We also have an issue tracker on GitHub.
About¶
Administration¶
Release¶
This project is released on PyPI as coveralls.
To cut a new release, ensure the latest master passes all tests. Then, create a release commit:
- Update the
CHANGELOG.md
with the new version (clog -C CHANGELOG.md -F --setversion x.y.z
). - Bump the version number in
version.py
. - Commit and push (
git commit -am 'chore(release): bump version' && git push
) - Tag and push that commit with the version number (
git tag x.y.z && git push origin x.y.z
). - Create a new GitHub release.
To create a new PyPI release, do the following:
- Build the sources (
python setup.py sdist bdist_wheel
). - Register & upload the sources. (
twine upload $PWD/dist/*
).
Then, to pin a new docker release, do the following:
- Build the new image (
docker build --build-arg COVERALLS="coveralls==x.y.z" -t thekevjames/coveralls:x.y.z .
. - Push it to dockerhub (
docker push thekevjames/coveralls:x.y.z
). - Note: the
:latest
tag will be handled automatically by Dockerhub’s automated infrastructure.
Conda should automatically create a PR on their coveralls-feedstock shortly with the updated version – if something goes wrong, the manual process would be to:
- Fork coveralls-feedstock.
- Update
recipe/meta.yaml
with the new version number and sha. - Create a PR.
- Comment on your own PR with: “@conda-forge-admin, please rerender”.
- Merge along with the automated commit from Conda.