Develop

Requires

• git

• Make. See Makefile for a list of make commands.

Developing documentation requires:

• pandoc for interpreting Jupyter notebooks

• LaTeX for pdf outputs. If you are using a Mac, MacTeX is the best option. Note that LaTeX can be fairly large to install (~6GB).

Environment

Create a dedicated virtual environment for development:

# create environment in <DIR>
$ python3 -m venv <DIR>

# activate environment (Unix-like)
$ source <DIR>/bin/activate

If using Anaconda / Miniconda Python, create a dedicated Anaconda environment:

# create conda environment
$ conda create -n contrails

# activate environment
$ conda activate contrails

Install

After activating the virtual environment, clone the pycontrails repository:

$ cd <install-path>
$ git clone git@github.com:contrailcirrus/pycontrails.git
$ cd pycontrails

These commands clone via SSH and may require adding an SSH key to your GitHub account. Alternatively, you can clone via HTTPS by running

$ cd <install-path>
$ git clone https://github.com/contrailcirrus/pycontrails.git
$ cd pycontrails

Install the development verison of pycontrails using make:

$ make dev-install

or install dependencies manually using pip in editable mode:

# core development installation
$ pip install -e ".[docs,dev]"

# install optional dependencies as above
$ pip install -e ".[ecmwf,gfs]"

# make sure to add pre-commit hooks if installing manually
$ pre-commit install

Test

Run all code quality checks and unit tests. This is run in the test workflow, but should also be run locally before submitting PRs:

$ make test

Lint the repository with ruff:

$ make ruff

Autoformat the repository with black:

$ make black

Run type checking with mypy:

$ make mypy

Run unit tests with pytest:

$ make pytest

Run notebook validation with nbval:

$ make nb-test

Run doctests with pytest:

$ make doctest

Notebook validation and doctests require Copernicus Climate Data Store (CDS) credentials, and doctests additionally require Google application credentials. If either are missing, the test suite will issue a warning and exit.

Documentation

Documentation is written in reStructuredText (rst) and built with Sphinx. The quick reStructuredText reference provides a decent rst syntax overview.

Sphinx includes many additional roles, directives, and extensions to enhance documentation.

Sphinx configuration is written in docs/conf.py. See the Sphinx configuration docs for the full list of configuration options.

Build HTML documentation:

# docs build to directory docs/_build/html
$ make docs-build

# automatically build docs on changes
# docs will be served at http://127.0.0.1:8000
$ make docs-serve

# clean up built documentation
$ make docs-clean

Build manually with sphinx-build:

$ sphinx-build -b html docs docs/_build/html      # HTML output

Sphinx caches builds between changes. To force the whole site to rebuild, use the options -aE:

$ sphinx-build -aE -b html docs docs/_build/html  # rebuild all output

See sphinx-build for a list of all the possible output builders.

Notebooks

Examples and tutorials should be written as isolated executable Jupyter Notebooks. The nbsphinx extension includes notebooks in the static documentation.

Notebooks will be automatically evaluated during tests, unless explicitly ignored. To exclude a notebook cell from evaluation during testing or automatic execution, add the tags nbval-skip and skip-execution to cell metadata.

To test notebooks locally, run:

$ make nb-test

To re-execute all notebooks, run:

$ make nb-execute

PDF Output

Building PDF output requires a LaTeX distribution.

Build pdf documentation:

$ make docs-pdf

A single pdf output (i.e. pycontrails.pdf) will be built within docs/_build/latex.

To build manually, run:

$ sphinx-build -b latex docs docs/_build/latex
$ cd docs/_build/latex
$ make

References

Literature references managed in the pycontrails Zotero library.

The documentation uses sphinxcontrib-bibtex to include citations and a bibliography.

All references should be cited through documentation and docstrings using the :cite: role.

To automatically sync the Zotero library with the docs/_static/pycontrails.bib Bibtex file:

• Install Zotero and add the pycontrails library.

• Install the Zotero Better Bibtex extension. Leave defaults during setup.

• Right click on the pycontrails library and select Export Library

• Export as Better Bibtex. You can optionally check Keep Updated if you want this file to update every time you make a change to the library.

• Select the file _static/pycontrails.bib and press Save to overwrite the file.

• Commit the updated _static/pycontrails.bib

Test

All doc tests first ensure ERA5 data is cached locally:

$ make ensure-era5-cached

Run rst linter with doc8:

$ make doc8

Run docstring example tests with doctest:

$ make doctest

Test notebook examples with nbval pytest plugin:

$ make nb-test

Conventions

Code

pycontrails aims to implement clear, consistent, performant data structures and models.

The project uses mypy for static type checking. All code should have specific, clear type annotations.

The project uses Black, ruff and doc8 to standardize code-formatting. These tools are run automatically in a pre-commit hook.

The project uses pytest to run unit tests. New code should include clear unit tests for implementation and output values. New test files should be included in the /tests/unit/ directory.

The project uses Github Actions to run code quality and unit tests on each pull request. Test locally before pushing using:

$ make test

Docstrings

Wherever possible, we adhere to the NumPy docstring conventions.

The following links are good references for writing numpy docstrings:

• numpydoc docstring guide

• pandas docstring guide

• scipy docstring guideline

General guidelines:

Use *italics*, **bold** and ``monospace`` if needed in any explanations
(but not for variable names and doctest code or multi-line code).
Variable, module, function, and class names
should be written between single back-ticks (`numpy`).

When specifying types in Parameters or See Also, Sphinx will automatically replace the text with the napolean_type_aliases specified in conf.py, e.g.

"""
Parameters
----------
x : np.ndarray
    Sphinx will automatically replace
    "np.ndarray" with the napolean type alias "numpy.ndarray"
"""

The See Also section is not a list. All of the following work:

"""
See Also
--------
:func:`segment_lengths`
segment_lengths
:class:`numpy.datetime64`
np.datetime64
"""

When you specify a type outside of Parameters, you have to use the sphinx cross-referencing syntax with the full module name:

"""
This is a :func:`pd.to_datetime`    # NO
and :func:`pandas.to_datetime`      # YES

This is a :class:`np.datetime64`    # NO
and :class:`numpy.datetime64`       # YES
"""