ECMWF#

Requires [ecmwf] optional dependencies:
$ pip install pycontrails[ecmwf]

Support provided for:

ERA5 via the Copernicus Data Store (CDS) using cdsapi or user provided files
HRES and ENS via MARS using ecmwf-api-client or user provided files

ERA5#

Access#

Requires account with Copernicus Data Portal
Provide url and key credentials on input, or refer to the CDS API Documentation for how to create ~/.cdsapirc file to configure access.

Reference#

ERA5 Pressure Levels#

ERA5 Pressure Level Parameters

[1]:

from pycontrails.datalib.ecmwf import ERA5

[2]:

# get a single time
era5 = ERA5(
    time="2022-03-01 00:00:00",
    variables=["t", "q", "u", "v", "w", "ciwc", "z", "cc"],  # supports CF name or short names
    pressure_levels=[300, 250, 200],
    # url="https://cds.climate.copernicus.eu/api/v2",
    # key="<key>"
)
era5

[2]:

ERA5
        Timesteps: ['2022-03-01 00']
        Variables: ['t', 'q', 'u', 'v', 'w', 'ciwc', 'z', 'cc']
        Pressure levels: [300, 250, 200]
        Grid: 0.25
        Dataset: reanalysis-era5-pressure-levels
        Product type: reanalysis

[3]:

# get a range of time
era5 = ERA5(
    time=("2022-03-01 00:00:00", "2022-03-01 03:00:00"),
    variables=[
        "air_temperature",
        "q",
        "u",
        "v",
        "w",
        "ciwc",
        "z",
        "cc",
    ],  # supports CF name or short names
    pressure_levels=[300, 250, 200],
    # url="https://cds.climate.copernicus.eu/api/v2",
    # key="<key>"
)
era5

[3]:

ERA5
        Timesteps: ['2022-03-01 00', '2022-03-01 01', '2022-03-01 02', '2022-03-01 03']
        Variables: ['t', 'q', 'u', 'v', 'w', 'ciwc', 'z', 'cc']
        Pressure levels: [300, 250, 200]
        Grid: 0.25
        Dataset: reanalysis-era5-pressure-levels
        Product type: reanalysis

[4]:

# this triggers a download from CDS if file isn't in cache store
met = era5.open_metdataset()
met

[4]:

MetDataset with data:

<xarray.Dataset>
Dimensions:                              (longitude: 1440, latitude: 721,
                                          level: 3, time: 4)
Coordinates:
  * longitude                            (longitude) float64 -180.0 ... 179.8
  * latitude                             (latitude) float64 -90.0 ... 90.0
  * level                                (level) float64 200.0 250.0 300.0
  * time                                 (time) datetime64[ns] 2022-03-01 ......
    air_pressure                         (level) float32 2e+04 2.5e+04 3e+04
    altitude                             (level) float32 1.178e+04 ... 9.164e+03
Data variables:
    air_temperature                      (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 3, 1), meta=np.ndarray>
    specific_humidity                    (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 3, 1), meta=np.ndarray>
    eastward_wind                        (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 3, 1), meta=np.ndarray>
    northward_wind                       (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 3, 1), meta=np.ndarray>
    lagrangian_tendency_of_air_pressure  (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 3, 1), meta=np.ndarray>
    specific_cloud_ice_water_content     (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 3, 1), meta=np.ndarray>
    geopotential                         (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 3, 1), meta=np.ndarray>
    fraction_of_cloud_cover              (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 3, 1), meta=np.ndarray>
Attributes:
    Conventions:          CF-1.6
    history:              2022-09-28 13:04:39 GMT by grib_to_netcdf-2.25.1: /...
    pycontrails_version:  0.42.1.dev24
    met_source:           ERA5

ERA5 Single Level#

[5]:

era5 = ERA5(
    time=("2022-03-01 00:00:00", "2022-03-01 03:00:00"),
    variables=["tsr", "ttr"],
    # url="https://cds.climate.copernicus.eu/api/v2",
    # key="<key>"
)
era5

[5]:

ERA5
        Timesteps: ['2022-03-01 00', '2022-03-01 01', '2022-03-01 02', '2022-03-01 03']
        Variables: ['tsr', 'ttr']
        Pressure levels: [-1]
        Grid: 0.25
        Dataset: reanalysis-era5-single-levels
        Product type: reanalysis

[6]:

met = era5.open_metdataset()
met

[6]:

MetDataset with data:

<xarray.Dataset>
Dimensions:                    (longitude: 1440, latitude: 721, level: 1,
                                time: 4)
Coordinates:
  * level                      (level) float64 -1.0
  * longitude                  (longitude) float64 -180.0 -179.8 ... 179.5 179.8
  * latitude                   (latitude) float64 -90.0 -89.75 ... 89.75 90.0
  * time                       (time) datetime64[ns] 2022-03-01 ... 2022-03-0...
Data variables:
    top_net_solar_radiation    (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 1, 1), meta=np.ndarray>
    top_net_thermal_radiation  (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 1, 1), meta=np.ndarray>
Attributes:
    Conventions:          CF-1.6
    history:              2022-09-28 13:06:13 GMT by grib_to_netcdf-2.25.1: /...
    pycontrails_version:  0.42.1.dev24
    met_source:           ERA5

HRES#

Access#

Users within ECMWF Member and Co-operating States may contact their Computing Representative to obtain access to MARS. All other users may request a username and password and then get an api key.

Provide url, key, and email credentials on input, or see ECMWF API Client documentation to configure local ~/.ecmwfapirc file:

{
    "url": "https://api.ecmwf.int/v1",
    "email": "<email>",
    "key": "<key>"
}

Reference#

HRES High resolution forecast
ENS Ensemble forecast

HRES Pressure Levels#

[7]:

from datetime import datetime

from pycontrails.datalib.ecmwf import HRES

[8]:

# NOTE / TODO: Including the "ciwc" variable here, the HRES request
# fails with on historic data. However, the request seems to go through
# when the time field is recent (within the last 48 hours?)
time = datetime(2022, 3, 26, 0), datetime(2022, 3, 26, 2)
hres = HRES(
    time=time,
    variables=["t", "q", "u", "v", "w", "z"],
    pressure_levels=[300, 250, 200],
    grid=1,
    # url="https://api.ecmwf.int/v1",
    # key="<key>"
    # email="<email>"
)
hres

[8]:

HRES
        Timesteps: ['2022-03-26 00', '2022-03-26 01', '2022-03-26 02']
        Variables: ['t', 'q', 'u', 'v', 'w', 'z']
        Pressure levels: [300, 250, 200]
        Grid: 1
        Forecast time: 2022-03-26 00:00:00
        Steps: [0, 1, 2]

[9]:

# convience method to see the underlying MARS request
print(hres.generate_mars_request())

retrieve,
        class=od,
        stream=oper,
        expver=1,
        date=20220326,
        time=00,
        type=fc,
        param=t/q/u/v/w/z,
        step=0/1/2,
        grid=1/1,
        levtype=pl,
        levelist=300/250/200

[10]:

# this triggers a download if file isn't in cache store
met = hres.open_metdataset()
met

[10]:

MetDataset with data:

<xarray.Dataset>
Dimensions:                              (longitude: 360, latitude: 181,
                                          level: 3, time: 3)
Coordinates:
    forecast_time                        datetime64[ns] 2022-03-26
  * level                                (level) float64 200.0 250.0 300.0
  * latitude                             (latitude) float64 -90.0 -89.0 ... 90.0
  * longitude                            (longitude) float64 -180.0 ... 179.0
  * time                                 (time) datetime64[ns] 2022-03-26 ......
    air_pressure                         (level) float64 2e+04 2.5e+04 3e+04
    altitude                             (level) float64 1.178e+04 ... 9.164e+03
Data variables:
    air_temperature                      (longitude, latitude, level, time) float32 dask.array<chunksize=(360, 181, 3, 1), meta=np.ndarray>
    specific_humidity                    (longitude, latitude, level, time) float32 dask.array<chunksize=(360, 181, 3, 1), meta=np.ndarray>
    eastward_wind                        (longitude, latitude, level, time) float32 dask.array<chunksize=(360, 181, 3, 1), meta=np.ndarray>
    northward_wind                       (longitude, latitude, level, time) float32 dask.array<chunksize=(360, 181, 3, 1), meta=np.ndarray>
    lagrangian_tendency_of_air_pressure  (longitude, latitude, level, time) float32 dask.array<chunksize=(360, 181, 3, 1), meta=np.ndarray>
    geopotential                         (longitude, latitude, level, time) float32 dask.array<chunksize=(360, 181, 3, 1), meta=np.ndarray>
Attributes:
    GRIB_edition:            1
    GRIB_centre:             ecmf
    GRIB_centreDescription:  European Centre for Medium-Range Weather Forecasts
    GRIB_subCentre:          0
    Conventions:             CF-1.7
    institution:             European Centre for Medium-Range Weather Forecasts
    history:                 2023-01-11T11:54 GRIB to CDM+CF via cfgrib-0.9.1...
    pycontrails_version:     0.32.2
    met_source:              HRES

HRES Single Level#

Note that accumulated parameters (i.e. top_net_thermal_radiation, toa_incident_solar_radiation and other radiation parameters) are accumulated from the start of the forecast

[10]:

hres = HRES(
    time=time,
    variables=["tsr", "ttr"],
    grid=1,
    # url="https://api.ecmwf.int/v1",
    # key="<key>"
    # email="<email>"
)

[12]:

met = hres.open_metdataset()
met

[12]:

MetDataset with data:

<xarray.Dataset>
Dimensions:                    (level: 1, latitude: 181, longitude: 360, time: 3)
Coordinates:
  * level                      (level) float32 -1.0
  * latitude                   (latitude) float64 -90.0 -89.0 ... 89.0 90.0
  * longitude                  (longitude) float64 -180.0 -179.0 ... 178.0 179.0
  * time                       (time) datetime64[ns] 2022-03-26 ... 2022-03-2...
    forecast_time              datetime64[ns] 2022-03-26
Data variables:
    top_net_solar_radiation    (longitude, latitude, level, time) float32 dask.array<chunksize=(360, 181, 1, 2), meta=np.ndarray>
    top_net_thermal_radiation  (longitude, latitude, level, time) float32 dask.array<chunksize=(360, 181, 1, 2), meta=np.ndarray>
Attributes:
    GRIB_edition:            1
    GRIB_centre:             ecmf
    GRIB_centreDescription:  European Centre for Medium-Range Weather Forecasts
    GRIB_subCentre:          0
    Conventions:             CF-1.7
    institution:             European Centre for Medium-Range Weather Forecasts
    history:                 2023-01-11T12:51 GRIB to CDM+CF via cfgrib-0.9.1...
    pycontrails_version:     0.32.2
    met_source:              HRES

Specify forecast by runtime#

Select data from specific forecast run by forecast_time

[11]:

hres = HRES(
    time=("2022-03-26 01:00:00", "2022-03-26 02:00:00"),
    variables=["t", "q"],
    pressure_levels=[300, 250, 200],
    forecast_time="2022-03-25 12:00:00",
    # url="https://api.ecmwf.int/v1",
    # key="<key>"
    # email="<email>"
)
hres

[11]:

HRES
        Timesteps: ['2022-03-26 01', '2022-03-26 02']
        Variables: ['t', 'q']
        Pressure levels: [300, 250, 200]
        Grid: 0.25
        Forecast time: 2022-03-25 12:00:00
        Steps: [13, 14]

IFS#

In development

Integrated Forecasting System from ECMWF

IFS Documentation

Access#

IFS files must be downloaded to a local directory before accessing.

Reference#

IFS Confluence

[12]:

from pycontrails.datalib.ecmwf import IFS

[13]:

ifs = IFS(
    time=("2021-10-02 00:00:00", "2021-10-02 14:00:00"),
    variables=["air_temperature"],
    forecast_path="ifs",
    forecast_date="2021-10-01",
)

ECMWF Variables#

ECMWF_VARIABLES attribute lists the supported parameters from the ECMWF Pameter DB as a list[MetVariable]

[14]:

from pycontrails.datalib.ecmwf import ECMWF_VARIABLES

[15]:

[met_var.standard_name for met_var in ECMWF_VARIABLES]

[15]:

['air_temperature',
 'specific_humidity',
 'geopotential',
 'eastward_wind',
 'northward_wind',
 'lagrangian_tendency_of_air_pressure',
 'relative_humidity',
 'atmosphere_upward_relative_vorticity',
 'fraction_of_cloud_cover',
 'specific_cloud_ice_water_content',
 'specific_cloud_liquid_water_content',
 'potential_vorticity',
 'surface_air_pressure',
 'toa_incident_solar_radiation',
 'top_net_solar_radiation',
 'top_net_thermal_radiation',
 'total_cloud_cover',
 'surface_solar_downward_radiation']

[16]:

from pycontrails.datalib.ecmwf import TopNetSolarRadiation

[17]:

# ECMWF variables contain a link to the param-db entry
TopNetSolarRadiation.ecmwf_link

[17]:

'https://apps.ecmwf.int/codes/grib/param-db?id=178'

Cache Data Files to GCP#

Requires [gcp] optional dependencies:
$ pip install pycontrails[gcp]

By default, data files are cached to the local disk in the users Caches directory.

To cache files to a remote Google Cloud Storage bucket, use the GCPCacheStore

ERA5#

[18]:

from pycontrails import GCPCacheStore

[19]:

variables = ["air_temperature", "relative_humidity"]

gcp = GCPCacheStore(bucket="contrails-301217-unit-test", cache_dir="test/era5", read_only=False)

era5 = ERA5(
    time=(datetime(2019, 1, 1, 0), datetime(2019, 1, 1, 2)),
    variables=variables,
    pressure_levels=[300, 250, 150],
    cachestore=gcp,
    # url="https://cds.climate.copernicus.eu/api/v2",
    # key="<key>"
)

[20]:

# download data to cache - uncomment to run
# met = era5.open_metdataset()