Load ECMWF data¶

Requires [ecmwf] optional dependencies:
$ pip install pycontrails[ecmwf]

Support provided for:

ERA5 via the Copernicus Data Store (CDS) using cdsapi or user provided files
HRES and ENS via MARS using ecmwf-api-client or user provided files

ERA5¶

Access¶

Requires account with Copernicus Data Portal
Provide url and key credentials on input, or refer to the CDS API Documentation for how to create ~/.cdsapirc file to configure access.

Reference¶

ERA5 Pressure Levels¶

ERA5 Pressure Level Parameters

[1]:

from pycontrails.datalib.ecmwf import ERA5

[2]:

# get a single time
era5 = ERA5(
    time="2022-03-01 00:00:00",
    variables=["t", "q", "u", "v", "w", "ciwc", "z", "cc"],  # supports CF name or short names
    pressure_levels=[300, 250, 200],
    # url="https://cds.climate.copernicus.eu/api/v2",
    # key="<key>"
)
era5

[2]:

ERA5
        Timesteps: ['2022-03-01 00']
        Variables: ['t', 'q', 'u', 'v', 'w', 'ciwc', 'z', 'cc']
        Pressure levels: [300, 250, 200]
        Grid: 0.25
        Dataset: reanalysis-era5-pressure-levels
        Product type: reanalysis

[3]:

# get a range of time
era5 = ERA5(
    time=("2022-03-01 00:00:00", "2022-03-01 03:00:00"),
    variables=[
        "air_temperature",
        "q",
        "u",
        "v",
        "w",
        "ciwc",
        "z",
        "cc",
    ],  # supports CF name or short names
    pressure_levels=[300, 250, 200],
    # url="https://cds.climate.copernicus.eu/api/v2",
    # key="<key>"
)
era5

[3]:

ERA5
        Timesteps: ['2022-03-01 00', '2022-03-01 01', '2022-03-01 02', '2022-03-01 03']
        Variables: ['t', 'q', 'u', 'v', 'w', 'ciwc', 'z', 'cc']
        Pressure levels: [300, 250, 200]
        Grid: 0.25
        Dataset: reanalysis-era5-pressure-levels
        Product type: reanalysis

[4]:

# this triggers a download from CDS if file isn't in cache store
met = era5.open_metdataset()
met

[4]:

MetDataset with data:

<xarray.Dataset>
Dimensions:                              (longitude: 1440, latitude: 721,
                                          level: 3, time: 4)
Coordinates:
  * longitude                            (longitude) float64 -180.0 ... 179.8
  * latitude                             (latitude) float64 -90.0 ... 90.0
  * level                                (level) float64 200.0 250.0 300.0
  * time                                 (time) datetime64[ns] 2022-03-01 ......
    air_pressure                         (level) float32 2e+04 2.5e+04 3e+04
    altitude                             (level) float32 1.178e+04 ... 9.164e+03
Data variables:
    air_temperature                      (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 3, 1), meta=np.ndarray>
    specific_humidity                    (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 3, 1), meta=np.ndarray>
    eastward_wind                        (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 3, 1), meta=np.ndarray>
    northward_wind                       (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 3, 1), meta=np.ndarray>
    lagrangian_tendency_of_air_pressure  (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 3, 1), meta=np.ndarray>
    specific_cloud_ice_water_content     (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 3, 1), meta=np.ndarray>
    geopotential                         (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 3, 1), meta=np.ndarray>
    fraction_of_cloud_cover              (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 3, 1), meta=np.ndarray>
Attributes:
    Conventions:          CF-1.6
    history:              2023-01-11 16:45:07 GMT by grib_to_netcdf-2.25.1: /...
    pycontrails_version:  0.32.2
    provider:             ECMWF
    dataset:              ERA5
    product:              reanalysis

ERA5 Single Level¶

[5]:

era5 = ERA5(
    time=("2022-03-01 00:00:00", "2022-03-01 03:00:00"),
    variables=["tsr", "ttr"],
    # url="https://cds.climate.copernicus.eu/api/v2",
    # key="<key>"
)
era5

[5]:

ERA5
        Timesteps: ['2022-03-01 00', '2022-03-01 01', '2022-03-01 02', '2022-03-01 03']
        Variables: ['tsr', 'ttr']
        Pressure levels: [-1]
        Grid: 0.25
        Dataset: reanalysis-era5-single-levels
        Product type: reanalysis

[6]:

met = era5.open_metdataset()
met

[6]:

MetDataset with data:

<xarray.Dataset>
Dimensions:                    (level: 1, time: 4, latitude: 721,
                                longitude: 1440)
Coordinates:
  * level                      (level) float64 -1.0
  * longitude                  (longitude) float64 -180.0 -179.8 ... 179.5 179.8
  * latitude                   (latitude) float64 -90.0 -89.75 ... 89.75 90.0
  * time                       (time) datetime64[ns] 2022-03-01 ... 2022-03-0...
Data variables:
    top_net_solar_radiation    (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 1, 1), meta=np.ndarray>
    top_net_thermal_radiation  (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 1, 1), meta=np.ndarray>
Attributes:
    Conventions:          CF-1.6
    history:              2023-01-11 16:50:14 GMT by grib_to_netcdf-2.25.1: /...
    pycontrails_version:  0.32.2
    provider:             ECMWF
    dataset:              ERA5
    product:              reanalysis

HRES¶

Access¶

Users within ECMWF Member and Co-operating States may contact their Computing Representative to obtain access to MARS. All other users may request a username and password and then get an api key.

Provide url, key, and email credentials on input, or see ECMWF API Client documentation to configure local ~/.ecmwfapirc file:

{
    "url": "https://api.ecmwf.int/v1",
    "email": "<email>",
    "key": "<key>"
}

Reference¶

HRES High resolution forecast
ENS Ensemble forecast

HRES Pressure Levels¶

[7]:

from datetime import datetime

from pycontrails.datalib.ecmwf import HRES

[8]:

# NOTE / TODO: Including the "ciwc" variable here, the HRES request
# fails with on historic data. However, the request seems to go through
# when the time field is recent (within the last 48 hours?)
time = datetime(2022, 3, 26, 0), datetime(2022, 3, 26, 2)
hres = HRES(
    time=time,
    variables=["t", "q", "u", "v", "w", "z"],
    pressure_levels=[300, 250, 200],
    grid=1,
    # url="https://api.ecmwf.int/v1",
    # key="<key>"
    # email="<email>"
)
hres

[8]:

HRES
        Timesteps: ['2022-03-26 00', '2022-03-26 01', '2022-03-26 02']
        Variables: ['t', 'q', 'u', 'v', 'w', 'z']
        Pressure levels: [300, 250, 200]
        Grid: 1
        Forecast time: 2022-03-26 00:00:00
        Steps: [0, 1, 2]

[9]:

# convience method to see the underlying MARS request
print(hres.generate_mars_request())

retrieve,
        class=od,
        stream=oper,
        expver=1,
        date=20220326,
        time=00,
        type=fc,
        param=t/q/u/v/w/z,
        step=0/1/2,
        grid=1/1,
        levtype=pl,
        levelist=300/250/200

[10]:

# this triggers a download if file isn't in cache store
met = hres.open_metdataset()
met

[10]:

MetDataset with data:

<xarray.Dataset>
Dimensions:                              (longitude: 360, latitude: 181,
                                          level: 3, time: 3)
Coordinates:
    forecast_time                        datetime64[ns] 2022-03-26
  * level                                (level) float64 200.0 250.0 300.0
  * latitude                             (latitude) float64 -90.0 -89.0 ... 90.0
  * longitude                            (longitude) float64 -180.0 ... 179.0
  * time                                 (time) datetime64[ns] 2022-03-26 ......
    air_pressure                         (level) float32 2e+04 2.5e+04 3e+04
    altitude                             (level) float32 1.178e+04 ... 9.164e+03
Data variables:
    air_temperature                      (longitude, latitude, level, time) float32 dask.array<chunksize=(360, 181, 3, 1), meta=np.ndarray>
    specific_humidity                    (longitude, latitude, level, time) float32 dask.array<chunksize=(360, 181, 3, 1), meta=np.ndarray>
    eastward_wind                        (longitude, latitude, level, time) float32 dask.array<chunksize=(360, 181, 3, 1), meta=np.ndarray>
    northward_wind                       (longitude, latitude, level, time) float32 dask.array<chunksize=(360, 181, 3, 1), meta=np.ndarray>
    lagrangian_tendency_of_air_pressure  (longitude, latitude, level, time) float32 dask.array<chunksize=(360, 181, 3, 1), meta=np.ndarray>
    geopotential                         (longitude, latitude, level, time) float32 dask.array<chunksize=(360, 181, 3, 1), meta=np.ndarray>
Attributes:
    GRIB_edition:            1
    GRIB_centre:             ecmf
    GRIB_centreDescription:  European Centre for Medium-Range Weather Forecasts
    GRIB_subCentre:          0
    Conventions:             CF-1.7
    institution:             European Centre for Medium-Range Weather Forecasts
    history:                 2023-01-11T11:54 GRIB to CDM+CF via cfgrib-0.9.1...
    pycontrails_version:     0.32.2
    provider:                ECMWF
    dataset:                 HRES
    product:                 forecast
    radiation_accumulated:   True

HRES Single Level¶

Note that accumulated parameters (i.e. top_net_thermal_radiation, toa_incident_solar_radiation and other radiation parameters) are accumulated from the start of the forecast

[11]:

hres = HRES(
    time=time,
    variables=["tsr", "ttr"],
    grid=1,
    # url="https://api.ecmwf.int/v1",
    # key="<key>"
    # email="<email>"
)

[12]:

met = hres.open_metdataset()
met

[12]:

MetDataset with data:

<xarray.Dataset>
Dimensions:                    (level: 1, time: 3, latitude: 181, longitude: 360)
Coordinates:
  * level                      (level) float64 -1.0
    forecast_time              datetime64[ns] 2022-03-26
  * latitude                   (latitude) float64 -90.0 -89.0 ... 89.0 90.0
  * longitude                  (longitude) float64 -180.0 -179.0 ... 178.0 179.0
  * time                       (time) datetime64[ns] 2022-03-26 ... 2022-03-2...
Data variables:
    top_net_solar_radiation    (longitude, latitude, level, time) float32 dask.array<chunksize=(360, 181, 1, 1), meta=np.ndarray>
    top_net_thermal_radiation  (longitude, latitude, level, time) float32 dask.array<chunksize=(360, 181, 1, 1), meta=np.ndarray>
Attributes:
    GRIB_edition:            1
    GRIB_centre:             ecmf
    GRIB_centreDescription:  European Centre for Medium-Range Weather Forecasts
    GRIB_subCentre:          0
    Conventions:             CF-1.7
    institution:             European Centre for Medium-Range Weather Forecasts
    history:                 2023-01-11T12:51 GRIB to CDM+CF via cfgrib-0.9.1...
    pycontrails_version:     0.32.2
    provider:                ECMWF
    dataset:                 HRES
    product:                 forecast
    radiation_accumulated:   True

Specify forecast by runtime¶

Select data from specific forecast run by forecast_time

[13]:

hres = HRES(
    time=("2022-03-26 01:00:00", "2022-03-26 02:00:00"),
    variables=["t", "q"],
    pressure_levels=[300, 250, 200],
    forecast_time="2022-03-25 12:00:00",
    # url="https://api.ecmwf.int/v1",
    # key="<key>"
    # email="<email>"
)
hres

[13]:

HRES
        Timesteps: ['2022-03-26 01', '2022-03-26 02']
        Variables: ['t', 'q']
        Pressure levels: [300, 250, 200]
        Grid: 0.25
        Forecast time: 2022-03-25 12:00:00
        Steps: [13, 14]

IFS¶

In development

Integrated Forecasting System from ECMWF

IFS Documentation

Access¶

IFS files must be downloaded to a local directory before accessing.

Reference¶

IFS Confluence

[14]:

from pycontrails.datalib.ecmwf import IFS

[15]:

ifs = IFS(
    time=("2021-10-02 00:00:00", "2021-10-02 14:00:00"),
    variables=["air_temperature"],
    forecast_path="ifs",
    forecast_date="2021-10-01",
)

ECMWF Variables¶

ECMWF_VARIABLES attribute lists the supported parameters from the ECMWF Pameter DB as a list[MetVariable]

[16]:

from pycontrails.datalib.ecmwf import ECMWF_VARIABLES

[17]:

[met_var.standard_name for met_var in ECMWF_VARIABLES]

[17]:

['air_temperature',
 'specific_humidity',
 'geopotential',
 'eastward_wind',
 'northward_wind',
 'lagrangian_tendency_of_air_pressure',
 'relative_humidity',
 'atmosphere_upward_relative_vorticity',
 'fraction_of_cloud_cover',
 'specific_cloud_ice_water_content',
 'specific_cloud_liquid_water_content',
 'potential_vorticity',
 'surface_air_pressure',
 'toa_incident_solar_radiation',
 'top_net_solar_radiation',
 'top_net_thermal_radiation',
 'total_cloud_cover',
 'surface_solar_downward_radiation']

[18]:

from pycontrails.datalib.ecmwf import TopNetSolarRadiation

[19]:

# ECMWF variables contain a link to the param-db entry
TopNetSolarRadiation.ecmwf_link

[19]:

'https://apps.ecmwf.int/codes/grib/param-db?id=178'

Cache Data Files to GCP¶

Requires [gcp] optional dependencies:
$ pip install pycontrails[gcp]

By default, data files are cached to the local disk in the users Caches directory.

To cache files to a remote Google Cloud Storage bucket, use the GCPCacheStore

ERA5¶

[20]:

from pycontrails import GCPCacheStore

[21]:

variables = ["air_temperature", "relative_humidity"]

gcp = GCPCacheStore(bucket="contrails-301217-unit-test", cache_dir="test/era5", read_only=False)

era5 = ERA5(
    time=(datetime(2019, 1, 1, 0), datetime(2019, 1, 1, 2)),
    variables=variables,
    pressure_levels=[300, 250, 150],
    cachestore=gcp,
    # url="https://cds.climate.copernicus.eu/api/v2",
    # key="<key>"
)

[22]:

# download data to cache - uncomment to run
# met = era5.open_metdataset()