Load ECMWF data#

Requires [ecmwf] optional dependencies:

$ pip install pycontrails[ecmwf]

Support provided for:

ERA5#

Access#

Reference#

ERA5 Pressure Levels#

[1]:
from pycontrails.datalib.ecmwf import ERA5
[2]:
# get a single time
era5 = ERA5(
    time="2022-03-01 00:00:00",
    variables=["t", "q", "u", "v", "w", "ciwc", "z", "cc"],  # supports CF name or short names
    pressure_levels=[300, 250, 200],
    # url="https://cds.climate.copernicus.eu/api/v2",
    # key="<key>"
)
era5
[2]:
ERA5
        Timesteps: ['2022-03-01 00']
        Variables: ['t', 'q', 'u', 'v', 'w', 'ciwc', 'z', 'cc']
        Pressure levels: [300, 250, 200]
        Grid: 0.25
        Dataset: reanalysis-era5-pressure-levels
        Product type: reanalysis
[3]:
# get a range of time
era5 = ERA5(
    time=("2022-03-01 00:00:00", "2022-03-01 03:00:00"),
    variables=[
        "air_temperature",
        "q",
        "u",
        "v",
        "w",
        "ciwc",
        "z",
        "cc",
    ],  # supports CF name or short names
    pressure_levels=[300, 250, 200],
    # url="https://cds.climate.copernicus.eu/api/v2",
    # key="<key>"
)
era5
[3]:
ERA5
        Timesteps: ['2022-03-01 00', '2022-03-01 01', '2022-03-01 02', '2022-03-01 03']
        Variables: ['t', 'q', 'u', 'v', 'w', 'ciwc', 'z', 'cc']
        Pressure levels: [300, 250, 200]
        Grid: 0.25
        Dataset: reanalysis-era5-pressure-levels
        Product type: reanalysis
[4]:
# this triggers a download from CDS if file isn't in cache store
met = era5.open_metdataset()
met
[4]:
MetDataset with data:

<xarray.Dataset>
Dimensions:                              (longitude: 1440, latitude: 721,
                                          level: 3, time: 4)
Coordinates:
  * longitude                            (longitude) float64 -180.0 ... 179.8
  * latitude                             (latitude) float64 -90.0 ... 90.0
  * level                                (level) float64 200.0 250.0 300.0
  * time                                 (time) datetime64[ns] 2022-03-01 ......
    air_pressure                         (level) float32 2e+04 2.5e+04 3e+04
    altitude                             (level) float32 1.178e+04 ... 9.164e+03
Data variables:
    air_temperature                      (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 3, 1), meta=np.ndarray>
    specific_humidity                    (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 3, 1), meta=np.ndarray>
    eastward_wind                        (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 3, 1), meta=np.ndarray>
    northward_wind                       (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 3, 1), meta=np.ndarray>
    lagrangian_tendency_of_air_pressure  (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 3, 1), meta=np.ndarray>
    specific_cloud_ice_water_content     (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 3, 1), meta=np.ndarray>
    geopotential                         (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 3, 1), meta=np.ndarray>
    fraction_of_cloud_cover              (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 3, 1), meta=np.ndarray>
Attributes:
    Conventions:          CF-1.6
    history:              2023-01-11 16:45:07 GMT by grib_to_netcdf-2.25.1: /...
    pycontrails_version:  0.32.2
    provider:             ECMWF
    dataset:              ERA5
    product:              reanalysis

ERA5 Single Level#

[5]:
era5 = ERA5(
    time=("2022-03-01 00:00:00", "2022-03-01 03:00:00"),
    variables=["tsr", "ttr"],
    # url="https://cds.climate.copernicus.eu/api/v2",
    # key="<key>"
)
era5
[5]:
ERA5
        Timesteps: ['2022-03-01 00', '2022-03-01 01', '2022-03-01 02', '2022-03-01 03']
        Variables: ['tsr', 'ttr']
        Pressure levels: [-1]
        Grid: 0.25
        Dataset: reanalysis-era5-single-levels
        Product type: reanalysis
[6]:
met = era5.open_metdataset()
met
[6]:
MetDataset with data:

<xarray.Dataset>
Dimensions:                    (level: 1, time: 4, latitude: 721,
                                longitude: 1440)
Coordinates:
  * level                      (level) float64 -1.0
  * longitude                  (longitude) float64 -180.0 -179.8 ... 179.5 179.8
  * latitude                   (latitude) float64 -90.0 -89.75 ... 89.75 90.0
  * time                       (time) datetime64[ns] 2022-03-01 ... 2022-03-0...
Data variables:
    top_net_solar_radiation    (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 1, 1), meta=np.ndarray>
    top_net_thermal_radiation  (longitude, latitude, level, time) float32 dask.array<chunksize=(1440, 721, 1, 1), meta=np.ndarray>
Attributes:
    Conventions:          CF-1.6
    history:              2023-01-11 16:50:14 GMT by grib_to_netcdf-2.25.1: /...
    pycontrails_version:  0.32.2
    provider:             ECMWF
    dataset:              ERA5
    product:              reanalysis

HRES#

Access#

Users within ECMWF Member and Co-operating States may contact their Computing Representative to obtain access to MARS. All other users may request a username and password and then get an api key.

Provide url, key, and email credentials on input, or see ECMWF API Client documentation to configure local ~/.ecmwfapirc file:

{
    "url": "https://api.ecmwf.int/v1",
    "email": "<email>",
    "key": "<key>"
}

Reference#

  • HRES High resolution forecast

  • ENS Ensemble forecast

HRES Pressure Levels#

[7]:
from datetime import datetime

from pycontrails.datalib.ecmwf import HRES
[8]:
# NOTE / TODO: Including the "ciwc" variable here, the HRES request
# fails with on historic data. However, the request seems to go through
# when the time field is recent (within the last 48 hours?)
time = datetime(2022, 3, 26, 0), datetime(2022, 3, 26, 2)
hres = HRES(
    time=time,
    variables=["t", "q", "u", "v", "w", "z"],
    pressure_levels=[300, 250, 200],
    grid=1,
    # url="https://api.ecmwf.int/v1",
    # key="<key>"
    # email="<email>"
)
hres
[8]:
HRES
        Timesteps: ['2022-03-26 00', '2022-03-26 01', '2022-03-26 02']
        Variables: ['t', 'q', 'u', 'v', 'w', 'z']
        Pressure levels: [300, 250, 200]
        Grid: 1
        Forecast time: 2022-03-26 00:00:00
        Steps: [0, 1, 2]
[9]:
# convience method to see the underlying MARS request
print(hres.generate_mars_request())
retrieve,
        class=od,
        stream=oper,
        expver=1,
        date=20220326,
        time=00,
        type=fc,
        param=t/q/u/v/w/z,
        step=0/1/2,
        grid=1/1,
        levtype=pl,
        levelist=300/250/200
[10]:
# this triggers a download if file isn't in cache store
met = hres.open_metdataset()
met
[10]:
MetDataset with data:

<xarray.Dataset>
Dimensions:                              (longitude: 360, latitude: 181,
                                          level: 3, time: 3)
Coordinates:
    forecast_time                        datetime64[ns] 2022-03-26
  * level                                (level) float64 200.0 250.0 300.0
  * latitude                             (latitude) float64 -90.0 -89.0 ... 90.0
  * longitude                            (longitude) float64 -180.0 ... 179.0
  * time                                 (time) datetime64[ns] 2022-03-26 ......
    air_pressure                         (level) float32 2e+04 2.5e+04 3e+04
    altitude                             (level) float32 1.178e+04 ... 9.164e+03
Data variables:
    air_temperature                      (longitude, latitude, level, time) float32 dask.array<chunksize=(360, 181, 3, 1), meta=np.ndarray>
    specific_humidity                    (longitude, latitude, level, time) float32 dask.array<chunksize=(360, 181, 3, 1), meta=np.ndarray>
    eastward_wind                        (longitude, latitude, level, time) float32 dask.array<chunksize=(360, 181, 3, 1), meta=np.ndarray>
    northward_wind                       (longitude, latitude, level, time) float32 dask.array<chunksize=(360, 181, 3, 1), meta=np.ndarray>
    lagrangian_tendency_of_air_pressure  (longitude, latitude, level, time) float32 dask.array<chunksize=(360, 181, 3, 1), meta=np.ndarray>
    geopotential                         (longitude, latitude, level, time) float32 dask.array<chunksize=(360, 181, 3, 1), meta=np.ndarray>
Attributes:
    GRIB_edition:            1
    GRIB_centre:             ecmf
    GRIB_centreDescription:  European Centre for Medium-Range Weather Forecasts
    GRIB_subCentre:          0
    Conventions:             CF-1.7
    institution:             European Centre for Medium-Range Weather Forecasts
    history:                 2023-01-11T11:54 GRIB to CDM+CF via cfgrib-0.9.1...
    pycontrails_version:     0.32.2
    provider:                ECMWF
    dataset:                 HRES
    product:                 forecast
    radiation_accumulated:   True

HRES Single Level#

Note that accumulated parameters (i.e. top_net_thermal_radiation, toa_incident_solar_radiation and other radiation parameters) are accumulated from the start of the forecast

[11]:
hres = HRES(
    time=time,
    variables=["tsr", "ttr"],
    grid=1,
    # url="https://api.ecmwf.int/v1",
    # key="<key>"
    # email="<email>"
)
[12]:
met = hres.open_metdataset()
met
[12]:
MetDataset with data:

<xarray.Dataset>
Dimensions:                    (level: 1, time: 3, latitude: 181, longitude: 360)
Coordinates:
  * level                      (level) float64 -1.0
    forecast_time              datetime64[ns] 2022-03-26
  * latitude                   (latitude) float64 -90.0 -89.0 ... 89.0 90.0
  * longitude                  (longitude) float64 -180.0 -179.0 ... 178.0 179.0
  * time                       (time) datetime64[ns] 2022-03-26 ... 2022-03-2...
Data variables:
    top_net_solar_radiation    (longitude, latitude, level, time) float32 dask.array<chunksize=(360, 181, 1, 1), meta=np.ndarray>
    top_net_thermal_radiation  (longitude, latitude, level, time) float32 dask.array<chunksize=(360, 181, 1, 1), meta=np.ndarray>
Attributes:
    GRIB_edition:            1
    GRIB_centre:             ecmf
    GRIB_centreDescription:  European Centre for Medium-Range Weather Forecasts
    GRIB_subCentre:          0
    Conventions:             CF-1.7
    institution:             European Centre for Medium-Range Weather Forecasts
    history:                 2023-01-11T12:51 GRIB to CDM+CF via cfgrib-0.9.1...
    pycontrails_version:     0.32.2
    provider:                ECMWF
    dataset:                 HRES
    product:                 forecast
    radiation_accumulated:   True

Specify forecast by runtime#

Select data from specific forecast run by forecast_time

[13]:
hres = HRES(
    time=("2022-03-26 01:00:00", "2022-03-26 02:00:00"),
    variables=["t", "q"],
    pressure_levels=[300, 250, 200],
    forecast_time="2022-03-25 12:00:00",
    # url="https://api.ecmwf.int/v1",
    # key="<key>"
    # email="<email>"
)
hres
[13]:
HRES
        Timesteps: ['2022-03-26 01', '2022-03-26 02']
        Variables: ['t', 'q']
        Pressure levels: [300, 250, 200]
        Grid: 0.25
        Forecast time: 2022-03-25 12:00:00
        Steps: [13, 14]

IFS#

In development

Integrated Forecasting System from ECMWF

Access#

IFS files must be downloaded to a local directory before accessing.

Reference#

[14]:
from pycontrails.datalib.ecmwf import IFS
[15]:
ifs = IFS(
    time=("2021-10-02 00:00:00", "2021-10-02 14:00:00"),
    variables=["air_temperature"],
    forecast_path="ifs",
    forecast_date="2021-10-01",
)

ECMWF Variables#

ECMWF_VARIABLES attribute lists the supported parameters from the ECMWF Pameter DB as a list[MetVariable]

[16]:
from pycontrails.datalib.ecmwf import ECMWF_VARIABLES
[17]:
[met_var.standard_name for met_var in ECMWF_VARIABLES]
[17]:
['air_temperature',
 'specific_humidity',
 'geopotential',
 'eastward_wind',
 'northward_wind',
 'lagrangian_tendency_of_air_pressure',
 'relative_humidity',
 'atmosphere_upward_relative_vorticity',
 'fraction_of_cloud_cover',
 'specific_cloud_ice_water_content',
 'specific_cloud_liquid_water_content',
 'potential_vorticity',
 'surface_air_pressure',
 'toa_incident_solar_radiation',
 'top_net_solar_radiation',
 'top_net_thermal_radiation',
 'total_cloud_cover',
 'surface_solar_downward_radiation']
[18]:
from pycontrails.datalib.ecmwf import TopNetSolarRadiation
[19]:
# ECMWF variables contain a link to the param-db entry
TopNetSolarRadiation.ecmwf_link
[19]:
'https://apps.ecmwf.int/codes/grib/param-db?id=178'

Cache Data Files to GCP#

Requires [gcp] optional dependencies:

$ pip install pycontrails[gcp]

By default, data files are cached to the local disk in the users Caches directory.

To cache files to a remote Google Cloud Storage bucket, use the GCPCacheStore

ERA5#

[20]:
from pycontrails import GCPCacheStore
[21]:
variables = ["air_temperature", "relative_humidity"]

gcp = GCPCacheStore(bucket="contrails-301217-unit-test", cache_dir="test/era5", read_only=False)

era5 = ERA5(
    time=(datetime(2019, 1, 1, 0), datetime(2019, 1, 1, 2)),
    variables=variables,
    pressure_levels=[300, 250, 150],
    cachestore=gcp,
    # url="https://cds.climate.copernicus.eu/api/v2",
    # key="<key>"
)
[22]:
# download data to cache - uncomment to run
# met = era5.open_metdataset()