pycontrails.datalib.ecmwf.hres_model_level

Model-level HRES data access from the ECMWF operational archive.

This module supports

  • Retrieving model-level HRES data by submitting MARS requests through the ECMWF API.

  • Processing retrieved model-level files to produce netCDF files on target pressure levels.

  • Local caching of processed netCDF files.

  • Opening processed and cached files as a pycontrails.MetDataset object.

Classes

HRESModelLevel(time, variables[, ...])

Class to support model-level HRES data access, download, and organization.
class pycontrails.datalib.ecmwf.hres_model_level.HRESModelLevel(time, variables, pressure_levels=None, timestep_freq=None, grid=None, forecast_time=None, model_levels=None, cachestore=<object object>, cache_download=False, url=None, key=None, email=None)

Bases: ECMWFAPI

Class to support model-level HRES data access, download, and organization.

The interface is similar to pycontrails.datalib.ecmwf.HRES, which downloads pressure-level data with much lower vertical resolution and single-level data. Note, however, that only a subset of the pressure-level data available through the operational archive is available as model-level data. As a consequence, this interface only supports access to nominal HRES forecasts (corresponding to stream = "oper" and field_type = "fc" in pycontrails.datalib.ecmwf.HRES) initialized at 00z and 12z.

Requires account with ECMWF and API key.

API credentials can be set in local ~/.ecmwfapirc file:

{
    "url": "https://api.ecmwf.int/v1",
    "email": "<email>",
    "key": "<key>"
}

Credentials can also be provided directly in url, key, and email keyword args.

See ecmwf-api-client documentation for more information.

Parameters:

  • time (metsource.TimeInput) – The time range for data retrieval, either a single datetime or (start, end) datetime range. Input must be datetime-like or tuple of datetime-like (datetime.datetime, pandas.Timestamp, numpy.datetime64) specifying the (start, end) of the date range, inclusive. All times will be downloaded in a single NetCDF file, which ensures that exactly one request is submitted per file on tape accessed. If forecast_time is unspecified, the forecast time will be assumed to be the nearest synoptic hour available in the operational archive (00 or 12). All subsequent times will be downloaded for relative to forecast_time.

  • variables (metsource.VariableInput) – Variable name (i.e. “t”, “air_temperature”, [“air_temperature, specific_humidity”])

  • pressure_levels (metsource.PressureLevelInput, optional) – Pressure levels for data, in hPa (mbar). To download surface-level parameters, use pycontrails.datalib.ecmwf.HRES. Defaults to pressure levels that match model levels at a nominal surface pressure.

  • timestep_freq (str, optional) – Manually set the timestep interval within the bounds defined by time. Supports any string that can be passed to pandas.date_range(freq=...). By default, this is set to the highest frequency that can supported the requested time range (“1h” out to 96 hours, “3h” out to 144 hours, and “6h” out to 240 hours)

  • grid (float, optional) – Specify latitude/longitude grid spacing in data. By default, this is set to 0.1.

  • forecast_time (DatetimeLike, optional) – Specify forecast by initialization time. By default, set to the most recent forecast that includes the requested time range.

  • levels (list[int], optional) – Specify ECMWF model levels to include in MARS requests. By default, this is set to include all model levels.

  • cachestore (CacheStore | None, optional) – Cache data store for staging processed netCDF files. Defaults to pycontrails.core.cache.DiskCacheStore. If None, cache is turned off.

  • cache_download (bool, optional) – If True, cache downloaded NetCDF files rather than storing them in a temporary file. By default, False.

  • url (str) – Override ecmwf-api-client url

  • key (str) – Override ecmwf-api-client key

  • email (str) – Override ecmwf-api-client email

cache_dataset(dataset)

Cache data from data source.

Parameters:

dataset (xarray.Dataset) – Dataset loaded from remote API or local files. The dataset must have the same format as the original data source API or files.

cachestore

Cache store for intermediates while processing data source If None, cache is turned off.

create_cachepath(t)

Return cachepath to local HRES data file based on datetime.

This uniquely defines a cached data file with class parameters.

Parameters:

t (datetime | pd.Timestamp) – Datetime of datafile

Returns:

str – Path to local HRES data file

download(**xr_kwargs)

Confirm all data files are downloaded and available locally in the cachestore.

Parameters:

**xr_kwargs – Passed into xarray.open_dataset() via is_datafile_cached().

download_dataset(times)

Download data from data source for input times.

Parameters:

times (list[datetime]) – List of datetimes to download a store in cache

get_forecast_steps(times)

Convert list of times to list of forecast steps.

Parameters:

times (list[datetime]) – Times to convert to forecast steps

Returns:

list[int] – Forecast step at each time

grid

Lat / Lon grid spacing

property hash

Generate a unique hash for this datasource.

Returns:

str – Unique hash for met instance (sha1)

is_datafile_cached(t, **xr_kwargs)

Check datafile defined by datetime for variables and pressure levels in class.

If using a cloud cache store (i.e. cache.GCPCacheStore), this is where the datafile will be mirrored to a local file for access.

Parameters:

  • t (datetime) – Datetime of datafile

  • **xr_kwargs (Any) – Additional kwargs passed directly to xarray.open_mfdataset() when opening files. By default, the following values are used if not specified:

    • chunks: {“time”: 1}

    • engine: “netcdf4”

    • parallel: False

Returns:

bool – True if data file exists for datetime with all variables and pressure levels, False otherwise

property is_single_level

Return True if the datasource is single level data.

Added in version 0.50.0.

list_timesteps_cached(**xr_kwargs)

Get a list of data files available locally in the cachestore.

Parameters:

**xr_kwargs – Passed into xarray.open_dataset() via is_datafile_cached().

list_timesteps_not_cached(**xr_kwargs)

Get a list of data files not available locally in the cachestore.

Parameters:

**xr_kwargs – Passed into xarray.open_dataset() via is_datafile_cached().

mars_request(times)

Generate MARS request for specific list of times.

Parameters:

times (list[datetime]) – Times included in MARS request.

Returns:

str – MARS request for submission to ECMWF API.

open_dataset(disk_paths, **xr_kwargs)

Open multi-file dataset in xarray.

Parameters:

  • disk_paths (str | list[str] | pathlib.Path | list[pathlib.Path]) – list of string paths to local files to open

  • **xr_kwargs (Any) – Additional kwargs passed directly to xarray.open_mfdataset() when opening files. By default, the following values are used if not specified:

    • chunks: {“time”: 1}

    • engine: “netcdf4”

    • parallel: False

    • lock: False

Returns:

xarray.Dataset – Open xarray dataset

open_metdataset(dataset=None, xr_kwargs=None, **kwargs)

Open MetDataset from data source.

This method should download / load any required datafiles and returns a MetDataset of the multi-file dataset opened by xarray.

Parameters:

  • dataset (xr.Dataset | None, optional) – Input xr.Dataset loaded manually. The dataset must have the same format as the original data source API or files.

  • xr_kwargs (dict[str, Any] | None, optional) – Dictionary of keyword arguments passed into xarray.open_mfdataset() when opening files. Examples include “chunks”, “engine”, “parallel”, etc. Ignored if dataset is input.

  • **kwargs (Any) – Keyword arguments passed through directly into MetDataset constructor.

Returns:

MetDataset – Meteorology dataset

See also

xarray.open_mfdataset()

paths

Path to local source files to load. Set to the paths of files cached in cachestore if no paths input is provided on init.

property pressure_level_variables

ECMWF pressure level parameters available on model levels.

Returns:

list[MetVariable] – List of MetVariable available in datasource

pressure_levels

List of pressure levels. Set to [-1] for data without level coordinate. Use parse_pressure_levels() to handle PressureLevelInput.

set_metadata(ds)

Set met source metadata on ds.attrs.

This is called within the open_metdataset() method to set metadata on the returned MetDataset instance.

Parameters:

ds (xr.Dataset | MetDataset) – Dataset to set metadata on. Mutated in place.

property single_level_variables

ECMWF single-level parameters available on model levels.

Returns:

list[MetVariable] – Always returns an empty list. To access single-level variables, use pycontrails.datalib.ecmwf.HRES.

property step_offset

Difference between forecast_time and first timestep.

Returns:

int – Number of steps to offset in order to retrieve data starting from input time.

property steps

Forecast steps from forecast_time corresponding within input time.

Returns:

list[int] – List of forecast steps relative to forecast_time

property supported_pressure_levels

Pressure levels available from datasource.

Returns:

list[int] | None – List of integer pressure levels for class. If None, no pressure level information available for class.

property supported_variables

Parameters available from data source.

Returns:

list[MetVariable] | None – List of MetVariable available in datasource

timesteps

List of individual timesteps from data source derived from time Use parse_time() to handle TimeInput.

property variable_ecmwfids

Return a list of variable ecmwf_ids.

Returns:

list[int] – List of int ECMWF param ids.

property variable_shortnames

Return a list of variable short names.

Returns:

list[str] – Lst of variable short names.

property variable_standardnames

Return a list of variable standard names.

Returns:

list[str] – Lst of variable standard names.

variables

Variables requested from data source Use parse_variables() to handle VariableInput.