pycontrails.datalib.ukmo.s3¶

Whole-file access layer for the UK Met Office global-deterministic-10km product.

Anonymous access to s3://met-office-atmospheric-model-data (eu-west-2). Requires the additional dependency s3fs.

Module Attributes

`BUCKET`	S3 bucket (anonymous, eu-west-2)
`REGION`	AWS region hosting the bucket
`PRODUCT_PREFIX`	Product prefix within the bucket
`RUN_CADENCE_HOURS`	Met Office global deterministic runs at 00/06/12/18Z
`RUN_HOURS`	Run hours (UTC)
`PARAMETER_VARIABLE`	Mapping from file-name parameter to in-file variable name

Functions

`check_forecast_time`(forecast_time)	Raise if `forecast_time` isn't a time at which forecasts are initialized.
`fetch_pressure_level_field`(fs, key, ...[, ...])	Download the object in full, then fetch the requested pressure levels.
`filesystem`()	Get an anonymous-access s3fs filesystem for `BUCKET`.
`forecast_for_time_range`(first_time, last_time)	Get the forecast with the shortest lead times that covers a time range.
`forecast_timestep_freq`(forecast_time, last_time)	Get the coarsest timestep needed to reach `last_time` in a forecast.
`last_3hourly_step`(forecast_time)	Get the time of the last forecast step with 3-hourly frequency.
`last_hourly_step`(forecast_time)	Get the time of the last forecast step with hourly frequency.
`last_step`(forecast_time)	Get the time of the last step of a forecast.
`latest_forecast`(time)	Get the most recent forecast cycle initialized at or before a time.
`lead_for_valid_time`(forecast_time, valid_time)	Get the lead time, in whole hours, of a valid time in a forecast.
`level_indices_for`(pressure_pa, levels_hpa[, key])	Get the index of each of `levels_hpa` within `pressure_pa`.
`list_run_keys`(fs, run)	List the object keys published for a run.
`object_key`(run, valid_time, lead_hours, ...)	Build the S3 object key for a given run, valid_time, lead and parameter.
`select_pressure_levels`(ds, parameter, levels_hpa)	Select the requested pressure levels, optionally region-cropped, of a parameter.

pycontrails.datalib.ukmo.s3.BUCKET = 'met-office-atmospheric-model-data'¶: S3 bucket (anonymous, eu-west-2)

pycontrails.datalib.ukmo.s3.PARAMETER_VARIABLE = {'relative_humidity_on_pressure_levels': 'relative_humidity', 'temperature_on_pressure_levels': 'air_temperature'}¶: Mapping from file-name parameter to in-file variable name

pycontrails.datalib.ukmo.s3.PRODUCT_PREFIX = 'global-deterministic-10km'¶: Product prefix within the bucket

pycontrails.datalib.ukmo.s3.REGION = 'eu-west-2'¶: AWS region hosting the bucket

pycontrails.datalib.ukmo.s3.RUN_CADENCE_HOURS = 6¶

Met Office global deterministic runs at 00/06/12/18Z

Type:: Run cadence (hours)

pycontrails.datalib.ukmo.s3.RUN_HOURS = (0, 6, 12, 18)¶: Run hours (UTC)

pycontrails.datalib.ukmo.s3.check_forecast_time(forecast_time)¶

Raise if forecast_time isn’t a time at which forecasts are initialized.

Raises:: ValueError – If forecast_time isn’t exactly 00, 06, 12, or 18 UTC.

pycontrails.datalib.ukmo.s3.fetch_pressure_level_field(fs, key, parameter, *, levels_hpa, run=None, valid_time=None, lead_hours=None, extent=None)¶

Download the object in full, then fetch the requested pressure levels.

Downloads the whole object to a local temp file via fs.get, opens it with netcdf4, and loads only the selected subset into memory. The temp file is removed once the dataset has been closed, even if the download or selection fails.

Parameters:

fs (s3fs.S3FileSystem) – Filesystem to read through, e.g. from filesystem().
key (str) – Object key relative to BUCKET, e.g. from object_key().
parameter (str) – Parameter name as it appears in the file name. Must be a key of PARAMETER_VARIABLE.
levels_hpa (Sequence[float]) – Pressure levels to load, in hPa.
run (datetime.datetime, optional) – Filename-derived run time. If given, checked against the in-file forecast_reference_time.
valid_time (datetime.datetime, optional) – Filename-derived valid time. If given, checked against the in-file time.
lead_hours (int, optional) – Filename-derived lead time. If given, checked against the in-file forecast_period.
extent (tuple[float, float, float, float], optional) – (lon_min, lon_max, lat_min, lat_max) to crop to. None (default) returns the full longitude/latitude range.

Returns:

xarray.DataArray – Loaded array with dims (pressure, latitude, longitude), restricted to levels_hpa and, if given, extent.

Raises:

ValueError – If any of run, valid_time, lead_hours doesn’t match the in-file coordinates, or for any of the reasons given in select_pressure_levels().

pycontrails.datalib.ukmo.s3.filesystem()¶

Get an anonymous-access s3fs filesystem for BUCKET.

s3fs caches filesystem instances per (args, kwargs) within a process, so repeated calls with the same arguments may return the same shared instance.

Returns:: s3fs.S3FileSystem – Anonymous, region-pinned S3 filesystem.

pycontrails.datalib.ukmo.s3.forecast_for_time_range(first_time, last_time)¶

Get the forecast with the shortest lead times that covers a time range.

This is the most recent forecast initialized at or before first_time that reaches last_time. It is the most recent forecast for time ranges that are short enough. Forecasts initialized at 00/12Z are longer than those initialized at 06/18Z, so a longer time range may need an older forecast.

Parameters:

first_time (datetime.datetime) – First time requested
last_time (datetime.datetime) – Last time requested

Returns:

datetime.datetime – Forecast initialization time.

Raises:

ValueError – If no single forecast covers the time range.

pycontrails.datalib.ukmo.s3.forecast_timestep_freq(forecast_time, last_time)¶

Get the coarsest timestep needed to reach last_time in a forecast.

The archive publishes a forecast hourly, then every 3 hours, then (00/12Z forecasts only) every 6 hours. Requesting times up to last_time limits timesteps to the coarsest frequency found up to that time.

Parameters:

forecast_time (datetime.datetime) – Forecast initialization time
last_time (datetime.datetime) – Last time requested. Must not be after last_step().

Returns:

str – "1h", "3h", or "6h".

pycontrails.datalib.ukmo.s3.last_3hourly_step(forecast_time)¶

Get the time of the last forecast step with 3-hourly frequency.

Forecasts that end before 6-hourly steps begin (06/18Z) end with their last 3-hourly step.

Parameters:: forecast_time (datetime.datetime) – Forecast initialization time
Returns:: datetime.datetime – Time of the last 3-hourly forecast step.
Raises:: ValueError – If forecast_time isn’t a time at which forecasts are initialized.

pycontrails.datalib.ukmo.s3.last_hourly_step(forecast_time)¶

Get the time of the last forecast step with hourly frequency.

Parameters:: forecast_time (datetime.datetime) – Forecast initialization time
Returns:: datetime.datetime – Time of the last hourly forecast step.
Raises:: ValueError – If forecast_time isn’t a time at which forecasts are initialized.

pycontrails.datalib.ukmo.s3.last_step(forecast_time)¶

Get the time of the last step of a forecast.

Parameters:: forecast_time (datetime.datetime) – Forecast initialization time
Returns:: datetime.datetime – Time of the last forecast step available in the archive.
Raises:: ValueError – If forecast_time isn’t a time at which forecasts are initialized.

pycontrails.datalib.ukmo.s3.latest_forecast(time)¶

Get the most recent forecast cycle initialized at or before a time.

Parameters:: time (datetime.datetime) – Specified time
Returns:: datetime.datetime – Initialization time of the most recent forecast at or before time.

pycontrails.datalib.ukmo.s3.lead_for_valid_time(forecast_time, valid_time)¶

Get the lead time, in whole hours, of a valid time in a forecast.

Parameters:

forecast_time (datetime.datetime) – Forecast initialization time
valid_time (datetime.datetime) – Valid time. Must be a time at which the forecast is published.

Returns:

int – Lead time in whole hours.

Raises:

ValueError – If forecast_time isn’t a time at which forecasts are initialized, or if the archive doesn’t publish the forecast at valid_time: it falls before the forecast, after its end, or between the steps of its 3-hourly or 6-hourly period.

pycontrails.datalib.ukmo.s3.level_indices_for(pressure_pa, levels_hpa, key='<dataset>')¶

Get the index of each of levels_hpa within pressure_pa.

Shared by select_pressure_levels() and _process_hour(), which both need to locate the same set of pressure levels within a fetched field’s native pressure coordinate.

Parameters:

pressure_pa (numpy.ndarray) – In-file pressure coordinate values, in Pa.
levels_hpa (Sequence[float]) – Target pressure levels, in hPa.
key (str, optional) – Identifier used only in the error message.

Returns:

list[int] – Index into pressure_pa of each of levels_hpa, in the same order.

Raises:

ValueError – If any level in levels_hpa isn’t matched by exactly one value in pressure_pa (within a 1e-3 Pa tolerance, since the in-file coordinate is floating point).

pycontrails.datalib.ukmo.s3.list_run_keys(fs, run)¶

List the object keys published for a run.

Costs a few paginated S3 requests per run, however many objects are then looked up in the result. The listing bypasses any directory cache held by fs, so a run still being published is not reported from a stale listing.

Parameters:

fs (s3fs.S3FileSystem) – Filesystem to read through, e.g. from filesystem().
run (datetime.datetime) – Run time.

Returns:

set[str] – Object keys relative to BUCKET, in the form returned by object_key(). Empty if the run has not been published.

pycontrails.datalib.ukmo.s3.object_key(run, valid_time, lead_hours, parameter)¶

Build the S3 object key for a given run, valid_time, lead and parameter.

String template (valid time leads, not run time):

global-deterministic-10km/{RUN}Z/{VALID_TIME}Z-PT{LEAD}H{MIN}M-{parameter}.nc

lead_hours is always zero-padded to 4 digits and minutes are hardcoded to 00M, since this product only ever publishes on-the-hour leads.

Returns:: str – Object key relative to BUCKET.

pycontrails.datalib.ukmo.s3.select_pressure_levels(ds, parameter, levels_hpa, key='<dataset>', *, extent=None)¶

Select the requested pressure levels, optionally region-cropped, of a parameter.

Pure selection logic, factored out so the same subsetting can be applied regardless of how ds was opened; notably used to build ground truth in tests from a locally-cached reference file.

Parameters:

ds (xarray.Dataset) – Opened dataset containing parameter’s variable, plus pressure and latitude/longitude coordinates.
parameter (str) – Parameter name as it appears in the file name. Must be a key of PARAMETER_VARIABLE.
levels_hpa (Sequence[float]) – Pressure levels to select, in hPa.
key (str, optional) – Identifier used only in error messages.
extent (tuple[float, float, float, float], optional) – (lon_min, lon_max, lat_min, lat_max) to crop to. None (default) returns the full longitude/latitude range, cropped only to levels_hpa.

Returns:

xarray.DataArray – Lazy array with dims (pressure, latitude, longitude), restricted to levels_hpa and, if given, extent. Carries um_version and mosg__grid_version in .attrs when present on ds, copied from the file’s global attrs so callers can log per-file provenance without a second fetch.

Raises:

ValueError – If latitude or longitude isn’t in ascending order, if any of levels_hpa isn’t matched by exactly one in-file pressure level, or if extent crops to an empty result.