`xcube serve`

Synopsis

Serve data cubes via web service.

xcube serve starts a light-weight web server that provides various services based on xcube datasets:

Catalogue services to query for xcube datasets and their variables and dimensions, and feature collections;
Tile map service, with some OGC WMTS 1.0 compatibility (REST and KVP APIs);
Dataset services to extract subsets like time-series and profiles for e.g. JavaScript clients.

$ xcube serve --help

Usage: xcube serve [OPTIONS] [PATHS...]

  Run the xcube Server for the given configuration and/or the given raster
  dataset paths given by PATHS.

  Each of the PATHS arguments can point to a raster dataset such as a Zarr
  directory (*.zarr), an xcube multi-level Zarr dataset (*.levels), a NetCDF
  file (*.nc), or a GeoTIFF/COG file (*.tiff).

  If one of PATHS is a directory that is not a dataset itself, it is scanned
  for readable raster datasets.

  The --show ASSET option can be used to inspect the current configuration of
  the server. ASSET is one of:

  apis            outputs the list of APIs provided by the server
  endpoints       outputs the list of all endpoints provided by the server
  openapi         outputs the OpenAPI document representing this server
  config          outputs the effective server configuration
  configschema    outputs the JSON Schema for the server configuration

  The ASSET may be suffixed by ".yaml" or ".json" forcing the respective
  output format. The default format is YAML.

  Note, if --show  is provided, the ASSET will be shown and the program will
  exit immediately.

Options:
  --framework FRAMEWORK           Web server framework. Defaults to "tornado"
  -p, --port PORT                 Service port number. Defaults to 8080
  -a, --address ADDRESS           Service address. Defaults to "0.0.0.0".
  -c, --config CONFIG             Configuration YAML or JSON file.  If
                                  multiple configuration files are passed, they will be merged in
                                  order.
  --base-dir BASE_DIR             Directory used to resolve relative paths in
                                  CONFIG files. Defaults to the parent
                                  directory of (last) CONFIG file.
  --prefix URL_PREFIX             Prefix path to be used for all endpoint
                                  URLs. May include template variables, e.g.,
                                  "api/{version}".
  --revprefix REVERSE_URL_PREFIX  Prefix path to be used for reverse endpoint
                                  URLs that may be reported by server
                                  responses. May include template variables,
                                  e.g., "/proxy/{port}". Defaults to value of
                                  URL_PREFIX.
  --traceperf                     Whether to output extra performance logs.
  --update-after TIME             Check for server configuration updates every
                                  TIME seconds.
  --stop-after TIME               Unconditionally stop service after TIME
                                  seconds.
  --show ASSET                    Show ASSET and exit. Possible values for
                                  ASSET are 'apis', 'endpoints', 'openapi',
                                  'config', 'configschema' optionally suffixed
                                  by '.yaml' or '.json'.
  --open-viewer                   After starting the server, open xcube Viewer
                                  in a browser tab.
  -q, --quiet                     Disable output of log messages to the
                                  console entirely. Note, this will also
                                  suppress error and warning messages.
  -v, --verbose                   Enable output of log messages to the
                                  console. Has no effect if --quiet/-q is
                                  used. May be given multiple times to control
                                  the level of log messages, i.e., -v refers
                                  to level INFO, -vv to DETAIL, -vvv to DEBUG,
                                  -vvvv to TRACE. If omitted, the log level of
                                  the console is WARNING.
  --help                          Show this message and exit.

Configuration File

The xcube server is used to configure the xcube datasets to be published.

xcube datasets are any datasets that

that comply to Unidata’s CDM and to the CF Conventions;
that can be opened with the xarray Python library;
that have variables that have the dimensions and shape (lat, lon) or (time, lat, lon);
that have 1D-coordinate variables corresponding to the dimensions;
that have their spatial grid defined in arbitrary spatial coordinate reference systems.

The xcube server supports xcube datasets stored as local NetCDF files, as well as Zarr directories in the local file system or remote object storage. Remote Zarr datasets must be stored AWS S3 compatible object storage.

As an example, here is the configuration of the demo server. The parts of the demo configuration file are explained in detail further down.

Some hints before, which are not addressed in the server demo configuration file. To increase imaging performance, xcube datasets can be converted to multi-resolution pyramids using the xcube level tool. In the configuration, the path should use the .levels extension. Leveled xcube datasets are configured this way:

Datasets:

  - Identifier: my_multi_level_dataset
    Title: My Multi-Level Dataset
    FileSystem: file
    Path: my_multi_level_dataset.levels

  - ...

To increase time-series extraction performance, xcube datasets may be rechunked with larger chunk size in the time dimension using the xcube chunk tool. In the xcube server configuration a hidden dataset is given, and the it is referred to by the non-hidden, actual dataset using the TimeSeriesDataset setting:

Datasets:

  - Identifier: my_dataset
    Title: My Dataset
    FileSystem: file
    Path: my_dataset.zarr
    TimeSeriesDataset: my_dataset_opt_for_ts

  - Identifier: my_dataset_opt_for_ts
    Title: My Dataset optimized for Time-Series
    FileSystem: file
    Path: my_ts_opt_dataset.zarr
    Hidden: True

  - ...

Server Demo Configuration File

The server configuration file consists of various parts, some of them are necessary others are optional. Here the demo configuration file used in the example is explained in detail.

The configuration file consists of five main parts authentication, dataset attribution, datasets, place groups and styles.

Authentication [optional]

In order to display data via xcube-viewer exclusively to registered and authorized users, the data served by xcube serve may be protected by adding Authentication to the server configuration. In order to ensure protection, an Authority and an Audience needs to be provided. Here authentication by Auth0 is used. Please note the trailing slash in the “Authority” URL.

Authentication:
  Authority: https://xcube-dev.eu.auth0.com/
  Audience: https://xcube-dev/api/

Example of OIDC configuration for Keycloak. Please note that there is no trailing slash in the “Authority” URL.

Authentication:
  Authority: https://kc.brockmann-consult.de/auth/realms/AVL
  Audience: avl-xc-api

Dataset Attribution [optional]

Dataset Attribution may be added to the server via DatasetAttribution.

DatasetAttribution:
  - "© by Brockmann Consult GmbH 2020, contains modified Copernicus Data 2019, processed by ESA"
  - "© by EU H2020 CyanoAlert project"

Base Directory [optional]

A typical xcube server configuration comprises many paths, and relative paths of known configuration parameters are resolved against the base_dir configuration parameter.

base_dir: s3://<bucket>/<path-to-your>/<resources>/

However, for values of parameters passed to user functions that represent paths in user code, this cannot be done automatically. For such situations, expressions can be used. An expression is any string between "${"` and `"}" in a configuration value. An expression can contain the variables base_dir (a string), ctx the current server context (type xcube.webapi.datasets.DatasetsContext), as well as the function resolve_config_path(path) that is used to make a path absolut with respect to base_dir and to normalize it. For example

Augmentation:
Path: augmentation/metadata.py
Function: metadata:update_metadata
InputParameters:
    bands_config: ${resolve_config_path("../common/bands.yaml")}

Viewer Configuration [optional]

The xcube server endpoint /viewer/config/{*path} allows for configuring the viewer accessible via endpoint /viewer. The actual source for the configuration items is configured by xcube server configuration using the new entry Viewer/Configuration/Path, for example:

Viewer:
  Configuration:
    Path: s3://<bucket>/<viewer-config-dir-path>

Path [mandatory] must be an absolute filesystem path or a S3 path as in the example above. It points to a directory that is expected to contain the the viewer configuration file config.json among other configuration resources, such as custom favicon.ico or logo.png. The file config.json should conform to the [configuration JSON Schema](https://github.com/dcs4cop/xcube-viewer/blob/main/src/resources/config.schema.json). All its values are optional, if not provided, [default values](https://github.com/dcs4cop/xcube-viewer/blob/main/src/resources/config.json) are used instead.

Datasets [mandatory]

In order to publish selected xcube datasets via xcube serve, the datasets need to be described in the server configuration.

Remotely stored xcube Datasets

The following configuration snippet demonstrates how to publish static (persistent) xcube datasets stored in S3-compatible object storage:

Datasets:
  - Identifier: remote
    Title: Remote OLCI L2C cube for region SNS
    BoundingBox: [0.0, 50, 5.0, 52.5]
    FileSystem: s3
    Endpoint: "https://s3.eu-central-1.amazonaws.com"
    Path: xcube-examples/OLCI-SNS-RAW-CUBE-2.zarr
    Region: eu-central-1
    Anonymous: true
    Style: default
    ChunkCacheSize: 250M
    PlaceGroups:
      - PlaceGroupRef: inside-cube
      - PlaceGroupRef: outside-cube
    AccessControl:
      RequiredScopes:
        - read:datasets

The above example of how to specify a xcube dataset to be served above is using a datacube stored in an S3 bucket within the Amazon Cloud. Please have a closer look at the parameter Anonymous: true. This means, the datasets permissions are set to public read in your source s3 bucket. If you have a dataset that is not public-read, set Anonymous: false. Furthermore, you need to have valid credentials on the machine where the server runs. Credentials may be saved either in a file called .aws/credentials with content like below:

[default]

aws_access_key_id=AKIAIOSFODNN7EXAMPLE

aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Or they may be exported as environment variables AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID.

Further down an example for a locally stored xcube datasets will be given, as well as an example of dynamic xcube datasets.

Identifier [mandatory] is a unique ID for each xcube dataset, it is ment for machine-to-machine interaction and therefore does not have to be a fancy human-readable name.

Title [optional] should be understandable for humans. This title that will be displayed within the viewer for the dataset selection. If omitted, the key title from the dataset metadata will be used. If that is missing too, the identifier will be used.

BoundingBox [optional] may be set in order to restrict the region which is served from a certain datacube. The notation of the BoundingBox is [lon_min,lat_min,lon_max,lat_max].

FileSystem [mandatory] is set to “s3” which lets xcube serve know, that the datacube is located in the cloud.

Endpoint [mandatory] contains information about the cloud provider endpoint, this will differ if you use a different cloud provider.

Path [mandatory] leads to the specific location of the datacube. The particular datacube is stored in an OpenTelecomCloud S3 bucket called “xcube-examples” and the datacube is called “OLCI-SNS-RAW-CUBE-2.zarr”.

Region [optional] is the region where the specified cloud provider is operating.

Styles [optional] influence the visualization of the xucbe dataset in the xcube viewer if specified in the server configuration file. The usage of Styles is described in section styles.

PlaceGroups [optional] allow to associate places (e.g. polygons or point-location) with a particular xcube dataset. Several different place groups may be connected to a xcube dataset, these different place groups are distinguished by the PlaceGroupRef. The configuration of PlaceGroups is described in section place groups.

AccessControl [optional] can only be used when providing authentication. Datasets may be protected by configuring the RequiredScopes entry whose value is a list of required scopes, e.g. “read:datasets”.

Variables [optional] enforces the order of variables reported by xcube server. Is a list of wildcard patterns that determines the order of variables and the subset of variables to be reported.

The following example reports only variables whose name starts with “conc_”:

Datasets:
  - Path: demo.zarr
    Variables:
      - "conc_*"

The next example reports all variables but ensures that conc_chl and conc_tsm are the first ones:

Datasets:
  - Path: demo.zarr
    Variables:
      - conc_chl
      - conc_tsm
      - "*"

Locally stored xcube Datasets

The following configuration snippet demonstrates how to publish static (persistent) xcube datasets stored in the local filesystem:

- Identifier: local
  Title: Local OLCI L2C cube for region SNS
  BoundingBox: [0.0, 50, 5.0, 52.5]
  FileSystem: file
  Path: cube-1-250-250.zarr
  Style: default
  TimeSeriesDataset: local_ts
  Augmentation:
    Path: compute_extra_vars.py
    Function: compute_variables
    InputParameters:
      factor_chl: 0.2
      factor_tsm: 0.7
  PlaceGroups:
    - PlaceGroupRef: inside-cube
    - PlaceGroupRef: outside-cube
  AccessControl:
    IsSubstitute: true

Most of the configuration of locally stored datasets is equal to the configuration of remotely stored xcube datasets.

FileSystem [mandatory] is set to “file” which lets xcube serve know, that the datacube is locally stored.

TimeSeriesDataset [optional] is not bound to local datasets, this parameter may be used for remotely stored datasets as well. By using this parameter a time optimized datacube will be used for generating the time series. The configuration of this time optimized datacube is shown below. By adding Hidden with true to the dataset configuration, the time optimized datacube will not appear among the displayed datasets in xcube viewer.

# Will not appear at all, because it is a "hidden" resource
- Identifier: local_ts
  Title: 'local' optimized for time-series
  BoundingBox: [0.0, 50, 5.0, 52.5]
  FileSystem: file
  Path: cube-5-100-200.zarr
  Hidden: true
  Style: default

Augmentation [optional] augments data cubes by new variables computed on-the-fly, the generation of the on-the-fly variables depends on the implementation of the python module specified in the Path within the Augmentation configuration.

AccessControl [optional] can only be used when providing authentication. By passing the IsSubstitute flag a dataset disappears for authorized requests. This might be useful for showing a demo dataset in the viewer for user who are not logged in.

Dynamic xcube Datasets

There is the possibility to define dynamic xcube datasets that are computed on-the-fly. Given here is an example that obtains daily or weekly averages of an xcube dataset named “local”.

- Identifier: local_1w
  Title: OLCI weekly L3 cube for region SNS computed from local L2C cube
  BoundingBox: [0.0, 50, 5.0, 52.5]
  FileSystem: memory
  Path: resample_in_time.py
  Function: compute_dataset
  InputDatasets: ["local"]
  InputParameters:
    period: 1W
    incl_stdev: True
  Style: default
  PlaceGroups:
    - PlaceGroupRef: inside-cube
    - PlaceGroupRef: outside-cube
  AccessControl:
    IsSubstitute: True

FileSystem [mandatory] must be “memory” for dynamically generated datasets.

Path [mandatory] points to a Python module. Can be a Python file, a package, or a Zip file.

Function [mandatory, mutually exclusive with Class] references a function in the Python file given by Path. Must be suffixed by colon-separated module name, if Path references a package or Zip file. The function receives one or more datasets of type xarray.Dataset as defined by InputDatasets and optional keyword-arguments as given by InputParameters, if any. It must return a new xarray.Dataset with same spatial coordinates as the inputs. If “resample_in_time.py” is compressed among any other modules in a zip archive, the original module name must be indicated by the prefix to the function name:

Path: modules.zip
Function: resample_in_time:compute_dataset
InputDatasets: ["local"]

Class [mandatory, mutually exclusive with Function] references a callable in the Python file given by Path. Must be suffixed by colon-separated module name, if Path references a package or Zip file. The callable is either a class derived from xcube.core.mldataset.MultiLevelDataset or a function that returns an instance of xcube.core.mldataset.MultiLevelDataset. The callable receives one or more datasets of type xcube.core.mldataset.MultiLevelDataset as defined by InputDatasets and optional keyword-arguments as given by InputParameters, if any.

InputDatasets [mandatory] specifies the input datasets passed to Function or Class.

InputParameters [mandatory] specifies optional keyword arguments passed to Function or Class. In the example, InputParameters defines which kind of resampling should be performed.

Again, the dataset may be associated with place groups.

Place Groups [optional]

Place groups are specified in a similar manner compared to specifying datasets within a server. Place groups may be stored e.g. in shapefiles or a geoJson.

PlaceGroups:
  - Identifier: outside-cube
    Title: Points outside the cube
    Path: places/outside-cube.geojson
    PropertyMapping:
      image: ${resolve_config_path("images/outside-cube/${ID}.jpg")}

Identifier [mandatory] is a unique ID for each place group, it is the one xcube serve uses to associate a place group to a particular dataset.

Title [mandatory] should be understandable for humans and this is the title that will be displayed within the viewer for the place selection if the selected xcube dataset contains a place group.

Path [mandatory] defines where the file storing the place group is located. Please note that the paths within the example config are relative.

PropertyMapping [mandatory] determines which information contained within the place group should be used for selecting a certain location of the given place group. This depends very strongly of the data used. In the above example, the image URL is determined by a feature’s ID property.

Property Mappings

The entry PropertyMapping is used to map a set of well-known properties (or roles) to the actual properties provided by a place feature in a place group. For example, the well-known properties are used to in xcube viewer to display information about the currently selected place. The possible well-known properties are:

label: The property that provides a label for the place, if any. Defaults to to case-insensitive names label, title, name, id in xcube viewer.
color: The property that provides a place’s color. Defaults to the case-insensitive name color in xcube viewer.
image: The property that provides a place’s image URL, if any. Defaults to case-insensitive names image, img, picture, pic in xcube viewer.
description: The property that provides a place’s description text, if any. Defaults to case-insensitive names description, desc, abstract, comment in xcube viewer.

In the following example, a place’s label is provided by the place feature’s NAME property, while an image is provided by the place feature’s IMG_URL property:

PlaceGroups:
    Identifier: my_group
    ...
    PropertyMapping:
        label: NAME
        image: IMG_URL

The values on the right side may either be feature property names or contain them as placeholders in the form ${PROPERTY}.

Styles [optional]

Within the Styles section, colorbars may be defined which should be used initially for a certain variable of a dataset, as well as the value ranges. For xcube viewer version 0.3.0 or higher the colorbars and the value ranges may be adjusted by the user within the xcube viewer.

Styles:
  - Identifier: default
    ColorMappings:
      conc_chl:
        ColorBar: plasma
        ValueRange: [0., 24.]
      conc_tsm:
        ColorBar: PuBuGn
        ValueRange: [0., 100.]
      kd489:
        ColorBar: jet
        ValueRange: [0., 6.]
      rgb:
        Red:
          Variable: conc_chl
          ValueRange: [0., 24.]
        Green:
          Variable: conc_tsm
          ValueRange: [0., 100.]
        Blue:
          Variable: kd489
          ValueRange: [0., 6.]

The ColorMapping may be specified for each variable of the datasets to be served. If not specified, xcube server will try to extract default values from attributes of dataset variables. The default value ranges are determined by:

xcube-specific variable attributes color_value_min and color_value_max;
The CF variable attributes valid_min, valid_max or valid_range.
Or otherwise, the value range [0, 1] is assumed.

The colorbar name can be set using the

xcube-specific variable attribute color_bar_name;
Otherwise, the default colorbar name will be viridis.

The special name rgb may be used to generate an RGB-image from any other three dataset variables used for the individual Red, Green and Blue channels of the resulting image. An example is shown in the configuration above.

Colormaps may be reversed by using name suffix “_r”. They also can have alpha blending indicated by name suffix “_alpha”. Both, reversed and alpha blending is possible as well and can be configured by name suffix “_r_alpha”.

Styles:
  - Identifier: default
    ColorMappings:
      conc_chl:
        ColorBar: plasma_r_alpha
        ValueRange: [0., 24.]

Colormaps may be user-defined within the configuration file, which can be configured in section customcolormaps. The colormap can be selected by setting ColorBar: my_cmap, where my_cmap is the identifier of the custom defined color map. If the ValueRange is given, it overwrites the value range defined in CustomColorMaps if the color map type is continuous or stepwise, and it is ignored if the color map type is categorical, where a warning is raised.

CustomColorMaps [optional]

In this section, the user can customize colormaps. Xcube server supports three types of colormaps, namely

Continuous: Continuous color assignment, where each <value> represents a support point of a color gradient.
Stepwise: Stepwise color mapping where values within the range of two subsequent <value>s are mapped to the same color. A <color> gets associated with the first <value> of each boundary range, while the last color gets ignored.
Categorical: Values represent unique categories or indexes that are mapped to a color. The data and the <value> must be of type integer. If a category does not have a <value> in the color mapping, it will be displayed as transparent. Suitable for categorical datasets.

which can be configured as shown below:

CustomColorMaps:
  - Identifier: my_cmap
    Type: categorical     # or continuous, stepwise
    Colors:
    - ColorEntry
    - ...

where the ColorEntry may have one of the following forms (in TypeScript notation):

type ColorEntry = ColorEntryObj | ColorEntryTuple;

interface ColorEntryObj {
  Value: number;
  Color: Color;
  Label?: string;
}

type ColorEntryTuple = [number, Color] | [number, Color: label];

type Color = ColorName | ColorHex | ColorTuple;
type ColorName = <A CSS color name>;
type ColorHex = <A CSS color in hexdecimal form>;
type ColorTuple = [number, number, number] | [number, number, number, number]

For example CustomColorMaps can look like this:

All colormaps defined in the CustomColorMaps section will be available in the xcube Viewer, even if they haven’t been selected in the Styles section for a specific data variable.

Example

xcube serve --port 8080 --config ./examples/serve/demo/config.yml --verbose

xcube Server: WMTS, catalogue, data access, tile, feature, time-series services for xarray-enabled data cubes, version 0.2.0
[I 190924 17:08:54 service:228] configuration file 'D:\\Projects\\xcube\\examples\\serve\\demo\\config.yml' successfully loaded
[I 190924 17:08:54 service:158] service running, listening on localhost:8080, try http://localhost:8080/datasets
[I 190924 17:08:54 service:159] press CTRL+C to stop service

Server Demo Configuration File for DataStores

The server configuration file consists of various parts, some of them are necessary, others are optional. Here the demo stores configuration file used in the example stores is explained in detail.

This configuration file differs only in one part compared to section Server Demo Configuration File: data stores. The other main parts (authentication, dataset attribution, place groups, and styles) can be used in combination with data stores.

DataStores [mandatory]

Datasets, which are stored in the same location, may be configured in the configuration file using DataStores.

DataStores:
  - Identifier: edc
    StoreId: s3
    StoreParams:
      root: xcube-dcfs/edc-xc-viewer-data
      max_depth: 1
      storage_options:
        anon: true
        # client_kwargs:
        #  endpoint_url: https://s3.eu-central-1.amazonaws.com
    Datasets:
      - Path: "*2.zarr"
        Style: default
        # ChunkCacheSize: 1G

Identifier [mandatory] is a unique ID for each DataStore.

StoreID [mandatory] can be file for locally stored datasets and s3 for datasets located in the cloud.

StoreParams [mandatory]
root [mandatory] states a common beginning part of the paths of the served datasets.
max_depth [optional] if wildcard is used in Dataset Path this indicated how far the server should step down and serve the discovered datasets.
storage_options [optional] is necessary when serving datasets from the cloud. With anon the accessibility is configured, if the datasets are public-read, anon is set to “true”, “false” indicates they are protected. Credentials may be set by keywords key and secret.

Datasets [optional] if not specified, every dataset in the indicated location supported by xcube will be read and served by xcube serve. In order to filter certain datasets you can list Paths that shall be served by xcube serve. Path may contain wildcards. Each Dataset entry may have Styles and PlaceGroups associated with them, the same way as described in section Server Demo Configuration File.

Example Stores

xcube serve --port 8080 --config ./examples/serve/demo/config-with-stores.yml --verbose

xcube Server: WMTS, catalogue, data access, tile, feature, time-series services for xarray-enabled data cubes, version
[I 190924 17:08:54 service:228] configuration file 'D:\\Projects\\xcube\\examples\\serve\\demo\\config.yml' successfully loaded
[I 190924 17:08:54 service:158] service running, listening on localhost:8080, try http://localhost:8080/datasets
[I 190924 17:08:54 service:159] press CTRL+C to stop service

Example Azure Blob Storage filesystem Stores

xcube server includes support for Azure Blob Storage filesystem by a data store abfs. This enables access to data cubes (.zarr or .levels) in Azure blob storage as shown here:

DataStores:
  - Identifier: siec
    StoreId: abfs
    StoreParams:
      root: my_blob_container
      max_depth: 1
      storage_options:
        anon: true
        account_name: "xxx"
        account_key': "xxx"
        # or
        # connection_string: "xxx"
    Datasets:
      - Path: "*.levels"
        Style: default

Web API

The xcube server has a dedicated self describing Web API Documentation. After starting the server, you can check the various functions provided by xcube Web API. To explore the functions, open <base-url>/openapi.html.

The xcube server implements the OGC WMTS RESTful and KVP architectural styles of the OGC WMTS 1.0.0 specification. The following operations are supported:

GetCapabilities: /xcube/wmts/1.0.0/WMTSCapabilities.xml
GetTile: /xcube/wmts/1.0.0/tile/{DatasetName}/{VarName}/{TileMatrix}/{TileCol}/{TileRow}.png
GetFeatureInfo: in progress

xcube serve