xcube Developer Guide

Version 0.2, draft

IMPORTANT NOTE: Any changes to this doc must be reviewed by dev-team through pull requests.

Versioning

We adhere to PEP-440. Therefore, the xcube software version uses the format <major>.<minor>.<micro> for released versions and <major>.<minor>.<micro>.dev<n> for versions in development.

  • <major> is increased for major enhancements. CLI / API changes may introduce incompatibilities with former version.

  • <minor> is increased for new features and and minor enhancements. CLI / API changes are backward compatible with former version.

  • <micro> is increased for bug fixes and micro enhancements. CLI / API changes are backward compatible with former version.

  • <n> is increased whenever the team (internally) deploys new builds of a development snapshot.

The current software version is in xcube/version.py.

Coding Style

We try adhering to PEP-8.

Main Packages

  • xcube.core - Hosts core API functions. Code in here should be maintained w.r.t. backward compatibility. Therefore think twice before adding new or change existing core API.

  • xcube.cli - Hosts CLI commands. CLI command implementations should be lightweight. Move implementation code either into core or util.
    CLI commands must be maintained w.r.t. backward compatibility. Therefore think twice before adding new or change existing CLI commands.

  • xcube.webapi - Hosts Web API functions. Web API command implementations should be lightweight. Move implementation code either into core or util.
    Web API interface must be maintained w.r.t. backward compatibility. Therefore think twice before adding new or change existing web API.

  • xcube.util - Mainly implementation helpers. Comprises classes and functions that are used by cli, core, webapi in order to maximize modularisation and testability but to minimize code duplication.
    The code in here must not be dependent on any of cli, core, webapi. The code in here may change often and in any way as desired by code implementing the cli, core, webapi packages.

The following sections will guide you through extending or changing the main packages that form xcube’s public interface.

Package xcube.cli

Checklist

Make sure your change

  1. is covered by unit-tests (package test/cli);

  2. is reflected by the CLI’s doc-strings and tools documentation (currently in README.md);

  3. follows existing xcube CLI conventions;

  4. follows PEP8 conventions;

  5. is reflected in API and WebAPI, if desired;

  6. is reflected in CHANGES.md.

Hints

Make sure your new CLI command is in line with the others commands regarding command name, option names, as well as metavar arguments names. The CLI command name shall ideally be a verb.

Avoid introducing new option arguments if similar options are already in use for existing commands.

In the following common arguments and options are listed.

Input argument:

@click.argument('input')

If input argument is restricted to an xcube dataset:

@click.argument('cube')

Output (directory) option:

@click.option('--output', '-o', metavar='OUTPUT',
              help='Output directory. If omitted, "INPUT.levels" will be used.')

Output format:

@click.option('--format', '-f', metavar='FORMAT', type=click.Choice(['zarr', 'netcdf']),
              help="Format of the output. If not given, guessed from OUTPUT.")

Output parameters:

@click.option('--param', '-p', metavar='PARAM', multiple=True,
              help="Parameter specific for the output format. Multiple allowed.")

Variable names:

@click.option('--variable',--var', metavar='VARIABLE', multiple=True,
              help="Name of a variable. Multiple allowed.")

For parsing CLI inputs, use helper functions that are already in use. In the CLI command implementation code, raise click.ClickException(message) with a clear message for users.

Common xcube CLI options like -f for FORMAT should be lower case letters and specific xcube CLI options like -S for SIZE in xcube gen are recommended to be uppercase letters.

Extensively validate CLI inputs to avoid that API functions raise ValueError, TypeError, etc. Such errors and their message texts are usually hard to understand by users. They are actually dedicated to to developers, not CLI users.

There is a global option --traceback flag that user can set to dump stack traces. You don’t need to print stack traces from your code.

Package xcube.core

Checklist

Make sure your change

  1. is covered by unit-tests (package test/core);

  2. is covered by API documentation;

  3. follows existing xcube API conventions;

  4. follows PEP8 conventions;

  5. is reflected in xarray extension class xcube.core.xarray.DatasetAccessor;

  6. is reflected in CLI and WebAPI if desired;

  7. is reflected in CHANGES.md.

Hints

Create new module in xcube.core and add your functions. For any functions added make sure naming is in line with other API. Add clear doc-string to the new API. Use Sphinx RST format.

Decide if your API methods requires xcube datasets as inputs, if so, name the primary dataset argument cube and add a keyword parameter cube_asserted: bool = False. Otherwise name the primary dataset argument dataset.

Reflect the fact, that a certain API method or function operates only on datasets that conform with the xcube dataset specifications by using cube in its name rather than dataset. For example compute_dataset can operate on any xarray datasets, while get_cube_values_for_points expects a xcube dataset as input or read_cube ensures it will return valid xcube datasets only.

In the implementation, if not cube_asserted, we must assert and verify the cube is a cube. Pass True to cube_asserted argument of other API called later on:

from xcube.core.verify import assert_cube

def frombosify_cube(cube: xr.Dataset, ..., cube_asserted: bool = False):  
    if not cube_asserted:
        assert_cube(cube)
    ...
    result = bibosify_cube(cube, ..., cube_asserted=True)
    ...

If import xcube.core.xarray is imported in client code, any xarray.Dataset object will have an extra property xcube whose interface is defined by the class xcube.core.xarray.DatasetAccessor. This class is an xarray extension that is used to reflect xcube.core functions and make it directly applicable to the xarray.Dataset object.

Therefore any xcube API shall be reflected in this extension class.

Package xcube.webapi

Checklist

Make sure your change

  1. is covered by unit-tests (package test/webapi);

  2. is covered by Web API specification and documentation (currently in webapi/res/openapi.yml);

  3. follows existing xcube Web API conventions;

  4. follows PEP8 conventions;

  5. is reflected in CLI and API, if desired;

  6. is reflected in CHANGES.md.

Hints

  • The Web API is defined in webapi.app which defines mapping from resource URLs to handlers

  • All handlers are implemented in webapi.handlers. Handler code just delegates to dedicated controllers.

  • All controllers are implemented in webapi.controllers.*. They might further delegate into core.*

Development Process

  1. Make sure there is an issue ticket for your code change work item

  2. Select issue, priorities are as follows

    1. “urgent” and (“important” and “bug”)

    2. “urgent” and (“important” or “bug”)

    3. “urgent”

    4. “important” and “bug”

    5. “important” or “bug”

    6. others

  3. Make sure issue is assigned to you, if unclear agree with team first.

  4. Add issue label “in progress”.

  5. Create development branch named “--” or “<developer>-<issue>-<title>-fix” (see below).</p></li> <li><p>Develop, having in mind the checklists and implementation hints above.</p> <ol class="simple"> <li><p>In your first commit, refer the issue so it will appear as link in the issue history</p></li> <li><p>Develop, test, and push to the remote branch as desired.</p></li> <li><p>In your last commit, utilize checklists above. (You can include the line “closes #<issue>” in your commit message to auto-close the issue once the PR is merged.)</p></li> </ol> </li> <li><p>Create PR if build servers succeed on your branch. If not, fix issue first.<br />For the PR assign the team for review, agree who is to merge. Also reviewers should have checklist in mind.</p></li> <li><p>Merge PR after all reviewers are accepted your change. Otherwise go back.</p></li> <li><p>Remove issue label “in progress”.</p></li> <li><p>Delete the development branch.</p></li> <li><p>If the PR is only partly solving an issue:</p> <ol class="simple"> <li><p>Make sure the issue contains a to-do list (checkboxes) to complete the issue.</p></li> <li><p>Do not include the line “closes #<issue>” in your last commit message.</p></li> <li><p>Add “relates to issue#” in PR.</p></li> <li><p>Make sure to check the corresponding to-do items (checkboxes) <em>after</em> the PR is merged.</p></li> <li><p>Remove issue label “in progress”.</p></li> <li><p>Leave issue open.</p></li> </ol> </li> </ol> </div> <div class="section" id="branches-and-releases"> <h2>Branches and Releases<a class="headerlink" href="#branches-and-releases" title="Permalink to this headline">¶</a></h2> <div class="section" id="target-branches"> <h3>Target Branches<a class="headerlink" href="#target-branches" title="Permalink to this headline">¶</a></h3> <ul class="simple"> <li><p>The <code class="docutils literal notranslate"><span class="pre">master</span></code> branch contains latest developments, including new features and fixes. It is used to generate <code class="docutils literal notranslate"><span class="pre"><major>.<minor>.0</span></code> releases. That is, either <code class="docutils literal notranslate"><span class="pre"><major></span></code> or <code class="docutils literal notranslate"><span class="pre"><minor></span></code> is increased.</p></li> <li><p>The <code class="docutils literal notranslate"><span class="pre"><major>.<minor>.x</span></code> branch is the maintenance branch for a former release tagged <code class="docutils literal notranslate"><span class="pre">v<major>.<minor>.0</span></code>. It is used to generate maintenance <code class="docutils literal notranslate"><span class="pre"><major>.<minor>.<fix></span></code> releases. That is, only <code class="docutils literal notranslate"><span class="pre"><fix></span></code> is increased. Most changes to <code class="docutils literal notranslate"><span class="pre"><major>.<minor>.x</span></code> branch must obviously be merged into <code class="docutils literal notranslate"><span class="pre">master</span></code> branch too.</p></li> </ul> <p>The software version string on all active branches is always <code class="docutils literal notranslate"><span class="pre"><major>.<minor>.<micro>.dev<n></span></code>. Only for a release, we remove the <code class="docutils literal notranslate"><span class="pre">.dev<n></span></code> suffix.</p> </div> <div class="section" id="development-branches"> <h3>Development Branches<a class="headerlink" href="#development-branches" title="Permalink to this headline">¶</a></h3> <p>Development branches that target the <code class="docutils literal notranslate"><span class="pre"><major>.<minor>.x</span></code> branch should indicate that by using the suffix <code class="docutils literal notranslate"><span class="pre">-fix</span></code>, e.g. <code class="docutils literal notranslate"><span class="pre">coolguy-7633-div_by_zero_in_mean-fix</span></code>. After a pull request, the development branch will first be merged into the <code class="docutils literal notranslate"><span class="pre"><major>.<minor>.x</span></code> branch then into <code class="docutils literal notranslate"><span class="pre">master</span></code>.</p> </div> </div> <div class="section" id="release-process"> <h2>Release Process<a class="headerlink" href="#release-process" title="Permalink to this headline">¶</a></h2> <div class="section" id="xcube"> <h3>xcube<a class="headerlink" href="#xcube" title="Permalink to this headline">¶</a></h3> <ul class="simple"> <li><p>Check issues in progress, close any open issues that have been fixed.</p></li> <li><p>In <code class="docutils literal notranslate"><span class="pre">xcube/version.py</span></code> remove the <code class="docutils literal notranslate"><span class="pre">.dev</span></code> suffix from version name.</p></li> <li><p>Make sure <code class="docutils literal notranslate"><span class="pre">CHANGES.md</span></code> is complete. Remove the suffix <code class="docutils literal notranslate"><span class="pre">(in</span> <span class="pre">development)</span></code> from the last version headline.</p></li> <li><p>Push changes to either master or a new maintenance branch (see above).</p></li> <li><p>Await results from Travis CI and ReadTheDocs builds. If broken, fix.</p></li> <li><p>Goto <a class="reference external" href="https://github.com/dcs4cop/xcube/releases">xcube/releases</a> and press button “Draft a new Release”.</p> <ul> <li><p>Tag version is: <code class="docutils literal notranslate"><span class="pre">v${version}</span></code> (with a “v” prefix)</p></li> <li><p>Release title is: <code class="docutils literal notranslate"><span class="pre">${version}</span></code></p></li> <li><p>Paste latest changes from <code class="docutils literal notranslate"><span class="pre">CHANGES.md</span></code> into field “Describe this release”</p></li> <li><p>Press “Publish release” button</p></li> </ul> </li> <li><p>After the release on GitHub, rebase sources, if the branch was <code class="docutils literal notranslate"><span class="pre">master</span></code>, create a new maintenance branch (see above)</p></li> <li><p>In <code class="docutils literal notranslate"><span class="pre">xcube/version.py</span></code> increase version number and append a <code class="docutils literal notranslate"><span class="pre">.dev0</span></code> suffix to version name so that it is still PEP-440 compatible.</p></li> <li><p>In <code class="docutils literal notranslate"><span class="pre">CHANGES.md</span></code> add a new version headline and attach <code class="docutils literal notranslate"><span class="pre">(in</span> <span class="pre">development)</span></code> to it.</p></li> <li><p>Push changes to either master or a new maintenance branch (see above).</p></li> <li><p>Activate new doc version on ReadTheDocs.</p></li> </ul> <p>Go through the same procedure for all xcube plugin packages dependent on this version of xcube.</p> <p>TODO: Describe deployment to xcube conda package after release TODO: Describe deployment of xcube Docker image after release</p> <p>If any changes apply to <code class="docutils literal notranslate"><span class="pre">xcube</span> <span class="pre">serve</span></code> and the xcube Web API:</p> <p>Make sure changes are reflected in <code class="docutils literal notranslate"><span class="pre">xcube/webapi/res/openapi.yml</span></code>. If there are changes, sync <code class="docutils literal notranslate"><span class="pre">xcube/webapi/res/openapi.yml</span></code> with xcube Web API docs on SwaggerHub.</p> <p>Check if changes affect the xcube-viewer code. If so make sure changes are reflected in xcube-viewer code and test viewer with latest xcube Web API. Then release a new xcube viewer.</p> </div> <div class="section" id="xcube-viewer"> <h3>xcube Viewer<a class="headerlink" href="#xcube-viewer" title="Permalink to this headline">¶</a></h3> <ul> <li><p>Cd into viewer project directory (<code class="docutils literal notranslate"><span class="pre">.../xcube-viewer/.</span></code>).</p></li> <li><p>Remove the <code class="docutils literal notranslate"><span class="pre">-dev</span></code> suffix from <code class="docutils literal notranslate"><span class="pre">version</span></code> property in <code class="docutils literal notranslate"><span class="pre">package.json</span></code>.</p></li> <li><p>Remove the <code class="docutils literal notranslate"><span class="pre">-dev</span></code> suffix from <code class="docutils literal notranslate"><span class="pre">version</span></code> constant in <code class="docutils literal notranslate"><span class="pre">src/config.ts</span></code>.</p></li> <li><p>Make sure <code class="docutils literal notranslate"><span class="pre">CHANGES.md</span></code> is complete. Remove the suffix <code class="docutils literal notranslate"><span class="pre">(in</span> <span class="pre">development)</span></code> from the last version headline.</p></li> <li><p>Build the app and test the build using a local http-server, e.g.:</p> <p>$ npm install -g http-server $ cd build $ http-server -p 3000 -c-1</p> </li> <li><p>Push changes to either master or a new maintenance branch (see above).</p></li> <li><p>Goto <a class="reference external" href="https://github.com/dcs4cop/xcube-viewer/releases">xcube-viewer/releases</a> and press button “Draft a new Release”.</p> <ul class="simple"> <li><p>Tag version is: <code class="docutils literal notranslate"><span class="pre">v${version}</span></code> (with a “v” prefix).</p></li> <li><p>Release title is: <code class="docutils literal notranslate"><span class="pre">${version}</span></code>.</p></li> <li><p>Paste latest changes from <code class="docutils literal notranslate"><span class="pre">CHANGES.md</span></code> into field “Describe this release”.</p></li> <li><p>Press “Publish release” button.</p></li> </ul> </li> <li><p>After the release on GitHub, if the branch was <code class="docutils literal notranslate"><span class="pre">master</span></code>, create a new maintenance branch (see above).</p></li> <li><p>Increase <code class="docutils literal notranslate"><span class="pre">version</span></code> property and <code class="docutils literal notranslate"><span class="pre">version</span></code> constant in <code class="docutils literal notranslate"><span class="pre">package.json</span></code> and <code class="docutils literal notranslate"><span class="pre">src/config.ts</span></code> and append <code class="docutils literal notranslate"><span class="pre">-dev.0</span></code> suffix to version name so it is SemVer compatible.</p></li> <li><p>In <code class="docutils literal notranslate"><span class="pre">CHANGES.md</span></code> add a new version headline and attach <code class="docutils literal notranslate"><span class="pre">(in</span> <span class="pre">development)</span></code> to it.</p></li> <li><p>Push changes to either master or a new maintenance branch (see above).</p></li> <li><p>Deploy builds of <code class="docutils literal notranslate"><span class="pre">master</span></code> branches to related web content providers.</p></li> </ul> </div> </div> </div> </div> </div> <footer> <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> <a href="plugins.html" class="btn btn-neutral float-right" title="Plugins" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a> <a href="cubespec.html" class="btn btn-neutral float-left" title="xcube Dataset Specification" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left"></span> Previous</a> </div> <hr/> <div role="contentinfo"> <p> © Copyright 2019, Brockmann Consult GmbH <span class="commit"> Revision <code>87c0c3ae</code>. </span> </p> </div> Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. </footer> </div> </div> </section> </div> <div class="rst-versions" data-toggle="rst-versions" role="note" aria-label="versions"> <span class="rst-current-version" data-toggle="rst-current-version"> <span class="fa fa-book"> Read the Docs</span> v: latest <span class="fa fa-caret-down"></span> </span> <div class="rst-other-versions"> <dl> <dt>Versions</dt> <dd><a href="/en/latest/">latest</a></dd> <dd><a href="/en/stable/">stable</a></dd> <dd><a href="/en/v0.4.2/">v0.4.2</a></dd> <dd><a href="/en/v0.4.0/">v0.4.0</a></dd> <dd><a href="/en/v0.3.0/">v0.3.0</a></dd> <dd><a href="/en/v0.2.1/">v0.2.1</a></dd> <dd><a href="/en/v0.2.0/">v0.2.0</a></dd> </dl> <dl> <dt>Downloads</dt> <dd><a href="//xcube.readthedocs.io/_/downloads/en/latest/pdf/">pdf</a></dd> <dd><a href="//xcube.readthedocs.io/_/downloads/en/latest/htmlzip/">html</a></dd> <dd><a href="//xcube.readthedocs.io/_/downloads/en/latest/epub/">epub</a></dd> </dl> <dl> <dt>On Read the Docs</dt> <dd> <a href="//readthedocs.org/projects/xcube/?fromdocs=xcube">Project Home</a> </dd> <dd> <a href="//readthedocs.org/builds/xcube/?fromdocs=xcube">Builds</a> </dd> </dl> <hr/> Free document hosting provided by <a href="http://www.readthedocs.org">Read the Docs</a>. </div> </div> <script type="text/javascript"> jQuery(function () { SphinxRtdTheme.Navigation.enable(true); }); </script> </body> </html>