dump-things-service/docs/configuration.rst

Configuration
=============


General Concepts
----------------

Each Dump Things Service instance maintains a persistent configuration. The
configuration is stored in a private area of the *storage root* and will be
loaded when a Dump Things Service instance is started.


Providing an initial configuration
..................................
If the persistent configuration is empty, for example, because the service is
started on an empty ``<storage root>``, an initial configuration can be
provided by configuration files:

1. via a configuration file in ``<storage root>/.dumpthings.yaml``, or

2. a configuration file can be passed to the command line with the ``-c/--config``
   option.



Performing dynamic configuration changes
........................................

Every Dump Things Service instance (since version 6) supports dynamic
reconfiguration via administration API endpoints. That means, a server can be
started with an empty configuration and the configuration can be modified at
runtime.

If a Dump Things Service instance is started with an empty configuration,
the command line option ``--admin-token-hash``  has to be provided.
This option will establish a "bootstrap" administrator token in the
Dump Things Server instance that can be used to access the
administration endpoints.

Note: the bootstrap administrator token does not become part of the persisted
configuration. That means, it will not be represented in a configuration dump,
i.e., in the result of ``GET /admin_tokens``.



Elements of the configuration
-----------------------------

Dump Things Service instances have three main configuration elements:

- Collections
- Tokens
- Administrator Tokens


A Dump Things Service instance has administration endpoints for all three
configuration elements, which allow to create, modify, and delete
these elements. Any change to the configuration is persisted immediately and
will be effective for all subsequent requests to the service.

All configuration changes are audited (see :doc:`audit` for details). The
audit log is stored in the storage root at the path
``<storage root>/__dumpthings__/config_audit``.



Configuration modification model
--------------------------------

A Dump Things Service instance maintains an internal representation of the
current configuration. The persistent configuration is kept up-to-date with
the internal configuration.

If, for example, a collection is added via the administration
interface, the service will add the collection to its internal configuration
representation. It will then persist the new configuration and updated its
internal state to "activate" the new configuration.

In other words if the administrator interface is used to add a collection, a
token, or an administrator-token, the element will be added to the already
existing configuration elements. If an element is removed via the administration
interface, this element will be removed from the internal configuration, while
all other configuration elements remain unchanged.

To determine the current configuration of the service, the administrator API
provides endpoints to read collection configurations, token configurations, and
admin token configurations.

To modify an existing configuration element, the administrator can fetch its
current state, modify it, and ``PUT`` it back to the service.

The tool ``dump-things-download-config`` (see :doc:`commands`) can be used to
fetch the current configuration of a running Dump Things Service instance and
write it to a configuration file (see :ref:`config_file:`). This file can be
used, for example, to create a Dump Things Service with an identical
configuration at another location.


For a detailed description of the configuration endpoints see :doc:`endpoints`.


.. _config_file:
Configuration File Structure
-----------------------------

A configuration file (version 2) has the following top-level structure:

.. code-block:: yaml

   type: collections   # must be "collections"
   version: 2          # must be 2

   collections:
     <collection-name>:
       ...

   tokens:
     <token-name>:
       ...

    admin_tokens:
      <admin-token-name>:
        ...



Collections
-----------

Each entry under ``<collection-name>`` defines a collection. The following keys
are supported:

``schema`` (required)
   URL of the `LinkML <https://linkml.io>`_ schema for this collection.

``default_token`` (required)
   Name of the token to use when no token is provided by a client. All
   permissions in the default token are OR-ed with any client-provided token.

   Note: a token of the given name must be defined in the ``tokens`` section
   of the configuration file. At least the default token must be created before
   a collection can be created. Although tokens reference collections, tokens
   can always be created because their creation does check whether referenced
   collections exist yet.

``curated`` (required)
   Path to the curated area of the collection. This should be a relative path,
   which will be interpreted relative to the storage root.

``incoming`` (required if tokens with write access exist for the collection)
   Path to the incoming area. Required if the collection has any tokens with
   write access associated to it.
   This should be a relative path, which will be interpreted relative to the
   storage root.

``backend`` (optional)
   Storage backend configuration. If omitted, ``record_dir+stl`` is used.
   See :doc:`backends` for details.

``use_classes`` (optional)
   List of LinkML class names that should receive store/validate endpoints
   (the classes must be subclasses of ``Thing``). All other classes defined
   in the schema are ignored.

``ignore_classes`` (optional)
   List of LinkML class names to exclude from store/validate endpoints
   (the classes must be subclasses of ``Thing``).

``auth_sources`` (optional)
   List of authentication source configurations. See :doc:`authentication`.

``submission_tags`` (optional)
   Tags for an optional automated annotation of submitted records (see
   `Submission Annotation Tags`_ below).

``audit_backends`` (optional)
   List of audit-backend configurations. See `Audit Backends`_ below.


Example
^^^^^^^

.. code-block:: yaml

   collections:
     personal_records:
       default_token: no_access
       schema: https://concepts.datalad.org/s/demo-research-information/unreleased.yaml
       curated: curated
       incoming: incoming
       use_classes:
         - Organization
         - Person
         - Project


Authentication Sources
^^^^^^^^^^^^^^^^^^^^^^

Authentication configuration is described in :doc:`authentication`. If no
authentication sources are defined for a collection, a ``config`` source is
used by default, that means, the service will use the tokens that are
configured in the configuration file.



Submission Annotation Tags
^^^^^^^^^^^^^^^^^^^^^^^^^^

The service might annotate uploaded records with a submitter ID and a
submission time (version below 6 will always annotate submitted records,
version 6 and higher will only annotate submitted records, if the query
parameter ``add_submission_tag`` is set to ``true``.

By default, the following tags are used for submission annotation:

- Submitter ID: ``http://purl.obolibrary.org/obo/NCIT_C54269``
- Submission time: ``http://semanticscience.org/resource/SIO_001083``

Override these defaults per collection with ``submission_tags``:

.. code-block:: yaml

   collections:
     collection_1:
       ...
       submission_tags:
         submitter_id_tag: schema:user_id
         submission_time_tag: schema:time


Audit Backends
^^^^^^^^^^^^^^

The service supports audit logs of changes made via the curation interface.
Configure them per collection with ``audit_backends``:

.. code-block:: yaml

   collections:
     collection_1:
       ...
       audit_backends:
         - type: gitaudit
           path: /path/to/audit-log-dir
           auto_flush_timeout: 60   # seconds (default: 60, minimum: 1)

Currently the only supported audit backend type is ``gitaudit``. A ``gitaudit``
backend stores changes in a bare Git repository at ``path``.
After ``auto_flush_timeout`` seconds without new entries the current changeset
is committed.


Use ``dump-things-gitaudit-report`` to display the audit log and
``dump-things-gitaudit-rebuild-index`` to rebuild its index (see :doc:`commands`).



Collection-and-Token: a Chicken-and-Egg Problem
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In configuration files, collections reference tokens and tokens reference collections.
This can create a chicken-and-egg problem when creating new collection and
new tokens dynamically. This is resolved by allowing tokens to refer to
collections that are not created. In other words:

  Create tokens first, then create collections.





Reserved Collection Names
^^^^^^^^^^^^^^^^^^^^^^^^^

The following collection names are reserved and must **not** be used:

- ``api``
- ``collections``
- ``tokens``
- ``admin_tokens``
- ``__dump_things__``



Tokens
------

Each entry under ``tokens`` defines a token. The following keys are supported:

``representation`` (required)
   The plain-text value the client must supply in the ``X-DumpThings-Token``
   HTTP header. If ``hashed`` is ``True`` this must be the SHA-256 hex digest
   of the plain-text token.

``hashed`` (optional, default: ``False``)
   If ``True``, the representation is treated as a SHA-256 hash.
   Use the command ``dump-things-hash-token`` (see :doc:commands) to compute the correct hash.

``user_id`` (required)
   An identifier that is added as an annotation to each uploaded record.

``collections`` (required)
   Mapping from collection name to per-collection rights.


Per-collection rights
^^^^^^^^^^^^^^^^^^^^^

Each collection entry inside a token has:

``mode``
   Access mode. Available modes (in the following list "own incoming area"
   refers to the incoming area that is associated with the token and the
   collection):

   - ``NOTHING`` – no access.
   - ``READ_CURATED`` – read the curated area only.
   - ``READ_COLLECTION`` – read curated data and own incoming area.
   - ``WRITE_COLLECTION`` – read curated data, read own incoming area, write to own incoming area.
   - ``READ_SUBMISSIONS`` - read own incoming area.
   - ``WRITE_SUBMISSIONS`` – read and write own incoming area.
   - ``SUBMIT`` - read curated area only, write to own incoming area.
   - ``SUBMIT_ONLY`` - write to own incoming area.
   - ``CURATOR`` - curator-level access: read & write to curated area, read & write to all incoming areas.


``incoming_label``
   Label that defines the subdirectory (aka zone) in the incoming area for this token and collection.
   Multiple tokens can share the same label, which means they share a zone.

Example
^^^^^^^

.. code-block:: yaml

   tokens:
     basic_access:
       representation: anonymous
       hashed: False
       user_id: anonymous_user
       collections:
         rooms_and_buildings:
           mode: READ_CURATED
           incoming_label: ''

     contributor_bob:
       representation: bob
       user_id: Bob
       collections:
         rooms_and_buildings:
           mode: WRITE_COLLECTION
           incoming_label: new_rooms_and_buildings


Tips & Tricks
-------------

Same Backend for Incoming and Curated Areas
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

You can configure the service so that incoming records are immediately visible
in the curated area by setting the final incoming path to equal the curated
path:

.. code-block:: yaml

   collections:
     datamgt:
       default_token: anon_read
       curated: datamgt/curated
       incoming: datamgt

   tokens:
     trusted-submitter-token:
       user_id: trusted_submitter
       collections:
         datamgt:
           mode: WRITE_COLLECTION
           incoming_label: curated   # datamgt/curated == curated path


Migrating to Version 6 from Version <=5
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Start the version 6 server on the same storage root as the version 5 server.
The version 6 server will automatically migrate the configuration from
``<storage root>/.dumpthings.yaml`` from the version 5 format to the new
format and persist it. Use the ``-c/--config`` option on first start to
initialise the persisted configuration from a different file than
``<storage root>/.dumpthings.yaml``.

Once started, the configuration is stored in the internal database and the
file ``<storage_root>/.dumpthings.yaml`` or the ``-c`` option can be omitted
on subsequent starts:

.. code-block:: bash

   dump-things-service /path/to/storage -c /path/to/old-config.yaml