Elevate API to cover all operations through the API #233

New issue

Open

opened 2026-06-25 09:08:35 +00:00 by mih · 1 comment

mih commented

2026-06-25 09:08:35 +00:00

Owner

This should include onboarding (loading a deployment with records and config), regular usage (store/retrieve records), and also offboarding (dump records and config, e.g. to migrate to a different deployment).

We talked about this in a meeting. A worthwhile goal is to be able to host many independent deployments for independent user groups on the same system. Having to give people shell access to such a machine is a (likely avoidable) security issue. It should be a goal/requirement for the API to support any operation necessary for an external (admin) user to run and maintain such a deployment -- without having to interact with the machine admins. This should include onboarding (loading a deployment with records and config), regular usage (store/retrieve records), and also offboarding (dump records and config, e.g. to migrate to a different deployment).

cmo commented

2026-07-08 08:53:17 +00:00

Owner

This should include onboarding (loading a deployment with records and config) ...

dump-things-upload-config allows you to configure a DTS instance with new collections, tokens, and admin_tokens via POST or PUT. dump-things-upload-config incrementally adds those elements to an existing configuration. That means, individual configuration files can be uploaded by dump-things-upload-config and the resulting configuration of the service would contain all elements that are defined in the individual config files (see Namespaces below for a discussion of restrictions).
API/dtc supports uploading records to an existing collection.

... regular usage (store/retrieve records), ...

API/dtc supports regular operations

... offboarding (dump records and config, e.g. to migrate to a different deployment)

API/dtc supports dumping all records. This includes records in inboxes (there is currently no single endpoint/command that creates a single dump that can be uploaded by a single endpoint/command).
API/ dump-things-download-config supports dumping the complete configuration and its re-establishment on a different server (see Namespaces below for possible information leaks in the scenario of individual administrators).
API/dtc supports uploading records to curated areas and existing inboxes on a service
API allows establishing a configuration (collections, tokens, admin tokens) on a running service.

Unsupported operations -- Missing features -- Possible pitfalls

Namespaces

All configured collections, tokens, and admin tokens share the same namespaces (there is one namespace for collections, one namespace for tokens, and one namespace for admin tokens). That means configuration files have to be externally synchronised to prevent name collisions. For example, if two configurations used the collection name public, those configurations would collide. The second uploader could not establish the collection because it already exists.

This could be mitigated by running individual service processes in separate storage root-directories. Another possibility to support individual namespaces would be virtual service instances. A virtual service instance could be assigned to an admin token or a set of admin tokens. This would require non-trivial changes to the service's code.

The common namespace poses another challenge. When a configuration is dumped, the complete current configuration state is dumped. That means any administrator can read all information that other administrators configured. This includes collections, tokens, and admin tokens.

Migration of inboxes

The current service supports reading records from inboxes and curated areas. It supports writing of records to existing inboxes and curated areas. It does not support the creation of inboxes by a curator. That means inboxes can not be migrated. Records from inboxes can still be migrated by moving them into the curated area. In other words, only a fully curated collection (all inboxes are empty) can be exactly replicated on a new server. Migrating a collection with non-empty inboxes currently requires a curation process before or during the migration.

This could be mitigated by allowing curators to create inboxes or by adding an initial set of inboxes to a newly created collection. Both require changes to the service's code.

Integrated record migration command

There is currently no single migration command to migrate all records of a collection, i.e., records from the curated areas and inboxes. Such a command would read the configuration, inboxes, and curated records from a collection and establish a collection with an identical state on a different service.

Note the restrictions described in Migration of inboxes, currently the API does not allow to create inboxes

The current dump-things-service (DTS) mostly supports these requirements (since version 6.3.1). An issue might arise with multiple independent administrators on a single service process. In more detail: > This should include onboarding (loading a deployment with records and config) ... - `dump-things-upload-config` allows you to configure a DTS instance with new collections, tokens, and admin_tokens via `POST` or `PUT`. `dump-things-upload-config` incrementally adds those elements to an existing configuration. That means, individual configuration files can be uploaded by `dump-things-upload-config` and the resulting configuration of the service would contain all elements that are defined in the individual config files (see Namespaces below for a discussion of restrictions). - API/`dtc` supports uploading records to an existing collection. > ... regular usage (store/retrieve records), ... - API/`dtc` supports regular operations > ... offboarding (dump records and config, e.g. to migrate to a different deployment) - API/`dtc` supports dumping all records. This includes records in inboxes (there is currently no single endpoint/command that creates a single dump that can be uploaded by a single endpoint/command). - API/ `dump-things-download-config` supports dumping the complete configuration and its re-establishment on a different server (see Namespaces below for possible information leaks in the scenario of individual administrators). - API/`dtc` supports uploading records to curated areas and existing inboxes on a service - API allows establishing a configuration (collections, tokens, admin tokens) on a running service. ## Unsupported operations -- Missing features -- Possible pitfalls #### Namespaces All configured collections, tokens, and admin tokens share the same namespaces (there is one namespace for collections, one namespace for tokens, and one namespace for admin tokens). That means configuration files have to be externally synchronised to prevent name collisions. For example, if two configurations used the collection name `public`, those configurations would collide. The second uploader could not establish the collection because it already exists. This could be mitigated by running individual service processes in separate storage root-directories. Another possibility to support individual namespaces would be virtual service instances. A virtual service instance could be assigned to an admin token or a set of admin tokens. This would require non-trivial changes to the service's code. The common namespace poses another challenge. When a configuration is dumped, the complete current configuration state is dumped. That means any administrator can read all information that other administrators configured. This includes collections, tokens, and admin tokens. #### Migration of inboxes The current service supports reading records from inboxes and curated areas. It supports writing of records to existing inboxes and curated areas. It does not support the creation of inboxes by a curator. That means inboxes can not be migrated. Records from inboxes can still be migrated by moving them into the curated area. In other words, only a fully curated collection (all inboxes are empty) can be exactly replicated on a new server. Migrating a collection with non-empty inboxes currently requires a curation process before or during the migration. This could be mitigated by allowing curators to create inboxes or by adding an initial set of inboxes to a newly created collection. Both require changes to the service's code. #### Integrated record migration command There is currently no single migration command to migrate all records of a collection, i.e., records from the curated areas and inboxes. Such a command would read the configuration, inboxes, and curated records from a collection and establish a collection with an identical state on a different service. Note the restrictions described in __Migration of inboxes__, currently the API does not allow to create inboxes