Add dynamic configuration #206

Merged
cmo merged 64 commits from dynamic-configuration into master 2026-06-12 10:47:49 +00:00

64 commits

Author SHA1 Message Date
Christian Monch
15e2f67631 add test for unicode IRI handling
All checks were successful
Test execution / Test-all (push) Successful in 2m12s
2026-06-12 12:41:54 +02:00
Christian Monch
0b61928067 clean up test code 2026-06-12 12:41:54 +02:00
Christian Monch
0005ee77e1 bump version to 6.0.0 2026-06-12 12:41:54 +02:00
Christian Monch
84561137e1 add submission-tag tests 2026-06-12 12:41:54 +02:00
Christian Monch
4aa902259b addapt dump-things-pid-check to the new structure
Adapt the code of `dump-things-pid-check` to the dynamic
configuration changes.
2026-06-12 12:41:54 +02:00
Christian Monch
7e51656e50 ensure that record-dir config files are created
If a record-dir backend is created on a directory,
check if the config file exists, if not write it.
2026-06-12 12:41:54 +02:00
Christian Monch
5bd4dc1dad fix circular import 2026-06-12 12:41:54 +02:00
Christian Monch
c056ce9dfd remove unused code 2026-06-12 12:41:54 +02:00
Christian Monch
dc88bce026 remove unused code, fix typos in comments 2026-06-12 12:41:54 +02:00
Christian Monch
76b14f7028 remove duplicated code 2026-06-12 12:41:54 +02:00
Christian Monch
29117e1f18 fix a typo in a comment 2026-06-12 12:41:54 +02:00
Christian Monch
edeccc5a03 add compatibility check for git audit directories
Add code to check whether the directory specified
int the git-audit configuration is either non-existing,
empty, or compatible with git-audit.
2026-06-12 12:41:54 +02:00
Christian Monch
9398ac207f remove unnecessary module mapping_functions.py 2026-06-12 12:41:54 +02:00
Christian Monch
728552fd21 replace schema-attribute in pydantic-objects
Replaces the `schema`-attribute with the
attribute `schema_location`. That prevents
shadowing of pydantic's internal `schema`-attribute
and gets rid of pydantic-issues warnings.
2026-06-12 12:41:54 +02:00
Christian Monch
2053997b53 return admin_token in test fixture
The test-fixture `fast_api_simple` now
returns a tuple containing:

- test_client instance
- store path
- admin token
2026-06-12 12:41:54 +02:00
Christian Monch
1747bf987c add enforce_reload-option to abstract config loading 2026-06-12 12:41:54 +02:00
Christian Monch
a753538b4c improve incoming-path verification
Verify that incoming paths exist when a collection
is created a token has write-access to the
collection. Checks that incoming-label is set
for the token.

Add tests for incoming path validation.
2026-06-12 12:41:54 +02:00
Christian Monch
753fe6a9fa handle configuration file structure failures
Do not allow unknown keys in configuration files.
Add a test for structure error detection.
2026-06-12 12:41:54 +02:00
Christian Monch
f231f190fd add test for config file error detection
Test that errors in the format of the configuration
file are caught and converted to `ConfigError`.
2026-06-12 12:41:54 +02:00
Christian Monch
9c92271aec improve configuration processing
Add scanner error detection to configuration loading. Raise
an error if the configuration YAML-file cannot be loaded.

Detect and reject token definitions with write permissions
and without incoming-labels or write access for
collections that do not define an incoming directory.

Rename `get_permissions` to `get_token_permissions`
2026-06-12 12:41:54 +02:00
Christian Monch
1a43b55081 add dump-things-hash-token description to README.md 2026-06-12 12:41:54 +02:00
Christian Monch
f9b65ea0c9 fix wording in README.md 2026-06-12 12:41:54 +02:00
Christian Monch
df8ec7f3ac remove unused code and fix wording 2026-06-12 12:41:54 +02:00
Christian Monch
639091474b add --admin-token-hash description to README.md 2026-06-12 12:41:54 +02:00
Christian Monch
850071076f add -c/--config description to README.md 2026-06-12 12:41:54 +02:00
Christian Monch
072283ec27 update changelog, bump version 2026-06-12 12:41:54 +02:00
Christian Monch
f6a525c8c5 fix backend handling in config format conversion 2026-06-12 12:41:54 +02:00
Christian Monch
1b286d4742 fix default token name setting in config conversion
Use the correct token names, when converting version 1
configuration files to version 2 configuration files.
2026-06-12 12:41:54 +02:00
Christian Monch
171f38bc73 initialize empty store from configuration file 2026-06-12 12:41:54 +02:00
Christian Monch
3675cac64e fix forgejo instance id setting 2026-06-12 12:41:54 +02:00
Christian Monch
b4ca7f555b refactor imports 2026-06-12 12:41:54 +02:00
Christian Monch
295e31e8bb add --admin-token-hash, remove --admin-token
This commit removes the option `--admin-token` and
adds the option `--admin-token-hash`, which accepts
a hashed token (sha256 in 64-digits hex representation).
2026-06-12 12:41:54 +02:00
Christian Monch
a853578482 add MIT license 2026-06-12 12:41:54 +02:00
Christian Monch
907f5ffbb6 bump version to 6.0.0b1 2026-06-12 12:41:54 +02:00
Christian Monch
911820a26d update README.md and CHANGELOG.md 2026-06-12 12:41:54 +02:00
Christian Monch
7b00685da5 fix method names of endpoints 2026-06-12 12:41:54 +02:00
Christian Monch
5bc17d4817 use sha256 for token checksums 2026-06-12 12:41:54 +02:00
Christian Monch
297a4c268e add dump-things-hash-token-command 2026-06-12 12:41:54 +02:00
Christian Monch
0cd1faf45b add collection configuration checks
Check the following two conditions when
new collections are created:

1. the name of the collection is not a
   reserved name.

2. collections do not share incoming or
   curated directories with other collections.
   This is done to prevent data leaks, which
   could happen when different collections
   share directories.
2026-06-12 12:41:54 +02:00
Christian Monch
cee9e11b67 remove unused attributes from instance_state 2026-06-12 12:41:54 +02:00
Christian Mönch
e91162bdd8 improve admin_token handling in config-loader 2026-06-12 12:41:54 +02:00
Christian Mönch
1855d62b9a fix tests
Remove schema-definitions from sqlite-backend configuration.
Use empty dicitionary as default for Configuration.admin_tokens.
2026-06-12 12:41:54 +02:00
Christian Monch
b4220a3421 replace schema-property with schema_location 2026-06-12 12:41:54 +02:00
Christian Monch
950e2618fe add the dump-things-load-config command
Add a command that interprets configuration
files and creates the objects defined in the
configuration file in a service.

This command can be used to migrate from
old dump-things server to new dump-things
server, when the option --old-format is used.

Note that there are a few differences
between the old and the new format:

1. Keys in `tokens` are now token names,
   that map to token configurations.

2. Token configurations now contain the
   additional attributes `representation`
   and `hashed`.

3. `schema` is now an attribute of a
   collection. It is removed from
   sqlite backend configurations.
2026-06-12 12:41:54 +02:00
Christian Monch
c0e100aa15 update comments 2026-06-12 12:41:54 +02:00
Christian Monch
0493499bf2 cleanup code
Update format, remove unused code and debug code
2026-06-12 12:41:54 +02:00
Christian Monch
2ee8782a70 update format 2026-06-12 12:41:54 +02:00
Christian Monch
3c8df4e580 clean up backend code
Format backend code, add __all__ variables.
2026-06-12 12:41:54 +02:00
Christian Monch
b63eb4af9a remove unused code and files 2026-06-12 12:41:54 +02:00
Christian Monch
938b76b281 fix code formatting 2026-06-12 12:41:54 +02:00
Christian Monch
d76a33698b use hashed admin token representations
When adding the plain admin token `t` via
`/admin_tokens`, `AdminTokenRequest.representation`
must contain `hash_token_representation(t)`.

The value of `AdminTokenRequest.representation`
is checked for the correct pattern (a 40-digit hex
number) and stored verbatim in the configuration.

To authenticate an administrator, the token
provided in the `x-dumpthings-token`-header is
hashed and compared to the stored values of
`AdminTokenRequest.representation`.
2026-06-12 12:41:54 +02:00
Christian Monch
e1d51eca8a adapt collection- and token-managing tests
Test that endpoints of deleted collections are
removed from the openapi documentation and
generate 404-errors when accessed.

Adapt token listing test to the modified token
result structure, which does not contain the
token representation anymore.
2026-06-12 12:41:54 +02:00
Christian Monch
355efa4f20 remove endpoints when deleting collections 2026-06-12 12:41:54 +02:00
Christian Monch
feded97346 remove token representation from GET /tokens-response 2026-06-12 12:41:54 +02:00
Christian Monch
9fb7704bd3 hash admin-token representation by default
By default the admin-token representation is hashed.
That means the request should contain the plain token,
unless the request-attribute `hashed` is set. In this
case, the `POST /admin-tokens` endpoint assumes that
the representation is already hashed. It will perform
a simple syntax verification to assure that the
representation is a 40-digit hex number.
2026-06-12 12:41:54 +02:00
Christian Monch
effcbbca2f ensure that api-doc tags are deleted
When a collection is deleted, the collection
specific openapi doc tags are deleted as well.
2026-06-12 12:41:54 +02:00
Christian Monch
fe0781a2ed add tag_info generation
Ensure that generated endpoints are tagged
and sorted correctly in the openapi docs.
2026-06-12 12:41:54 +02:00
Christian Monch
0edba2a2fb add token- and collection deletion 2026-06-12 12:41:11 +02:00
Christian Monch
392dbf6bed add explicit admin-token endpoints
Handle admin tokens seperately from the
traditional token handling
2026-06-12 12:41:11 +02:00
Christian Monch
e93b6b6a48 add audit backend generation and general fixes 2026-06-12 12:39:32 +02:00
Christian Monch
adb100d67e [temp] add authentication source to instance state 2026-06-12 12:39:32 +02:00
Christian Monch
8e7c82dd75 introduce abstract_configuration concept 2026-06-12 12:38:20 +02:00
Christian Monch
0cc51a33a4 add dynamic collection creation 2026-06-12 12:35:46 +02:00
Christian Monch
79aa831668 fix typos in CHANGELOG.md 2026-04-15 08:33:44 +02:00