Resolve broken value encoding for `HexBinary` #226

New issue

Closed

opened 2026-06-24 12:26:03 +00:00 by mih · 1 comment

mih commented

2026-06-24 12:26:03 +00:00

Owner

Using the #222 setup and a thing/v2 schema that was modified like this:

diff --git a/testschema.yaml b/testschema.yaml
index dc7accf..872a790 100644
--- a/testschema.yaml
+++ b/testschema.yaml
@@ -656,3 +656,30 @@ classes:
     is_a: Thing
     title: Annotation tag
     description: A tag identifying an annotation.
+
+  RealThing:
+    is_a: Thing
+    attributes:
+      testme:
+        range: Checksum
+        inlined: true
+        multivalued: false
+
+  Checksum:
+    is_a: Identifier
+    description: >-
+      A Checksum is a value that allows to check the integrity of the contents
+      of a file. Even small changes to the content of the file will change its
+      checksum. This class allows the results of a variety of checksum and
+      cryptographic message digest algorithms to be represented.
+    slot_usage:
+      creator:
+        description: >-
+          Identifies the software agent (algorithm) used to produce the subject
+          `Checksum`.
+        required: true
+      notation:
+        description: >-
+          Lower case hexadecimal encoded checksum digest value.
+        range: HexBinary
+        required: true

I can submit the following JSON record fine

curl -X 'POST' \
  'http://localhost:8000/collection1/record/RealThing?format=json' \
  -H 'accept: application/json' \
  -H 'X-DumpThings-Token: mytoken' \
  -H 'Content-Type: application/json' \
  -d '{
  "pid": "ex:thing1",
  "testme": {
    "creator": "ex:org1",
    "notation": "1234567890abcdef"
  }
}'

It lands as

pid: ex:thing1
testme:
  schema_type: dlthings:Checksum
  creator: ex:org1
  notation: 1234567890abcdef
annotations:
  obo:NCIT_C54269: user1
  sio:SIO_001083: '2026-06-24T14:03:05.282873'

Submitting the equivalent record in TTL format (shown below) corrupts the record:

❯ cat dummy.ttl
@prefix dcterms: <http://purl.org/dc/terms/> .
@prefix dlthings: <https://concepts.datalad.org/s/things/v2/> .
@prefix ex: <http://example.org/> .
@prefix obo: <http://purl.obolibrary.org/obo/> .
@prefix sio: <http://semanticscience.org/resource/> .
@prefix skos: <http://www.w3.org/2004/02/skos/core#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

ex:thing1 a dlthings:RealThing ;
    dlthings:annotations [ a dlthings:Annotation ;
            dlthings:annotation_tag sio:SIO_001083 ;
            dlthings:annotation_value "2026-06-24T13:11:58.610615" ],
        [ a dlthings:Annotation ;
            dlthings:annotation_tag obo:NCIT_C54269 ;
            dlthings:annotation_value "user1" ] ;
    dlthings:testme [ a dlthings:Checksum ;
            dcterms:creator ex:org1 ;
            skos:notation "1234567890abcdef"^^xsd:hexBinary ] .

❯ curl -X 'POST' \
  'http://localhost:8000/collection1/record/RealThing?format=ttl' \
  -H 'X-DumpThings-Token: mytoken' \
  -H 'Content-Type: text/turtle' \
  --data-binary @- < dummy.ttl
@prefix dcterms: <http://purl.org/dc/terms/> .
@prefix dlthings: <https://concepts.datalad.org/s/things/v2/> .
@prefix ex: <http://example.org/> .
@prefix obo: <http://purl.obolibrary.org/obo/> .
@prefix sio: <http://semanticscience.org/resource/> .
@prefix skos: <http://www.w3.org/2004/02/skos/core#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

ex:thing1 a dlthings:RealThing ;
    dlthings:annotations [ a dlthings:Annotation ;
            dlthings:annotation_tag obo:NCIT_C54269 ;
            dlthings:annotation_value "user1" ],
        [ a dlthings:Annotation ;
            dlthings:annotation_tag sio:SIO_001083 ;
            dlthings:annotation_value "2026-06-24T14:05:35.022916" ] ;
    dlthings:testme [ a dlthings:Checksum ;
            dcterms:creator ex:org1 ;
            skos:notation "b'\\x124Vx\\x90\\xab\\xcd\\xef'"^^xsd:hexBinary ] .

in the backend the record lands as

annotations:
  sio:SIO_001083: '2026-06-24T14:05:35.022916'
  obo:NCIT_C54269: user1
pid: ex:thing1
testme:
  schema_type: dlthings:Checksum
  creator: ex:org1
  notation: b'\x124Vx\x90\xab\xcd\xef'

Serverside logs warn about this, but user-facing no error pops up. This is #219

It was suggested elsewhere that the following change could sidestep the issue

diff --git a/testschema.yaml b/testschema.yaml
index dc7accf..b831855 100644
--- a/testschema.yaml
+++ b/testschema.yaml
@@ -126,7 +126,7 @@ types:
 
   HexBinary:
     uri: xsd:hexBinary
-    base: str
+    base: bytes
     typeof: string
     pattern: "^[a-fA-F0-9]+$"
     description: >-

This does NOT appear to be the case. The outcome is unchanged. Neither does removing typeof" string ("string argument without an encoding").

It was also suggested that the type uri setting confuses rdflib. A possible change would be

diff --git a/testschema.yaml b/testschema.yaml
index dc7accf..1d5d730 100644
--- a/testschema.yaml
+++ b/testschema.yaml
@@ -125,7 +125,7 @@ types:
       - The regex is single-quoted for YAML encoding, hence all inner "'" have been doubled
 
   HexBinary:
-    uri: xsd:hexBinary
+    uri: dlthings:HexBinary
     base: str
     typeof: string
     pattern: "^[a-fA-F0-9]+$"

When the TTL document is modified to say

    dlthings:testme [ a dlthings:Checksum ;
            dcterms:creator ex:org1 ;
            skos:notation "1234567s890abcdef"^^dlthings:HexBinary ] .

this does indeed submit, and a non-corrupted record lands in the backend!! Passing an invalid value for the HexBinary also get's caught (unlike the situation described in #219). Sadly, the respective error is not very nice

Conversion error: Conversion Format.ttl -> Format.json. Error: notation must be supplied, target class RealThing, data:
...
ex:thing1 a dlthings:RealThing ;
    dlthings:annotations [ a dlthings:Annotation ;
            dlthings:annotation_tag sio:SIO_001083 ;
            dlthings:annotation_value "2026-06-24T13:11:58.610615" ],
        [ a dlthings:Annotation ;
            dlthings:annotation_tag obo:NCIT_C54269 ;
            dlthings:annotation_value "user1" ] ;
    dlthings:testme [ a dlthings:Checksum ;
            dcterms:creator ex:org1 ;
            skos:notation "1234567s890abcdef"^^dlthings:HexBinary ] .

(Notice the s in the middle of the value). A notation is supplied, but is invalid.

Conclusion:

Switching away from xsd:hexBinary feels wrong. However, it does fix both the encoding, and the inconsequential validation described in #219.

Related to https://hub.psychoinformatics.de/orinoco/shacl-vue/issues/351 and https://hub.psychoinformatics.de/orinoco/dump-things-server/issues/219 Using the #222 setup and a thing/v2 schema that was modified like this: ```diff diff --git a/testschema.yaml b/testschema.yaml index dc7accf..872a790 100644 --- a/testschema.yaml +++ b/testschema.yaml @@ -656,3 +656,30 @@ classes: is_a: Thing title: Annotation tag description: A tag identifying an annotation. + + RealThing: + is_a: Thing + attributes: + testme: + range: Checksum + inlined: true + multivalued: false + + Checksum: + is_a: Identifier + description: >- + A Checksum is a value that allows to check the integrity of the contents + of a file. Even small changes to the content of the file will change its + checksum. This class allows the results of a variety of checksum and + cryptographic message digest algorithms to be represented. + slot_usage: + creator: + description: >- + Identifies the software agent (algorithm) used to produce the subject + `Checksum`. + required: true + notation: + description: >- + Lower case hexadecimal encoded checksum digest value. + range: HexBinary + required: true ``` I can submit the following JSON record fine ```json curl -X 'POST' \ 'http://localhost:8000/collection1/record/RealThing?format=json' \ -H 'accept: application/json' \ -H 'X-DumpThings-Token: mytoken' \ -H 'Content-Type: application/json' \ -d '{ "pid": "ex:thing1", "testme": { "creator": "ex:org1", "notation": "1234567890abcdef" } }' ``` It lands as ```yaml pid: ex:thing1 testme: schema_type: dlthings:Checksum creator: ex:org1 notation: 1234567890abcdef annotations: obo:NCIT_C54269: user1 sio:SIO_001083: '2026-06-24T14:03:05.282873' ``` Submitting the equivalent record in TTL format (shown below) corrupts the record: ``` ❯ cat dummy.ttl @prefix dcterms: <http://purl.org/dc/terms/> . @prefix dlthings: <https://concepts.datalad.org/s/things/v2/> . @prefix ex: <http://example.org/> . @prefix obo: <http://purl.obolibrary.org/obo/> . @prefix sio: <http://semanticscience.org/resource/> . @prefix skos: <http://www.w3.org/2004/02/skos/core#> . @prefix xsd: <http://www.w3.org/2001/XMLSchema#> . ex:thing1 a dlthings:RealThing ; dlthings:annotations [ a dlthings:Annotation ; dlthings:annotation_tag sio:SIO_001083 ; dlthings:annotation_value "2026-06-24T13:11:58.610615" ], [ a dlthings:Annotation ; dlthings:annotation_tag obo:NCIT_C54269 ; dlthings:annotation_value "user1" ] ; dlthings:testme [ a dlthings:Checksum ; dcterms:creator ex:org1 ; skos:notation "1234567890abcdef"^^xsd:hexBinary ] . ``` ``` ❯ curl -X 'POST' \ 'http://localhost:8000/collection1/record/RealThing?format=ttl' \ -H 'X-DumpThings-Token: mytoken' \ -H 'Content-Type: text/turtle' \ --data-binary @- < dummy.ttl @prefix dcterms: <http://purl.org/dc/terms/> . @prefix dlthings: <https://concepts.datalad.org/s/things/v2/> . @prefix ex: <http://example.org/> . @prefix obo: <http://purl.obolibrary.org/obo/> . @prefix sio: <http://semanticscience.org/resource/> . @prefix skos: <http://www.w3.org/2004/02/skos/core#> . @prefix xsd: <http://www.w3.org/2001/XMLSchema#> . ex:thing1 a dlthings:RealThing ; dlthings:annotations [ a dlthings:Annotation ; dlthings:annotation_tag obo:NCIT_C54269 ; dlthings:annotation_value "user1" ], [ a dlthings:Annotation ; dlthings:annotation_tag sio:SIO_001083 ; dlthings:annotation_value "2026-06-24T14:05:35.022916" ] ; dlthings:testme [ a dlthings:Checksum ; dcterms:creator ex:org1 ; skos:notation "b'\\x124Vx\\x90\\xab\\xcd\\xef'"^^xsd:hexBinary ] . ``` in the backend the record lands as ```yaml annotations: sio:SIO_001083: '2026-06-24T14:05:35.022916' obo:NCIT_C54269: user1 pid: ex:thing1 testme: schema_type: dlthings:Checksum creator: ex:org1 notation: b'\x124Vx\x90\xab\xcd\xef' ``` Serverside logs warn about this, but user-facing no error pops up. This is #219 It was suggested elsewhere that the following change could sidestep the issue ```diff diff --git a/testschema.yaml b/testschema.yaml index dc7accf..b831855 100644 --- a/testschema.yaml +++ b/testschema.yaml @@ -126,7 +126,7 @@ types: HexBinary: uri: xsd:hexBinary - base: str + base: bytes typeof: string pattern: "^[a-fA-F0-9]+$" description: >- ``` This does NOT appear to be the case. The outcome is unchanged. Neither does removing `typeof" string` ("string argument without an encoding"). It was also suggested that the type `uri` setting confuses rdflib. A possible change would be ```diff diff --git a/testschema.yaml b/testschema.yaml index dc7accf..1d5d730 100644 --- a/testschema.yaml +++ b/testschema.yaml @@ -125,7 +125,7 @@ types: - The regex is single-quoted for YAML encoding, hence all inner "'" have been doubled HexBinary: - uri: xsd:hexBinary + uri: dlthings:HexBinary base: str typeof: string pattern: "^[a-fA-F0-9]+$" ``` When the TTL document is modified to say ``` dlthings:testme [ a dlthings:Checksum ; dcterms:creator ex:org1 ; skos:notation "1234567s890abcdef"^^dlthings:HexBinary ] . ``` this does indeed submit, and a non-corrupted record lands in the backend!! Passing an invalid value for the HexBinary also get's caught (unlike the situation described in #219). Sadly, the respective error is not very nice ``` Conversion error: Conversion Format.ttl -> Format.json. Error: notation must be supplied, target class RealThing, data: ... ex:thing1 a dlthings:RealThing ; dlthings:annotations [ a dlthings:Annotation ; dlthings:annotation_tag sio:SIO_001083 ; dlthings:annotation_value "2026-06-24T13:11:58.610615" ], [ a dlthings:Annotation ; dlthings:annotation_tag obo:NCIT_C54269 ; dlthings:annotation_value "user1" ] ; dlthings:testme [ a dlthings:Checksum ; dcterms:creator ex:org1 ; skos:notation "1234567s890abcdef"^^dlthings:HexBinary ] . ``` (Notice the `s` in the middle of the value). A notation is supplied, but is invalid. Conclusion: Switching away from `xsd:hexBinary` feels wrong. However, it does fix both the encoding, and the inconsequential validation described in #219.

mih commented

2026-06-25 05:55:19 +00:00

Author

Owner

This is now implemented and published as of https://github.com/psychoinformatics-de/datalad-concepts/pull/552

mih closed this issue

2026-06-25 05:55:19 +00:00

mih referenced this issue from orinoco/shacl-vue

2026-06-25 05:56:27 +00:00

Fix checksum encoding issue. Proposal: do not use explicit hexBinary-type declaration in submitted records #351