201 lines
No EOL
9.2 KiB
Markdown
201 lines
No EOL
9.2 KiB
Markdown
# Rules and formats for machine enrichment and annotation
|
|
|
|
Machine-driven metadata record enrichment or generation is anticipated to be a common pattern. Known use cases are:
|
|
|
|
Importing bibliographic information from a DOI
|
|
Importing affiliations and output of a person from ORCID
|
|
Importing funding opportunities from externally managed collections
|
|
|
|
This document is a reference manual for our common implementation pattern and approach to organize and provide such implementations.
|
|
|
|
## General principles and their consequences
|
|
|
|
- Enrichment/Machine-generation usually runs repeatedly
|
|
- Existing records should not be duplicated
|
|
- Upstream updates should propagate into enriched/generated records
|
|
- Machine-enrichment and human-enrichment must both be possible
|
|
- Human-created knowledge must be prioritized over machine-created information, i.e., machines are not allowed to overwrite information provided by a human
|
|
- Enrichment of existing records and machine-generation of new records need to go through a human creator
|
|
- Metadata (updates) are submitted into a dumpthings inbox, no machine curates
|
|
- Information can come from different machines (i.e., several different enrichers) and different sources; their provenance should be detailed enough to
|
|
- records need to identify which exact information was generated by which tool and from which information
|
|
- enrichers need to be uniquely identified
|
|
- every enricher needs a pool record pid, through which it can be linked
|
|
- breaking changes / releases in the enricher warrant a new record (such that records can display if they've been created by an older or newer version of an enricher)
|
|
|
|
|
|
## Annotation patterns
|
|
|
|
- For any metadata ``object``, machine-annotations are added via ``annotions`` with two keys, ``pav:importedBy`` and ``pav:importedFrom``, identifying the creating script and information source.
|
|
|
|
```yaml
|
|
# example of a DOI identifier, provided by an enricher script
|
|
"identifiers": [
|
|
{
|
|
"schema_type": "dlthings:DOI",
|
|
"notation": "10.1186/s12916-023-02941-4",
|
|
"annotations": {
|
|
"http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d",
|
|
"http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/1014812"
|
|
}
|
|
}
|
|
],
|
|
```
|
|
|
|
- For any metadata ``data property``, machine-generation is acknowledged via a list of ``attributes``. Each machine-generated information gets one ``attribute`` with
|
|
- the ``predicate`` being a metadta term defining the data property
|
|
- the ``value`` being the machine-generated information about the data property
|
|
- and an ``annotation`` following the pattern above
|
|
|
|
```yaml
|
|
# example of annotation a machine-generated title via attributes
|
|
"attributes": [
|
|
{
|
|
"predicate": "dcterms:title", # however a standard-term would be called
|
|
"value": "Sampling inequalities affect generalization of neuroimaging-based diagnostic classifiers in psychiatry",
|
|
"annotations": {
|
|
"http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d",
|
|
"http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/1014812"}
|
|
}
|
|
]
|
|
```
|
|
|
|
|
|
## Example records
|
|
|
|
### XYZCompetition scraped from funding calendar
|
|
|
|
```yaml
|
|
{
|
|
"description": "Prizes/ Awards Career Status: Postdocs Mobility: Staying Deadline: May 2026 Information Funding Organization: AWK- NRW<https://go.fzj.de/junges-kolleg> Information FZJ: Junges kolleg of NRW<https://www.fz-juelich.de/en/research/research-fields/young-scientists/funding-calendar/prizes-awards/junges_kolleg> (Funding Calendar) FZJ Contact: Dr. Katharina Klein, email: k.klein@fz-juelich.de<mailto:k.klein@fz-juelich.de> ",
|
|
"kind": "xyzrins:competition-types/16e38ddd-6323-4fc7-abad-398609bf8541",
|
|
"schema_type": "xyzri:XYZCompetition",
|
|
"pid": "xyzrins:competition/040000008200E00074C5B7101A82E0080000000002B95B7EC99CDB01000000000000000010000000DF9A3E2F0B1567479D68C54FF7DFB4C1",
|
|
"application_deadline": "2026-05-01",
|
|
"title": "Junges Kolleg of the North Rhine-Westphalian Academy of Sciences, Humanities and the Arts",
|
|
"attributes": [
|
|
{
|
|
"predicate": "dcterms:title",
|
|
"value": "Junges Kolleg of the North Rhine-Westphalian Academy of Sciences, Humanities and the Arts",
|
|
"annotations": {
|
|
"http://purl.org/pav/importedBy": "xyzrins:instruments/54be0232-d05e-4941-bfba-70716cfd6b05",
|
|
"http://purl.org/pav/importedFrom": "https://webmail.fz-juelich.de/owa/calendar/d61ec0ce8d704cb293df97fbb3c8fe23@fz-juelich.de/2f6c8a7e45e5416f89c22258bba6ae0114099503221118509202/calendar.ics"}
|
|
},
|
|
{"predicate": "dcterms:description",
|
|
"value": "",
|
|
"annotations": {
|
|
"http://purl.org/pav/importedBy": "xyzrins:instruments/54be0232-d05e-4941-bfba-70716cfd6b05",
|
|
"http://purl.org/pav/importedFrom": "https://webmail.fz-juelich.de/owa/calendar/d61ec0ce8d704cb293df97fbb3c8fe23@fz-juelich.de/2f6c8a7e45e5416f89c22258bba6ae0114099503221118509202/calendar.ics"}
|
|
},
|
|
{"predicate": "dcterms:application-deadline",
|
|
"value": "2026-05-01",
|
|
"annotations": {
|
|
"http://purl.org/pav/importedBy": "xyzrins:instruments/54be0232-d05e-4941-bfba-70716cfd6b05",
|
|
"http://purl.org/pav/importedFrom": "https://webmail.fz-juelich.de/owa/calendar/d61ec0ce8d704cb293df97fbb3c8fe23@fz-juelich.de/2f6c8a7e45e5416f89c22258bba6ae0114099503221118509202/calendar.ics"}
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
### XYZPublication scraped from Juser
|
|
|
|
```yaml
|
|
{
|
|
"attributed_to": [
|
|
{
|
|
"schema_type": "dlthings:Attribution",
|
|
"object": "xyzrins:persons/simon-eickhoff",
|
|
"annotations": {
|
|
"http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d",
|
|
"http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/1014812"
|
|
}
|
|
}
|
|
],
|
|
"generated_by": [
|
|
{
|
|
"schema_type": "dlthings:Generation",
|
|
"object": "xyzrins:projects/858d0ddb-3c51-4cea-a0e5-a73224b59ae5",
|
|
"annotations": {
|
|
"http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d",
|
|
"http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/1014812"
|
|
}
|
|
},
|
|
{
|
|
"at_location": "ISSN:1741-7015",
|
|
"at_time": "2023",
|
|
"schema_type": "dlthings:Generation",
|
|
"object": "http://purl.obolibrary.org/obo/IAO_0000444",
|
|
"annotations": {
|
|
"http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d",
|
|
"http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/1014812"
|
|
}
|
|
}
|
|
],
|
|
"description": "Abstract blablablabla....",
|
|
"identifiers": [
|
|
{
|
|
"schema_type": "dlthings:DOI",
|
|
"notation": "10.1186/s12916-023-02941-4",
|
|
"annotations": {
|
|
"http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d",
|
|
"http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/1014812"
|
|
}
|
|
}
|
|
],
|
|
"kind": "bibo:AcademicArticle",
|
|
"schema_type": "xyzri:XYZPublication",
|
|
"pid": "xyzrins:publications/fe03c744-3aaa-469e-b789-c4d6198358a0",
|
|
"title": "Sampling inequalities affect generalization of neuroimaging-based diagnostic classifiers in psychiatry",
|
|
"attributes": [
|
|
{
|
|
"predicate": "dcterms:title", # however a standard-term would be called
|
|
"value": "Sampling inequalities affect generalization of neuroimaging-based diagnostic classifiers in psychiatry",
|
|
"annotations": {
|
|
"http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d",
|
|
"http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/1014812"}
|
|
},
|
|
{"predicate": "dcterms:description", # however a standard-term would be called
|
|
"value": "Abstract blablbablablabal...",
|
|
"annotations": {
|
|
"http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d",
|
|
"http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/1014812"}
|
|
},
|
|
{"predicate": "dcterms:kind-of-publication", # however a standard-term would be called
|
|
"value": "bibo:AcademicArticle",
|
|
"annotations": {
|
|
"http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d",
|
|
"http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/1014812"}
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
### XYZPerson scraped from Juser
|
|
|
|
```yaml
|
|
{
|
|
"display_label": "Amunts, J.",
|
|
"identifiers": [
|
|
{
|
|
"annotations": {
|
|
"http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d",
|
|
"http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/https://juser.fz-juelich.de/record/834576"
|
|
},
|
|
"schema_type": "xyzri:ORCID",
|
|
"creator": "ror:04fa4r544",
|
|
"notation": "0000-0001-8575-4124"
|
|
},
|
|
{
|
|
"annotations": {
|
|
"http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d",
|
|
"http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/https://juser.fz-juelich.de/record/834576"
|
|
},
|
|
"schema_type": "dlthings:Identifier",
|
|
"creator": "https://w3id.org/isil/DE-Juel1",
|
|
"notation": "P:(DE-Juel1)172863"
|
|
}
|
|
],
|
|
"schema_type": "xyzri:XYZPerson",
|
|
"pid": "xyzrins:persons/345660c6-c3e2-4fba-bc22-35cacaf13b3d"
|
|
}
|
|
``` |