orinoco/things-enrichment-tools
Watch
5
0
Fork
You've already forked things-enrichment-tools
 1
Code Issues 3 Pull requests Packages Activity Actions
things-enrichment-tools/docs/machine_annotations.md

9.2 KiB
Raw Permalink Blame History

Rules and formats for machine enrichment and annotation

Machine-driven metadata record enrichment or generation is anticipated to be a common pattern. Known use cases are:

Importing bibliographic information from a DOI
Importing affiliations and output of a person from ORCID
Importing funding opportunities from externally managed collections

This document is a reference manual for our common implementation pattern and approach to organize and provide such implementations.

General principles and their consequences

  • Enrichment/Machine-generation usually runs repeatedly
    • Existing records should not be duplicated
    • Upstream updates should propagate into enriched/generated records
  • Machine-enrichment and human-enrichment must both be possible
    • Human-created knowledge must be prioritized over machine-created information, i.e., machines are not allowed to overwrite information provided by a human
  • Enrichment of existing records and machine-generation of new records need to go through a human creator
    • Metadata (updates) are submitted into a dumpthings inbox, no machine curates
  • Information can come from different machines (i.e., several different enrichers) and different sources; their provenance should be detailed enough to
    • records need to identify which exact information was generated by which tool and from which information
  • enrichers need to be uniquely identified
    • every enricher needs a pool record pid, through which it can be linked
    • breaking changes / releases in the enricher warrant a new record (such that records can display if they've been created by an older or newer version of an enricher)

Annotation patterns

  • For any metadata object, machine-annotations are added via annotions with two keys, pav:importedBy and pav:importedFrom, identifying the creating script and information source.
  # example of a DOI identifier, provided by an enricher script
  "identifiers": [
    {
      "schema_type": "dlthings:DOI",
      "notation": "10.1186/s12916-023-02941-4",
      "annotations": {
        "http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d",
        "http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/1014812"
      } 
    }
  ],
  • For any metadata data property, machine-generation is acknowledged via a list of attributes. Each machine-generated information gets one attribute with
    • the predicate being a metadta term defining the data property
    • the value being the machine-generated information about the data property
    • and an annotation following the pattern above
  # example of annotation a machine-generated title via attributes
  "attributes": [
    {
    "predicate": "dcterms:title",    # however a standard-term would be called
    "value": "Sampling inequalities affect generalization of neuroimaging-based diagnostic classifiers in psychiatry",
    "annotations": {
        "http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d",
        "http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/1014812"}
    }
  ]

Example records

XYZCompetition scraped from funding calendar

{
 "description": "Prizes/ Awards  Career Status: Postdocs  Mobility: Staying  Deadline: May 2026  Information Funding Organization: AWK- NRW<https://go.fzj.de/junges-kolleg>  Information FZJ: Junges kolleg of NRW<https://www.fz-juelich.de/en/research/research-fields/young-scientists/funding-calendar/prizes-awards/junges_kolleg> (Funding Calendar)  FZJ Contact: Dr. Katharina Klein, email: k.klein@fz-juelich.de<mailto:k.klein@fz-juelich.de>  ", 
 "kind": "xyzrins:competition-types/16e38ddd-6323-4fc7-abad-398609bf8541",
 "schema_type": "xyzri:XYZCompetition", 
 "pid": "xyzrins:competition/040000008200E00074C5B7101A82E0080000000002B95B7EC99CDB01000000000000000010000000DF9A3E2F0B1567479D68C54FF7DFB4C1", 
 "application_deadline": "2026-05-01", 
 "title": "Junges Kolleg of the North Rhine-Westphalian Academy of Sciences, Humanities and the Arts", 
   "attributes": [
    {
    "predicate": "dcterms:title", 
    "value": "Junges Kolleg of the North Rhine-Westphalian Academy of Sciences, Humanities and the Arts",
    "annotations": {
        "http://purl.org/pav/importedBy": "xyzrins:instruments/54be0232-d05e-4941-bfba-70716cfd6b05",
        "http://purl.org/pav/importedFrom": "https://webmail.fz-juelich.de/owa/calendar/d61ec0ce8d704cb293df97fbb3c8fe23@fz-juelich.de/2f6c8a7e45e5416f89c22258bba6ae0114099503221118509202/calendar.ics"}
    },
    {"predicate": "dcterms:description",
    "value": "",
    "annotations": {
        "http://purl.org/pav/importedBy": "xyzrins:instruments/54be0232-d05e-4941-bfba-70716cfd6b05",
        "http://purl.org/pav/importedFrom": "https://webmail.fz-juelich.de/owa/calendar/d61ec0ce8d704cb293df97fbb3c8fe23@fz-juelich.de/2f6c8a7e45e5416f89c22258bba6ae0114099503221118509202/calendar.ics"}
    },
    {"predicate": "dcterms:application-deadline",
    "value": "2026-05-01",
    "annotations": {
        "http://purl.org/pav/importedBy": "xyzrins:instruments/54be0232-d05e-4941-bfba-70716cfd6b05",
        "http://purl.org/pav/importedFrom": "https://webmail.fz-juelich.de/owa/calendar/d61ec0ce8d704cb293df97fbb3c8fe23@fz-juelich.de/2f6c8a7e45e5416f89c22258bba6ae0114099503221118509202/calendar.ics"}
    }
    ]    
}

XYZPublication scraped from Juser

{
  "attributed_to": [
    {
      "schema_type": "dlthings:Attribution",
      "object": "xyzrins:persons/simon-eickhoff",
      "annotations": {
        "http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d",
        "http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/1014812"
      }  
    }
  ],
  "generated_by": [
    {
      "schema_type": "dlthings:Generation",
      "object": "xyzrins:projects/858d0ddb-3c51-4cea-a0e5-a73224b59ae5",
      "annotations": {
        "http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d",
        "http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/1014812"
      } 
    },
    {
      "at_location": "ISSN:1741-7015",
      "at_time": "2023",
      "schema_type": "dlthings:Generation",
      "object": "http://purl.obolibrary.org/obo/IAO_0000444",
      "annotations": {
        "http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d",
        "http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/1014812"
      } 
    }
  ],
  "description": "Abstract blablablabla....",
  "identifiers": [
    {
      "schema_type": "dlthings:DOI",
      "notation": "10.1186/s12916-023-02941-4",
      "annotations": {
        "http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d",
        "http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/1014812"
      } 
    }
  ],
  "kind": "bibo:AcademicArticle",
  "schema_type": "xyzri:XYZPublication",
  "pid": "xyzrins:publications/fe03c744-3aaa-469e-b789-c4d6198358a0",
  "title": "Sampling inequalities affect generalization of neuroimaging-based diagnostic classifiers in psychiatry",
  "attributes": [
    {
    "predicate": "dcterms:title",    # however a standard-term would be called
    "value": "Sampling inequalities affect generalization of neuroimaging-based diagnostic classifiers in psychiatry",
    "annotations": {
        "http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d",
        "http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/1014812"}
    },
    {"predicate": "dcterms:description",    # however a standard-term would be called
    "value": "Abstract blablbablablabal...",
    "annotations": {
        "http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d",
        "http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/1014812"}
    },
    {"predicate": "dcterms:kind-of-publication",     # however a standard-term would be called
    "value": "bibo:AcademicArticle",
    "annotations": {
        "http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d",
        "http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/1014812"}
    }
    ]  
}

XYZPerson scraped from Juser

{
  "display_label": "Amunts, J.",
  "identifiers": [
    {
      "annotations": {
        "http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d",
        "http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/https://juser.fz-juelich.de/record/834576"
      },
      "schema_type": "xyzri:ORCID",
      "creator": "ror:04fa4r544",
      "notation": "0000-0001-8575-4124"
    },
    {
      "annotations": {
        "http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d",
        "http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/https://juser.fz-juelich.de/record/834576"
      },
      "schema_type": "dlthings:Identifier",
      "creator": "https://w3id.org/isil/DE-Juel1",
      "notation": "P:(DE-Juel1)172863"
    }
  ],
  "schema_type": "xyzri:XYZPerson",
  "pid": "xyzrins:persons/345660c6-c3e2-4fba-bc22-35cacaf13b3d"
}