Machine-annotation schema for calendar and juser

adina commented

2026-07-20 15:34:14 +00:00

Owner

@mih and I had an extensive discussion on how to properly annotate the origin agent and source of enriched/scraped records, initially in the context of the juser scraping script (#6), but soon also on harmonizing approaches for the calendar scraping script and records.

As a result, the hedgedoc with original implementation plans and discussions was amended with an updated idea, up to discussion in the lab chat.

Creating this issue so that the hedge doc doesn't get lost.

TODOs, in case nothing substantially changes:

edit calendar scraping script
fix existing funding records (cave, calendar still broken)
adopt implementation in juser scraping script
document properly

@mih and I had an extensive discussion on how to properly annotate the origin agent and source of enriched/scraped records, initially in the context of the juser scraping script (#6), but soon also on harmonizing approaches for the calendar scraping script and records. As a result, the [hedgedoc](https://hedgedoc.psychoinformatics.de/3cSouq0YSJ6m64_ArWpJEg?both) with original implementation plans and discussions was amended with an updated idea, up to discussion in the lab chat. Creating this issue so that the hedge doc doesn't get lost. TODOs, in case nothing substantially changes: - [x] edit calendar scraping script - [x] fix existing funding records (cave, calendar still broken) - [x] adopt implementation in juser scraping script - [ ] document properly

adina commented

2026-07-24 12:45:50 +00:00

Author

Owner

The discussion has - for now - reached a conclusion, recorded in the hedgedoc and the current best practice example repeated below:

{
  "attributed_to": [
    {
      "schema_type": "dlthings:Attribution",
      "object": "xyzrins:persons/simon-eickhoff",
      "annotations": {
        "http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d",
        "http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/1014812"
      }  
    }
  ],
  "generated_by": [
    {
      "schema_type": "dlthings:Generation",
      "object": "xyzrins:projects/858d0ddb-3c51-4cea-a0e5-a73224b59ae5",
      "annotations": {
        "http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d",
        "http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/1014812"
      } 
    },
    {
      "at_location": "ISSN:1741-7015",
      "at_time": "2023",
      "schema_type": "dlthings:Generation",
      "object": "http://purl.obolibrary.org/obo/IAO_0000444",
      "annotations": {
        "http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d",
        "http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/1014812"
      } 
    }
  ],
  "description": "Abstract blablablabla....",
  "identifiers": [
    {
      "schema_type": "dlthings:DOI",
      "notation": "10.1186/s12916-023-02941-4",
      "annotations": {
        "http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d",
        "http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/1014812"
      } 
    }
  ],
  "kind": "bibo:AcademicArticle",
  "schema_type": "xyzri:XYZPublication",
  "pid": "xyzrins:publications/fe03c744-3aaa-469e-b789-c4d6198358a0",
  "title": "Sampling inequalities affect generalization of neuroimaging-based diagnostic classifiers in psychiatry",
  "attributes": [
    {
    "predicate": "dcterms:title",    # however a standard-term would be called
    "value": "Sampling inequalities affect generalization of neuroimaging-based diagnostic classifiers in psychiatry",
    "annotations": {
        "obo:NCIT_C42704": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d",
        "obo:NCIT_P378": "https://juser.fz-juelich.de/record/1014812"}
    },
    {"predicate": "dcterms:description",    # however a standard-term would be called
    "value": "Abstract blablbablablabal...",
    "annotations": {
        "obo:NCIT_C42704": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d",
        "obo:NCIT_P378": "https://juser.fz-juelich.de/record/1014812"}
    },
    {"predicate": "dcterms:kind-of-publication",     # however a standard-term would be called
    "value": "bibo:AcademicArticle",
    "annotations": {
        "obo:NCIT_C42704": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d",
        "obo:NCIT_P378": "https://juser.fz-juelich.de/record/1014812"}
    }
    ]  
}

The juser scraping script implements the pattern.
The calendar scraper and existing records need to be fixed, but this is somewhat reliant on the calendar working again.

The discussion has - for now - reached a conclusion, recorded in [the hedgedoc](https://hedgedoc.psychoinformatics.de/3cSouq0YSJ6m64_ArWpJEg) and the current best practice example repeated below: ``` { "attributed_to": [ { "schema_type": "dlthings:Attribution", "object": "xyzrins:persons/simon-eickhoff", "annotations": { "http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d", "http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/1014812" } } ], "generated_by": [ { "schema_type": "dlthings:Generation", "object": "xyzrins:projects/858d0ddb-3c51-4cea-a0e5-a73224b59ae5", "annotations": { "http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d", "http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/1014812" } }, { "at_location": "ISSN:1741-7015", "at_time": "2023", "schema_type": "dlthings:Generation", "object": "http://purl.obolibrary.org/obo/IAO_0000444", "annotations": { "http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d", "http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/1014812" } } ], "description": "Abstract blablablabla....", "identifiers": [ { "schema_type": "dlthings:DOI", "notation": "10.1186/s12916-023-02941-4", "annotations": { "http://purl.org/pav/importedBy": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d", "http://purl.org/pav/importedFrom": "https://juser.fz-juelich.de/record/1014812" } } ], "kind": "bibo:AcademicArticle", "schema_type": "xyzri:XYZPublication", "pid": "xyzrins:publications/fe03c744-3aaa-469e-b789-c4d6198358a0", "title": "Sampling inequalities affect generalization of neuroimaging-based diagnostic classifiers in psychiatry", "attributes": [ { "predicate": "dcterms:title", # however a standard-term would be called "value": "Sampling inequalities affect generalization of neuroimaging-based diagnostic classifiers in psychiatry", "annotations": { "obo:NCIT_C42704": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d", "obo:NCIT_P378": "https://juser.fz-juelich.de/record/1014812"} }, {"predicate": "dcterms:description", # however a standard-term would be called "value": "Abstract blablbablablabal...", "annotations": { "obo:NCIT_C42704": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d", "obo:NCIT_P378": "https://juser.fz-juelich.de/record/1014812"} }, {"predicate": "dcterms:kind-of-publication", # however a standard-term would be called "value": "bibo:AcademicArticle", "annotations": { "obo:NCIT_C42704": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d", "obo:NCIT_P378": "https://juser.fz-juelich.de/record/1014812"} } ] } ``` The juser scraping script implements the pattern. The calendar scraper and existing records need to be fixed, but this is somewhat reliant on the calendar working again.

adina changed title from ~~Machine-annotation meanderings~~ to Machine-annotation schema for calendar and juser

2026-07-24 12:46:11 +00:00

adina referenced this issue

2026-08-03 11:14:54 +00:00

House-keeping: Remove competitions with an application deadline in the past? #10

adina referenced this issue from a commit

2026-08-03 11:46:54 +00:00

Rewrite calendar scraper to comply to new machine-annotation schema

adina commented

2026-08-03 13:47:10 +00:00

Author

Owner