orinoco/things-enrichment-tools
Watch
5
0
Fork
You've already forked things-enrichment-tools
 1
Code Issues 3 Pull requests Packages Activity Actions

Juser scraper #6

Merged
adina merged 47 commits from juser into main 2026-07-24 12:16:27 +00:00
Conversation 8 Commits 47 Files changed 3 +890 -1
adina commented 2026-06-30 14:12:34 +00:00
Owner
Copy link

To test run the current state:

  1. Download xml data with one of JUSERs generated search URLs
    (see https://juser.fz-juelich.de/search_generator.py)

mkdir /tmp/.cache
curl 'https://juser.fz-juelich.de/PubExporter.py?p=cid%3A%22I%3A%28DE-Juel1%29INM-7-20090406%22+AND+pub%3A%222026%22&sf=author&so=d&rg=&of=xm' > /tmp/.cache/juser-pubs.xml

  1. Cache Person and Publication records from your pool

dtc get-records $DUMPTHINGS_APIURL public -C XYZPublication > /tmp/.cache/Publications.jsonl
dtc get-records $DUMPTHINGS_APIURL public -C XZYPerson > /tmp/.cache/Person.jsonl

  1. set DTC_TOKEN, JUSER_PW, and JUSER_USER in your environment

export DTC_TOKEN=...addmeforreal
export JUSER_PW=...addmeforreal
export JUSER_USER=...addmeforreal

  1. Invoke the script

uv run tools/scrape-juser.py --file /tmp/.cache/juser-pubs.xml --persons /tmp/.cache/Person.jsonl --publications /tmp/.cache/Publications.jsonl

The script prints the generated publication and person records instead of submitting them.

To test run the current state: 1) Download xml data with one of JUSERs generated search URLs (see https://juser.fz-juelich.de/search_generator.py) > mkdir /tmp/.cache > curl 'https://juser.fz-juelich.de/PubExporter.py?p=cid%3A%22I%3A%28DE-Juel1%29INM-7-20090406%22+AND+pub%3A%222026%22&sf=author&so=d&rg=&of=xm' > /tmp/.cache/juser-pubs.xml 2) Cache Person and Publication records from your pool > dtc get-records $DUMPTHINGS_APIURL public -C XYZPublication > /tmp/.cache/Publications.jsonl > dtc get-records $DUMPTHINGS_APIURL public -C XZYPerson > /tmp/.cache/Person.jsonl 3) set DTC_TOKEN, JUSER_PW, and JUSER_USER in your environment > export DTC_TOKEN=...addmeforreal > export JUSER_PW=...addmeforreal > export JUSER_USER=...addmeforreal 4) Invoke the script > uv run tools/scrape-juser.py --file /tmp/.cache/juser-pubs.xml --persons /tmp/.cache/Person.jsonl --publications /tmp/.cache/Publications.jsonl The script prints the generated publication and person records instead of submitting them. 
the majority of psyinf records to not have such
a key in their orcid identifier dict
adina commented 2026-07-02 07:02:22 +00:00
Author
Owner
Copy link

TODOs:

  • anything marked in the code as #TODO

  • add Juser-related secrets this the repo: ${{ secrets.JUSER_PW }}, ${{ secrets.JUSER_USER }}

  • look up / mapping mechanism for 'rules' and 'about' (PoF topics)

  • figure out how to ingest grant information: Publications can acknowledge grants (I have yet to find one that does), but only if a Juser-specific record for the grant exists (under a Juser-specific ID), for which we then need to figure out the look-up as well (similar to looking up Julich IDs to match them with ORCIDs)

  • improve logging

  • implement and enable submission

  • record JulIDs as additional identifiers -> issued by ZB (which means we can get the entire INM-7 crew (or more people!) for free into the pool)

    • -> this means person records need to be protected!
    • -> this requires public and protected submissions
    • -> this requires caching records from several collections
    • ZB Org entry in pool
    • adjust lookup to not only rely on ORCID

  • MARCXML author names should be 'display_label', not 'formatted_name'

  • Publications are 'generated_by' PoF Topics

    • PoF Topics need to become projects
    • remove old topics and their links
TODOs: - [ ] anything marked in the code as #TODO - [x] add Juser-related secrets this the repo: ``${{ secrets.JUSER_PW }}``, ``${{ secrets.JUSER_USER }}`` - [x] look up / mapping mechanism for 'rules' and 'about' (PoF topics) - [x] figure out how to ingest grant information: Publications *can* acknowledge grants (I have yet to find one that does), but only if a Juser-specific record for the grant exists (under a Juser-specific ID), for which we then need to figure out the look-up as well (similar to looking up Julich IDs to match them with ORCIDs) - [ ] improve logging - [x] implement and enable submission - [x] record JulIDs as additional identifiers -> issued by ZB (which means we can get the entire INM-7 crew (or more people!) for free into the pool) - [x] -> this means person records need to be protected! - [x] -> this requires public and protected submissions - [x] -> this requires caching records from several collections - [x] ZB Org entry in pool - [x] adjust lookup to not only rely on ORCID - [x] MARCXML author names should be 'display_label', not 'formatted_name' - [x] Publications are 'generated_by' PoF Topics - [x] PoF Topics need to become projects - [x] remove old topics and their links
adina commented 2026-07-02 11:19:04 +00:00
Author
Owner
Copy link

ZB uses a controlled vocabulary for licenses: https://juser.fz-juelich.de/collection/HGFVOC?ln=en. Maybe this can be used for a look-up/mapping...

ZB uses a controlled vocabulary for licenses: https://juser.fz-juelich.de/collection/HGFVOC?ln=en. Maybe this can be used for a look-up/mapping...
adina commented 2026-07-02 13:04:09 +00:00
Author
Owner
Copy link

Example publication records:

{
  "schema_type": "xyzri:XYZPublication",
  "pid": "xyzrins:publications/d88f42d9-4f3f-452d-ae88-8a946b1903fd",
  "identifiers": [
    {
      "schema_type": "dlthings:DOI",
      "notation": "10.1002/advs.202509009"
    }
  ],
  "title": "Machine‐Learning Decomposition Identifies a Big Two Structure in Human Personality with Distinct Neurocognitive Profiles",
  "description": "Personality neuroscience has traditionally relied on the Big Five model to investigate trait structure and its relationship to individual differences in brain organization and life outcomes. However, existing theoretical frameworks explain only part of the item-level covariance, raising questions about whether alternative factor solutions might complement the canonical five-factor model. Here, we applied an additive and part-based machine learning decomposition to a mega-scale, global dataset (n = 1,336,840) to systematically evaluate trait structure across factor resolutions. Beyond reproducing the canonical Big Five, we identified a robust Big Two comprising Social Adaptation and Spontaneous Mentation. Social Adaptation integrates covarying questionnaire items from Extraversion, Agreeableness, and Conscientiousness, indexing externally oriented social functioning. Spontaneous Mentation, in turn, aggregates Neuroticism with introspective facets of Openness, capturing internally directed affective-cognitive exploration. Embedding individuals in this Big Two space revealed structured manifolds along which neurocognitive profiles aligned with distinct trait orientations. Importantly, this lower-dimensional representation improved prediction of functional brain connectivity relative to Big Five scores, while preserving comparable associations with cognition and mental health. Together, these results establish a neurocognitively grounded Big Two framework that complements the Big Five and offers an interpretable bridge between personality structure, cognitive functioning, and psychopathology.",
  "attributed_to": [
    {
      "schema_type": "dlthings:Attribution",
      "object": "xyzrins:persons/simon-eickhoff"
    }
  ],
  "generated_by": [
    {
      "at_location": "ISSN:2198-3844",
      "at_time": "2026",
      "schema_type": "dlthings:Generation",
      "object": "obo:IAO_0000444"
    }
  ],
  "attributes": [
    {
      "predicate": "http://purl.org/pav/importedFrom",
      "value": "https://juser.fz-juelich.de/record/1055806",
      "attributes": [
        {
          "predicate": "prov:generated",
          "value": "title",
          "characterized_by": [
            {
              "predicate": "prov:generated_by",
              "object": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d"
            }
          ]
        },
        {
          "predicate": "prov:generated",
          "value": "description",
          "characterized_by": [
            {
              "predicate": "prov:generated_by",
              "object": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d"
            }
          ]
        },
        {
          "predicate": "prov:generated",
          "value": "attributed_to",
          "characterized_by": [
            {
              "predicate": "prov:generated_by",
              "object": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d"
            }
          ]
        },
        {
          "predicate": "prov:generated",
          "value": "generated_by",
          "characterized_by": [
            {
              "predicate": "prov:generated_by",
              "object": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d"
            }
          ]
        }
      ]
    }
  ]
}

Example person record:

{
  "pid": "xyzrins:persons/497e549d-3adf-4205-8260-9dd313668b97",
  "display_label": "Maleki Balajoo, Somayeh",
  "identifiers": [
    {
      "schema_type": "xyzri:ORCID",
      "creator": "ror:04fa4r544",
      "notation": "0000-0002-5238-2448"
    },
    {
      "schema_type": "dlthings:Identifier",
      "creator": "https://w3id.org/isil/DE-Juel1",
      "notation": "P:(DE-Juel1)178767"
    }
  ],
  "attributes": [
    {
      "predicate": "http://purl.org/pav/importedFrom",
      "value": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d"
    }
  ]
}
Example publication records: ```json { "schema_type": "xyzri:XYZPublication", "pid": "xyzrins:publications/d88f42d9-4f3f-452d-ae88-8a946b1903fd", "identifiers": [ { "schema_type": "dlthings:DOI", "notation": "10.1002/advs.202509009" } ], "title": "Machine‐Learning Decomposition Identifies a Big Two Structure in Human Personality with Distinct Neurocognitive Profiles", "description": "Personality neuroscience has traditionally relied on the Big Five model to investigate trait structure and its relationship to individual differences in brain organization and life outcomes. However, existing theoretical frameworks explain only part of the item-level covariance, raising questions about whether alternative factor solutions might complement the canonical five-factor model. Here, we applied an additive and part-based machine learning decomposition to a mega-scale, global dataset (n = 1,336,840) to systematically evaluate trait structure across factor resolutions. Beyond reproducing the canonical Big Five, we identified a robust Big Two comprising Social Adaptation and Spontaneous Mentation. Social Adaptation integrates covarying questionnaire items from Extraversion, Agreeableness, and Conscientiousness, indexing externally oriented social functioning. Spontaneous Mentation, in turn, aggregates Neuroticism with introspective facets of Openness, capturing internally directed affective-cognitive exploration. Embedding individuals in this Big Two space revealed structured manifolds along which neurocognitive profiles aligned with distinct trait orientations. Importantly, this lower-dimensional representation improved prediction of functional brain connectivity relative to Big Five scores, while preserving comparable associations with cognition and mental health. Together, these results establish a neurocognitively grounded Big Two framework that complements the Big Five and offers an interpretable bridge between personality structure, cognitive functioning, and psychopathology.", "attributed_to": [ { "schema_type": "dlthings:Attribution", "object": "xyzrins:persons/simon-eickhoff" } ], "generated_by": [ { "at_location": "ISSN:2198-3844", "at_time": "2026", "schema_type": "dlthings:Generation", "object": "obo:IAO_0000444" } ], "attributes": [ { "predicate": "http://purl.org/pav/importedFrom", "value": "https://juser.fz-juelich.de/record/1055806", "attributes": [ { "predicate": "prov:generated", "value": "title", "characterized_by": [ { "predicate": "prov:generated_by", "object": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d" } ] }, { "predicate": "prov:generated", "value": "description", "characterized_by": [ { "predicate": "prov:generated_by", "object": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d" } ] }, { "predicate": "prov:generated", "value": "attributed_to", "characterized_by": [ { "predicate": "prov:generated_by", "object": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d" } ] }, { "predicate": "prov:generated", "value": "generated_by", "characterized_by": [ { "predicate": "prov:generated_by", "object": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d" } ] } ] } ] } ``` Example person record: ```json { "pid": "xyzrins:persons/497e549d-3adf-4205-8260-9dd313668b97", "display_label": "Maleki Balajoo, Somayeh", "identifiers": [ { "schema_type": "xyzri:ORCID", "creator": "ror:04fa4r544", "notation": "0000-0002-5238-2448" }, { "schema_type": "dlthings:Identifier", "creator": "https://w3id.org/isil/DE-Juel1", "notation": "P:(DE-Juel1)178767" } ], "attributes": [ { "predicate": "http://purl.org/pav/importedFrom", "value": "xyzrins:instruments/d3126ff5-623c-48af-ac48-4d921ef9b80d" } ] } ```
adina commented 2026-07-02 19:27:11 +00:00
Author
Owner
Copy link

In addition, we can get 'content type' from field 336 a. The "Content type term" however does not seem to follow a standard vocabulary (see below, and also note the various capitalizations between terms...). We can create a mapping with common types to our schema, e.g.,

'article' -> http://purl.org/ontology/bibo/AcademicArticle
'Preprint' -> ?
'lecture' -> http://purl.org/spar/fabio/Presentation
'Conference Paper' -> http://purl.org/ontology/bibo/Proceedings
'PATENT' -> http://purl.org/ontology/bibo/Patent

In addition, we can get 'content type' from field 336 a. The "Content type term" however does not seem to follow a standard vocabulary (see below, and also note the various capitalizations between terms...). We can create a mapping with common types to our schema, e.g., 'article' -> http://purl.org/ontology/bibo/AcademicArticle 'Preprint' -> ? 'lecture' -> http://purl.org/spar/fabio/Presentation 'Conference Paper' -> http://purl.org/ontology/bibo/Proceedings 'PATENT' -> http://purl.org/ontology/bibo/Patent
adina commented 2026-07-03 06:52:53 +00:00
Author
Owner
Copy link

Licenses are even more difficult. They use field 915 for it, which is not a standard part of MARC but 'local bibliographic information'. Its also overloaded with everything else, from APCs to Peer Review types...

Licenses are even more difficult. They use field 915 for it, which is not a standard part of MARC but 'local bibliographic information'. Its also overloaded with everything else, from APCs to Peer Review types...
its not relevant for us strictly, but publications often
use it nevertheless
adina commented 2026-07-18 15:16:33 +00:00
Author
Owner
Copy link

Of note: This submits records into different inbox collections of the same user:

  • Publications go into 'public'
  • Persons go into 'protected'

Curation needs to take this into account.

Of note: This submits records into different inbox collections of the same user: - Publications go into 'public' - Persons go into 'protected' Curation needs to take this into account.
source document: https://hedgedoc.psychoinformatics.de/3cSouq0YSJ6m64_ArWpJEg?view
adina commented 2026-07-23 12:45:18 +00:00
Author
Owner
Copy link

okay, this is getting somewhere closer to a finished state. Finding and fixing bugs now.

  • Updating records works in principle, but not for authors with role declarations (they get duplicated):
    image
okay, this is getting somewhere closer to a finished state. Finding and fixing bugs now. - Updating records works in principle, but not for authors with role declarations (they get duplicated): ![image](/attachments/0cf3ee67-01db-4bd2-a63f-1ff35280e1d2)
image.png
23 KiB
adina changed title from WIP: Juser scraper to Juser scraper 2026-07-23 15:00:17 +00:00
adina commented 2026-07-24 12:16:17 +00:00
Author
Owner
Copy link

I'll merge this PR now, and toy around with the CI workflow. As all submissions only go into an inbox, they are sandboxed and this should be "safe", even if the CI runs the workflow successfully.

I'll merge this PR now, and toy around with the CI workflow. As all submissions only go into an inbox, they are sandboxed and this should be "safe", even if the CI runs the workflow successfully.
adina merged commit 4273db1ded into main 2026-07-24 12:16:27 +00:00
adina referenced this pull request from a commit 2026-07-24 12:16:29 +00:00
Juser scraper (#6)
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
orinoco/things-enrichment-tools!6
No description provided.