orinoco/things-enrichment-tools
Watch
5
0
Fork
You've already forked things-enrichment-tools
 1
Code Issues 3 Pull requests Packages Activity Actions
things-enrichment-tools/README.md
adina 4273db1ded Juser scraper (#6) 
To test run the current state:

1) Download xml data with one of JUSERs generated search URLs
(see https://juser.fz-juelich.de/search_generator.py)
> mkdir /tmp/.cache
> curl 'https://juser.fz-juelich.de/PubExporter.py?p=cid%3A%22I%3A%28DE-Juel1%29INM-7-20090406%22+AND+pub%3A%222026%22&sf=author&so=d&rg=&of=xm' > /tmp/.cache/juser-pubs.xml

2) Cache Person and Publication records from your pool
> dtc get-records $DUMPTHINGS_APIURL public -C XYZPublication > /tmp/.cache/Publications.jsonl
> dtc get-records $DUMPTHINGS_APIURL public -C XZYPerson > /tmp/.cache/Person.jsonl

3) set DTC_TOKEN, JUSER_PW, and JUSER_USER in your environment
> export DTC_TOKEN=...addmeforreal
> export JUSER_PW=...addmeforreal
> export JUSER_USER=...addmeforreal

4) Invoke the script
> uv run tools/scrape-juser.py --file /tmp/.cache/juser-pubs.xml --persons /tmp/.cache/Person.jsonl --publications /tmp/.cache/Publications.jsonl

The script prints the generated publication and person records instead of submitting them.

Co-authored-by: Adina Wagner <adina.wagner@t-online.de>
Reviewed-on: #6
2026-07-24 12:16:26 +00:00

1.4 KiB
Raw Permalink Blame History

Knowledge Enrichment

This repository hosts tools and workflows to enrich dumpthings knowledge pools with additional machine-generated records.

available tools

  • tools/scrape-calendar.py (used by .forgejo/workflows/scrape.yml): scrapes three FZJ funding calendars and adds their events as XYZCompetition records (research information scheme)
  • tools/enrich-via-doi.py (used by .forgejo/workflows/enrich_publications.yml): reads publication records and extends them with external metadata available via doi.org content negotiation
  • tools/get-depiction-urls.py (used by the 'from-model'-websites, e.g., https://hub.psychoinformatics.de/www/www-from-model/src/branch/main/.forgejo/workflows/register-depictions.yaml): Given metadata input, this script extracts download URLs for each Depiction of the record, provided the depiction has a kind included in the depiction-type argument. For each depiction distribution, it will output the record curie, the file extension and the url, to stdout.
  • tools/scrape-juser.py (used by .forgejo/workflows/scrape-juser.yml): Given an xml export of publications in MARC21 format as provided by Juser (juser.fz-juelich.de), as well as a local cache of XYZPerson and XYZPublication records, this script generates new or updates existing Publication and Person records. Person records are submitted into a 'protected' inbox, whereas Publication records are submitted into a 'public' inbox.