adina
4273db1ded
To test run the current state: 1) Download xml data with one of JUSERs generated search URLs (see https://juser.fz-juelich.de/search_generator.py) > mkdir /tmp/.cache > curl 'https://juser.fz-juelich.de/PubExporter.py?p=cid%3A%22I%3A%28DE-Juel1%29INM-7-20090406%22+AND+pub%3A%222026%22&sf=author&so=d&rg=&of=xm' > /tmp/.cache/juser-pubs.xml 2) Cache Person and Publication records from your pool > dtc get-records $DUMPTHINGS_APIURL public -C XYZPublication > /tmp/.cache/Publications.jsonl > dtc get-records $DUMPTHINGS_APIURL public -C XZYPerson > /tmp/.cache/Person.jsonl 3) set DTC_TOKEN, JUSER_PW, and JUSER_USER in your environment > export DTC_TOKEN=...addmeforreal > export JUSER_PW=...addmeforreal > export JUSER_USER=...addmeforreal 4) Invoke the script > uv run tools/scrape-juser.py --file /tmp/.cache/juser-pubs.xml --persons /tmp/.cache/Person.jsonl --publications /tmp/.cache/Publications.jsonl The script prints the generated publication and person records instead of submitting them. Co-authored-by: Adina Wagner <adina.wagner@t-online.de> Reviewed-on: #6
15 lines
No EOL
1.4 KiB
Markdown
15 lines
No EOL
1.4 KiB
Markdown
# Knowledge Enrichment
|
|
|
|
This repository hosts tools and workflows to enrich dumpthings knowledge pools
|
|
with additional machine-generated records.
|
|
|
|
### available tools
|
|
|
|
- ``tools/scrape-calendar.py`` (used by
|
|
``.forgejo/workflows/scrape.yml``): scrapes three FZJ funding calendars and
|
|
adds their events as XYZCompetition records (research information scheme)
|
|
- ``tools/enrich-via-doi.py`` (used by
|
|
``.forgejo/workflows/enrich_publications.yml``): reads publication records and
|
|
extends them with external metadata available via doi.org content negotiation
|
|
- ``tools/get-depiction-urls.py`` (used by the 'from-model'-websites, e.g., https://hub.psychoinformatics.de/www/www-from-model/src/branch/main/.forgejo/workflows/register-depictions.yaml): Given metadata input, this script extracts download URLs for each ``Depiction`` of the record, provided the depiction has a ``kind`` included in the depiction-type argument. For each depiction distribution, it will output the record curie, the file extension and the url, to stdout.
|
|
- ``tools/scrape-juser.py`` (used by ``.forgejo/workflows/scrape-juser.yml``): Given an xml export of publications in MARC21 format as provided by Juser (juser.fz-juelich.de), as well as a local cache of XYZPerson and XYZPublication records, this script generates new or updates existing Publication and Person records. Person records are submitted into a 'protected' inbox, whereas Publication records are submitted into a 'public' inbox. |