Michael Hanke
88be5943f8
This removes the need to have admin-level access to a machine for running the handbook code. This should make testing in a much broader range of environments possible (think HPC accounts, etc). The contributing guide and appveyor setup are adjusted.
321 lines
14 KiB
ReStructuredText
321 lines
14 KiB
ReStructuredText
.. _gin:
|
|
|
|
Walk-through: Dataset hosting on GIN
|
|
------------------------------------
|
|
|
|
`GIN <https://gin.g-node.org/G-Node/Info/wiki>`__ (G-Node infrastructure) is a
|
|
free data management system designed for comprehensive and reproducible management
|
|
of scientific data. It is a web-based repository store and provides
|
|
fine-grained access control to share data. :term:`GIN` builds up on :term:`Git` and
|
|
:term:`git-annex`, and is an easy alternative to other third-party services to host
|
|
and share your DataLad datasets [#f1]_. It allows to share datasets and their
|
|
contents with selected collaborators or making them publicly and anonymously
|
|
available.
|
|
:ref:`And even if you prefer to expose and share your datasets via GitHub, you can still use GIN to host your data <ginbts>`.
|
|
|
|
.. figure:: ../artwork/src/publishing/publishing_network_publishgin.svg
|
|
:width: 80%
|
|
|
|
Some repository hosting services such as GIN have annex support, and can thus hold the complete dataset. This makes publishing datasets very easy.
|
|
|
|
Prerequisites
|
|
^^^^^^^^^^^^^
|
|
|
|
In order to use GIN for hosting and sharing your datasets, you need to
|
|
|
|
- register
|
|
- upload your public :term:`SSH key` for SSH access
|
|
|
|
Once you have `registered <https://gin.g-node.org/user/sign_up>`_
|
|
an account on the GIN server by providing your e-mail address, affiliation,
|
|
and name, and selecting a user name and password, you should upload your
|
|
:term:`SSH key` to allow SSH access
|
|
(you can find an explanation of what SSH keys are and how you can create one in :ref:`this Findoutmore <fom-sshkey>` in the general section :ref:`share_hostingservice`).
|
|
To do this, visit the settings of your user account. On the left hand side, select
|
|
the tab "SSH Keys", and click the button "Add Key":
|
|
|
|
.. figure:: ../artwork/src/GIN_SSH_1.png
|
|
|
|
Upload your SSH key to GIN
|
|
|
|
|
|
You should copy the contents of your public key file into the field labeled
|
|
``content``, and enter an arbitrary but informative ``Key Name``, such as
|
|
"My private work station". Afterwards, you are done!
|
|
|
|
|
|
.. index::
|
|
pair: create-sibling-gin; DataLad command
|
|
|
|
Publishing your dataset to GIN
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
As outlined in the section :ref:`share_hostingservice`, there are two ways in which you can publish your dataset to GIN.
|
|
Either by 1) creating a new, empty repository on GIN via the web interface, or 2) via the :dlcmd:`create-sibling-gin` command.
|
|
|
|
**1) via webinterface:** If you choose to create a new repository via GIN's web interface, make sure to not initialize it with a README:
|
|
|
|
.. figure:: ../artwork/src/GIN_newrepo.png
|
|
|
|
Create a new repository on GIN using the web interface.
|
|
|
|
Afterwards, add this repository as a sibling of your dataset. To do this, use the
|
|
:dlcmd:`siblings add` command and the SSH URL of the repository as shown below.
|
|
Note that since this is the first time you will be connecting to the GIN server
|
|
via SSH, you will likely be asked to confirm to connect. This is a safety measure,
|
|
and you can type "yes" to continue:
|
|
|
|
.. code-block:: text
|
|
|
|
$ datalad siblings add -d . \
|
|
--name gin \
|
|
--url git@gin.g-node.org:/adswa/DataLad-101.git
|
|
|
|
The authenticity of host 'gin.g-node.org (141.84.41.219)' can't be established.
|
|
ECDSA key fingerprint is SHA256:E35RRG3bhoAm/WD+0dqKpFnxJ9+yi0uUiFLi+H/lkdU.
|
|
Are you sure you want to continue connecting (yes/no)? yes
|
|
[INFO ] Failed to enable annex remote gin, could be a pure git or not accessible
|
|
[WARNING] Failed to determine if gin carries annex.
|
|
.: gin(-) [git@gin.g-node.org:/adswa/DataLad-101.git (git)]
|
|
|
|
.. ifconfig:: internal
|
|
|
|
.. runrecord:: _examples/DL-101-139-101
|
|
:language: console
|
|
|
|
$ python3 $HOME/makepushtarget.py "$HOME/dl-101/DataLad-101" 'gin' "$HOME/pushes/DataLad-101" True True
|
|
|
|
**2) via command-line:**
|
|
If you choose to use the :dlcmd:`create-sibling-gin` command, supply the command with a name for the repository, and optionally add a ``-s/--siblingname [NAME]`` parameter (if unconfigured it will be ``gin``), and ``--access-protocol [https|ssh|https-ssh]`` (ideally ``ssh``).
|
|
The command has a number of additional useful parameters, so make sure to take a look at its ``--help``.
|
|
|
|
|
|
Afterwards, you can publish your dataset with :dlcmd:`push`. As the
|
|
repository on GIN supports a dataset annex, there is no publication dependency
|
|
to an external data hosting service necessary, and the dataset contents
|
|
stored in Git and in git-annex are published to the same place:
|
|
|
|
.. runrecord:: _examples/DL-101-139-102
|
|
:language: console
|
|
:workdir: dl-101/DataLad-101
|
|
|
|
$ datalad push --to gin
|
|
|
|
On the GIN web interface you will find all of your dataset -- including annexed contents!
|
|
What is especially cool is that the GIN web interface (unlike :term:`GitHub`) can even preview your annexed contents.
|
|
|
|
.. figure:: ../artwork/src/GIN_dl101_repo.png
|
|
|
|
A published dataset in a GIN repository at gin.g-node.org.
|
|
|
|
.. _access:
|
|
|
|
Sharing and accessing the dataset
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
Once your dataset is published, you can point collaborators and friends to it.
|
|
|
|
If it is a **public** repository, retrieving the dataset and getting access to
|
|
all published data contents (in a read-only fashion) is done by cloning the
|
|
repository's ``https`` url. This does not require a user account on GIN.
|
|
|
|
.. index::
|
|
pair: clone; DataLad command
|
|
.. importantnote:: Take the URL in the browser, not the copy-paste URL
|
|
|
|
Please note that you need to use the browser URL of the repository, not the copy-paste URL on the upper right hand side of the repository if you want to get anonymous HTTPS access!
|
|
The two URLs differ only by a ``.git`` extension:
|
|
|
|
* Browser bar: ``https://gin.g-node.org/<user>/<repo>``
|
|
* Copy-paste "HTTPS clone": ``https://gin.g-node.org/<user>/<repo>.git``
|
|
|
|
A dataset cloned from ``https://gin.g-node.org/<user>/<repo>.git``, however, cannot retrieve annexed files!
|
|
|
|
.. runrecord:: _examples/DL-101-139-107
|
|
:language: console
|
|
:workdir: dl-101/clone_of_dl-101
|
|
|
|
$ datalad clone https://gin.g-node.org/adswa/DataLad-101
|
|
|
|
Subsequently, :dlcmd:`get` calls will be able to retrieve all annexed
|
|
file contents that have been published to the repository.
|
|
|
|
.. index::
|
|
pair: clone; DataLad command
|
|
|
|
If it is a **private** dataset, cloning the dataset from GIN requires a user
|
|
name and password for anyone you want to share your dataset with.
|
|
The "Collaboration" tab under Settings lets you set fine-grained access rights,
|
|
and it is possible to share datasets with collaborators that are not registered
|
|
on GIN with provided Guest accounts.
|
|
If you are unsure if your dataset is private, :ref:`this find-out-more shows you how to find out <fom-private-gin>`.
|
|
In order to get access to annexed contents, cloning *requires* setting up
|
|
an SSH key as detailed above, and cloning via the SSH url:
|
|
|
|
.. code-block:: console
|
|
|
|
$ datalad clone git@gin.g-node.org:/adswa/DataLad-101.git
|
|
|
|
Likewise, in order to publish changes back to a GIN repository, the repository needs
|
|
to be cloned via its SSH url.
|
|
|
|
.. index:: dataset hosting; GIN
|
|
.. find-out-more:: How do I know if my repository is private?
|
|
:name: fom-private-gin
|
|
:float:
|
|
|
|
Private repos are marked with a lock sign. To make it public, untick the
|
|
"Private" box, found under "Settings":
|
|
|
|
..
|
|
the image below can't become a figure because it can't be used in LaTeXs minipage environment
|
|
|
|
.. image:: ../artwork/src/GIN_private.png
|
|
|
|
|
|
|
|
.. index::
|
|
pair: subdatasets; DataLad command
|
|
.. _subdspublishing:
|
|
|
|
Subdataset publishing
|
|
^^^^^^^^^^^^^^^^^^^^^
|
|
Just as the input subdataset ``iris_data`` in your published ``midterm_project``
|
|
was referencing its source on :term:`GitHub`, the ``longnow`` subdataset in your
|
|
published ``DataLad-101`` dataset directly references the original
|
|
dataset on :term:`GitHub`. If you click onto ``recordings`` and then ``longnow`` in GIN's webinterface, you will
|
|
be redirected to the podcast's original dataset.
|
|
|
|
The subdataset ``midterm_project``, however, is not successfully referenced. If
|
|
you click on it, you would get to a 404 Error page. The crucial difference between this
|
|
subdataset and the longnow dataset is its entry in the ``.gitmodules`` file of
|
|
``DataLad-101``:
|
|
|
|
.. code-block:: ini
|
|
:emphasize-lines: 4, 8
|
|
|
|
$ cat .gitmodules
|
|
[submodule "recordings/longnow"]
|
|
path = recordings/longnow
|
|
url = https://github.com/datalad-datasets/longnow-podcasts.git
|
|
datalad-id = b3ca2718-8901-11e8-99aa-a0369f7c647e
|
|
[submodule "midterm_project"]
|
|
path = midterm_project
|
|
url = ./midterm_project
|
|
datalad-id = e5a3d370-223d-11ea-af8b-e86a64c8054c
|
|
|
|
While the longnow subdataset is referenced with a valid URL to GitHub, the midterm
|
|
project's URL is a relative path from the root of the superdataset. This is because
|
|
the ``longnow`` subdataset was installed with :dlcmd:`clone -d .`
|
|
(that records the source of the subdataset), and the ``midterm_project`` dataset
|
|
was created as a subdataset with :dlcmd:`create -d . midterm_project`.
|
|
Since there is no repository at
|
|
``https://gin.g-node.org/<USER>/DataLad-101/midterm_project`` (which this submodule
|
|
entry would resolve to), accessing the subdataset fails.
|
|
|
|
However, since you have already published this dataset (to GitHub), you could
|
|
update the submodule entry and provide the accessible GitHub URL instead. This
|
|
can be done via the ``set-property <NAME> <VALUE>`` option of
|
|
:dlcmd:`subdatasets` [#f3]_ (replace the URL shown here with the URL
|
|
your dataset was published to -- likely, you only need to change the user name):
|
|
|
|
.. runrecord:: _examples/DL-101-139-103
|
|
:language: console
|
|
:workdir: dl-101/DataLad-101
|
|
|
|
$ datalad subdatasets --contains midterm_project \
|
|
--set-property url https://github.com/adswa/midtermproject
|
|
|
|
.. runrecord:: _examples/DL-101-139-104
|
|
:language: console
|
|
:workdir: dl-101/DataLad-101
|
|
|
|
$ cat .gitmodules
|
|
|
|
Handily, the :dlcmd:`subdatasets` command saved this change to the
|
|
``.gitmodules`` file automatically and the state of the dataset is clean:
|
|
|
|
.. runrecord:: _examples/DL-101-139-105
|
|
:language: console
|
|
:workdir: dl-101/DataLad-101
|
|
|
|
$ datalad status
|
|
|
|
Afterwards, publish these changes to ``gin`` and see for yourself how this fixed
|
|
the problem:
|
|
|
|
.. runrecord:: _examples/DL-101-139-106
|
|
:language: console
|
|
:workdir: dl-101/DataLad-101
|
|
|
|
$ datalad push --to gin
|
|
|
|
If the subdataset was not published before, you could publish the subdataset to
|
|
a location of your choice, and modify the ``.gitmodules`` entry accordingly.
|
|
|
|
.. index::
|
|
single: configuration item; remote.<name>.annex-ignore
|
|
pair: configure sibling; with DataLad
|
|
.. _ginbts:
|
|
|
|
Using GIN as a data source behind the scenes
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
Even if you do not want to point collaborators to yet another hosting site but want to be able to expose your datasets via services they use and know already (such as GitHub or GitLab), GIN can be very useful:
|
|
You can let GIN perform data hosting in the background by using it as an "autoenabled data source" that a dataset :term:`sibling` (even if it is published to GitHub or GitLab) can retrieve data from.
|
|
You will need to have a GIN account and SSH key setup, so please take a look at the first part of this section if you do not yet know how to do this.
|
|
|
|
Then, follow these steps:
|
|
|
|
- First, create a new repository on GIN (see step by step instructions above).
|
|
- In your to-be-published dataset, add this repository as a sibling, this time setting `--url` and `--pushurl` arguments explicitly. Make sure to configure a :term:`SSH` URL as a ``--pushurl`` but a :term:`HTTPS` URL as a ``url``.
|
|
Please also note that the :term:`HTTPS` URL written after ``--url`` DOES NOT have the ``.git`` suffix.
|
|
Here is the command:
|
|
|
|
.. code-block:: console
|
|
|
|
$ datalad siblings add \
|
|
-d . \
|
|
--name gin \
|
|
--pushurl git@gin.g-node.org:/studyforrest/aggregate-fmri-timeseries.git \
|
|
--url https://gin.g-node.org/studyforrest/aggregate-fmri-timeseries \
|
|
|
|
- Locally, run ``git config --unset-all remote.gin.annex-ignore`` to prevent :term:`git-annex` from ignoring this new dataset
|
|
- Push your data to the repository on GIN (``datalad push --to gin``). This pushes the actual state of the repository, including content, but also adjusts the :term:`git-annex` configuration.
|
|
- Configure this sibling as a "common data source". Use the same name as previously in ``--name`` (to indicate which sibling you are configuring) and give a new, different, name after ``--as-common-datasrc``:
|
|
|
|
.. code-block:: console
|
|
|
|
$ datalad siblings configure \
|
|
--name gin \
|
|
--as-common-datasrc gin-src
|
|
|
|
- Push to the repository on GIN again (``datalad push --to gin``) to make the configuration change known to the Gin sibling.
|
|
|
|
- Publish your dataset to GitHub/GitLab/..., or update an existing published dataset (``datalad push``)
|
|
|
|
Afterwards, :dlcmd:`get` retrieves files from GIN, even if the dataset has been cloned from GitHub.
|
|
|
|
.. index::
|
|
pair: common data source; DataLad concept
|
|
.. gitusernote:: Siblings as a common data source
|
|
|
|
The argument ``as-common-datasrc <name>`` configures a sibling as a common data source -- in technical terms, as an auto-enabled git-annex special remote.
|
|
|
|
.. rubric:: Footnotes
|
|
|
|
.. [#f1] GIN looks and feels similar to GitHub, and among a number advantages, it can
|
|
assign a :term:`DOI` to your dataset, making it cite-able. Moreover, its
|
|
`web interface <https://gin.g-node.org/G-Node/Info/wiki/WebInterface>`_
|
|
and `client <https://gin.g-node.org/G-Node/Info/wiki/GinUsageTutorial>`_ are
|
|
useful tools with a variety of features that are worthwhile to check out, as well.
|
|
|
|
.. [#f3] Alternatively, you can configure the siblings URL with :gitcmd:`config`:
|
|
|
|
.. code-block:: console
|
|
|
|
$ git config -f .gitmodules --replace-all submodule.midterm_project.url https://github.com/adswa/midtermproject
|
|
|
|
Remember, though, that this command modifies ``.gitmodules`` *without*
|
|
an automatic, subsequent :dlcmd:`save`, so that you will have to save
|
|
this change manually.
|