Recompute analysis results in dockerized environment #24

Merged

mih merged 6 commits from docker into master

2023-10-12 17:06:29 +00:00

mih commented

2023-10-12 07:47:11 +00:00

(Migrated from github.com)

Even with pinned versions of essential software dependencies, issues
of incompatibilities with modern Python versions slowly arise.

The container setup used for this recomputation is the result of a
detailed exploration on the effect of software versions and deployment
methods. A reports is provided at
https://github.com/psychoinformatics-de/paper-remodnav/issues/20

Importantly, the employed setup is NOT capable of yielded exactly
identical results. While all statistical scores reported in the paper
remain indeed identical, there is a visually small change to one
histogram panel in Fig 4. The change is illustrated at
https://github.com/psychoinformatics-de/paper-remodnav/issues/20#issuecomment-1757462683

Given the overall state of reproducibility, and the anticipated
longevity of the containerized computation, we decided that this small
difference with respect to the journal publication is tolerable.

This changeset support a DataLad-based re-execution (for verification):

datalad rerun <commitsha>

After this changeset, a complete manuscript can be compiled, also
via DataLad via a:

datalad containers-run -n docker-make main.pdf

Closes #20

TODO:

Deposit the container image layers somewhere. They have been put on storage that should have some longevity. The respective URLs are public and registered with the annex keys for the two docker layers

The creation of a containerized environment for the analysis became necessary, because 3+ years after the "final" results have been computed originally, it is getting difficult to recreate a matching computational environment. Even with pinned versions of essential software dependencies, issues of incompatibilities with modern Python versions slowly arise. The container setup used for this recomputation is the result of a detailed exploration on the effect of software versions and deployment methods. A reports is provided at https://github.com/psychoinformatics-de/paper-remodnav/issues/20 Importantly, the employed setup is NOT capable of yielded exactly identical results. While all statistical scores reported in the paper remain indeed identical, there is a visually small change to one histogram panel in Fig 4. The change is illustrated at https://github.com/psychoinformatics-de/paper-remodnav/issues/20#issuecomment-1757462683 Given the overall state of reproducibility, and the anticipated longevity of the containerized computation, we decided that this small difference with respect to the journal publication is tolerable. This changeset support a DataLad-based re-execution (for verification): ``` datalad rerun <commitsha> ``` After this changeset, a complete manuscript can be compiled, also via DataLad via a: ``` datalad containers-run -n docker-make main.pdf ``` Closes #20 TODO: - [x] Deposit the container image layers somewhere. They have been put on storage that should have some longevity. The respective URLs are public and registered with the annex keys for the two docker layers