DANDI¶
Addresses, streaming and download of files on the DANDI archive. Users only pass a DANDI address to NWBSession (see Data sources); these functions are what it uses.
DANDI addresses for NWBSession.
Opens NWB files of the DANDI Archive without the dandi client package: the archive's REST API
(https://api.dandiarchive.org/api/, through requests) finds the version and the asset, and the
file is either streamed (remfile, with a disk cache of the chunks read) or downloaded once into a
download folder.
Addresses accepted (parse_reference, no network call):
dandi://000728/sub-x/sub-x_ses-y.nwb latest published version (else draft)
dandi://000728@0.240827.1809/sub-x/sub-x_ses-y.nwb that version
dandi://DANDI/000728@0.240827.1809/sub-x/... (form printed by the dandi CLI)
https://dandiarchive.org/dandiset/000728[/<version>]/files?location=sub-x/sub-x_ses-y.nwb
https://api.dandiarchive.org/api/assets/<asset_id>/download/ (also without download/)
https://api.dandiarchive.org/api/dandisets/000728/versions/<v>/assets/<asset_id>/download/
https://<any host>/<...>.nwb, https://dandiarchive.s3.amazonaws.com/blobs/... (streamed as is)
Whole dandisets or folders (parse_dandiset_reference, list_assets; listed, not opened):
dandi://000728[@<version>][/<folder>], DANDI:000728[/<version>], 000728
https://dandiarchive.org/dandiset/000728[/<version>][/files?location=<folder>]
https://doi.org/10.48324/dandi.000728[/<version>]
Modes (open_nwb):
stream remfile + disk cache (default)
download one-asset download into the download folder, reused when present; streams instead if the
file is bigger than the free disk space or `max_download_gb`
auto download when the asset is <= 1 GB, else stream
Mode: argument (GUI dialog, dandi_mode= of NWBSession and of the runner functions, cicada-run --dandi-mode),
else "stream".
Stream chunks: environment variable CICADA_DANDI_CHUNKS, else stream_cache_dir()); nothing removes them except clear_stream_cache() (the GUI calls it when CICADA
quits; cicada-dandi cache clear).
Downloads: the folder given (GUI "Download to" row, NWBSession(dandi_download_dir=...) and the runner
functions, --dandi-download-dir, open_nwb(download_folder=...)), else download_dir()); either way as
DandiError ¶
Bases: Exception
A DANDI address could not be resolved or opened (unknown dandiset, version or asset, API error).
DandiRef
dataclass
¶
A parsed address. kind: 'dandi' (dandiset + path), 'asset' (asset id) or 'url' (any file URL).
DandisetRef
dataclass
¶
cache_dir ¶
cache_dir()
CICADA's DANDI cache folder in the user cache folder (stream chunks in its "chunks" folder).
canonical_address ¶
canonical_address(dandiset_id, version, path)
Version-pinned address written in the sessions and combined YAMLs.
choose_mode ¶
choose_mode(mode, source, max_download_gb=DEFAULT_MAX_DOWNLOAD_GB, folder=None)
'stream' or 'download' for this file, and a message when the mode asked for is not used. Plain URLs (no DANDI metadata, no known size) are always streamed.
clear_stream_cache ¶
clear_stream_cache(folder=None)
Delete the stream chunk cache (downloaded files are kept). Safe to call when it does not exist. Returns the number of bytes freed. The GUI calls it when CICADA quits; streamed files are then fetched again at the next use.
dandiset_info ¶
dandiset_info(dandiset_id, version=None, api_url=API_URL)
{dandiset_id, version, name, asset_count, size} of a dandiset version (latest published by default).
download ¶
download(source, folder=None, chunk_size=8 * 1024 * 1024)
Download the asset into folder (default download_dir(); kept between runs) and return its path. A file already there with the right size is reused; a new download is checked against the sha256 of the archive.
download_path ¶
download_path(source, folder=None)
Where a DANDI asset is (or would be) downloaded in folder (default download_dir()).
find_asset ¶
find_asset(dandiset_id, version, path, api_url=API_URL)
{asset_id, path, size, ...} of the asset at exactly that path in that version.
is_reference ¶
is_reference(ref)
True if ref is an address NWBSession opens through this module (no network call).
list_assets ¶
list_assets(dandiset_id, version, path_prefix='', nwb_only=True, api_url=API_URL, page_size=1000)
Files of a dandiset version, sorted by path: list of {path, size, asset_id, address} (address: version-pinned dandi:// address). Only .nwb files unless nwb_only=False. path_prefix restricts to a folder (prefix match, as the API does).
open_nwb ¶
open_nwb(ref, mode=None, max_download_gb=DEFAULT_MAX_DOWNLOAD_GB, folder=None, download_folder=None, api_url=API_URL)
Resolve and open a DandiRef.
folder: cache folder of the stream chunks (in its "chunks" folder); default see stream_cache_dir(). download_folder: where downloads go (e.g. chosen by the user in the GUI); default download_dir().
Returns (io, handles, source): io is the NWBHDF5IO to read, handles the objects to close after
it (h5py file and remfile object when streaming), source the dict of resolve() with mode and
local_path filled in.
open_stream ¶
open_stream(url, folder=None)
(remfile file, h5py file, NWBHDF5IO) reading the file at url; chunks are cached on disk.
parse_dandiset_reference ¶
parse_dandiset_reference(ref)
Return a DandisetRef for an address of a whole dandiset or of a folder in it, None for anything else (single files included: use parse_reference for those).
parse_reference ¶
parse_reference(ref)
Return a DandiRef for a DANDI address or a file URL, None for anything else (e.g. local paths).
resolve ¶
resolve(ref, api_url=API_URL)
Look up a DandiRef on the archive.
Returns the source dict kept by NWBSession as session.source:
kind ('dandi' / 'asset' / 'url'), address (version-pinned address to record), dandiset_id,
version, asset_path, asset_id, size (bytes), sha256, url (file address streamed or downloaded),
mode and local_path (set when opened).
resolve_version ¶
resolve_version(dandiset_id, version=None, api_url=API_URL)
The version asked for, else the most recent published version, else 'draft' (with a warning).
stream_cache_dir ¶
stream_cache_dir(folder=None)
Folder of the stream chunk cache (pieces of streamed files, reused while they are kept):