Skip to content

DANDI

Addresses, streaming and download of files on the DANDI archive. Users only pass a DANDI address to NWBSession (see Data sources); these functions are what it uses.

DANDI addresses for NWBSession.

Opens NWB files of the DANDI Archive without the dandi client package: the archive's REST API (https://api.dandiarchive.org/api/, through requests) finds the version and the asset, and the file is either streamed (remfile, with a disk cache of the chunks read) or downloaded once into a download folder.

Addresses accepted (parse_reference, no network call):

dandi://000728/sub-x/sub-x_ses-y.nwb                       latest published version (else draft)
dandi://000728@0.240827.1809/sub-x/sub-x_ses-y.nwb         that version
dandi://DANDI/000728@0.240827.1809/sub-x/...               (form printed by the dandi CLI)
https://dandiarchive.org/dandiset/000728[/<version>]/files?location=sub-x/sub-x_ses-y.nwb
https://api.dandiarchive.org/api/assets/<asset_id>/download/   (also without download/)
https://api.dandiarchive.org/api/dandisets/000728/versions/<v>/assets/<asset_id>/download/
https://<any host>/<...>.nwb, https://dandiarchive.s3.amazonaws.com/blobs/...   (streamed as is)

Whole dandisets or folders (parse_dandiset_reference, list_assets; listed, not opened):

dandi://000728[@<version>][/<folder>], DANDI:000728[/<version>], 000728
https://dandiarchive.org/dandiset/000728[/<version>][/files?location=<folder>]
https://doi.org/10.48324/dandi.000728[/<version>]

Modes (open_nwb):

stream    remfile + disk cache (default)
download  one-asset download into the download folder, reused when present; streams instead if the
          file is bigger than the free disk space or `max_download_gb`
auto      download when the asset is <= 1 GB, else stream

Mode: argument (GUI dialog, dandi_mode= of NWBSession and of the runner functions, cicada-run --dandi-mode), else "stream". Stream chunks: environment variable CICADA_DANDI_CHUNKS, else /cicada/dandi/chunks (stream_cache_dir()); nothing removes them except clear_stream_cache() (the GUI calls it when CICADA quits; cicada-dandi cache clear). Downloads: the folder given (GUI "Download to" row, NWBSession(dandi_download_dir=...) and the runner functions, --dandi-download-dir, open_nwb(download_folder=...)), else /cicada (download_dir()); either way as ///, so files of different dandisets never collide and a file already downloaded there is reused. Embargoed dandisets: the API key in the environment variable DANDI_API_KEY.

API_URL module-attribute

API_URL = 'https://api.dandiarchive.org/api'

AUTO_DOWNLOAD_MAX_BYTES module-attribute

AUTO_DOWNLOAD_MAX_BYTES = 1000000000

DEFAULT_MAX_DOWNLOAD_GB module-attribute

DEFAULT_MAX_DOWNLOAD_GB = 10

DEFAULT_MODE module-attribute

DEFAULT_MODE = 'stream'

MODES module-attribute

MODES = ('stream', 'download', 'auto')

DandiError

Bases: Exception

A DANDI address could not be resolved or opened (unknown dandiset, version or asset, API error).

DandiRef dataclass

A parsed address. kind: 'dandi' (dandiset + path), 'asset' (asset id) or 'url' (any file URL).

asset_id class-attribute instance-attribute

asset_id: str = None

dandiset_id class-attribute instance-attribute

dandiset_id: str = None

kind instance-attribute

kind: str

path class-attribute instance-attribute

path: str = None

url class-attribute instance-attribute

url: str = None

version class-attribute instance-attribute

version: str = None

DandisetRef dataclass

A whole dandiset, or a folder of it (path_prefix), to list its NWB files.

dandiset_id instance-attribute

dandiset_id: str

path_prefix class-attribute instance-attribute

path_prefix: str = ''

version class-attribute instance-attribute

version: str = None

asset_metadata

asset_metadata(asset_id, api_url=API_URL)

cache_dir

cache_dir()

CICADA's DANDI cache folder in the user cache folder (stream chunks in its "chunks" folder).

canonical_address

canonical_address(dandiset_id, version, path)

Version-pinned address written in the sessions and combined YAMLs.

choose_mode

choose_mode(mode, source, max_download_gb=DEFAULT_MAX_DOWNLOAD_GB, folder=None)

'stream' or 'download' for this file, and a message when the mode asked for is not used. Plain URLs (no DANDI metadata, no known size) are always streamed.

clear_stream_cache

clear_stream_cache(folder=None)

Delete the stream chunk cache (downloaded files are kept). Safe to call when it does not exist. Returns the number of bytes freed. The GUI calls it when CICADA quits; streamed files are then fetched again at the next use.

dandiset_info

dandiset_info(dandiset_id, version=None, api_url=API_URL)

{dandiset_id, version, name, asset_count, size} of a dandiset version (latest published by default).

download

download(source, folder=None, chunk_size=8 * 1024 * 1024)

Download the asset into folder (default download_dir(); kept between runs) and return its path. A file already there with the right size is reused; a new download is checked against the sha256 of the archive.

download_dir

download_dir()

Default folder of downloads (no folder chosen): /cicada.

download_path

download_path(source, folder=None)

Where a DANDI asset is (or would be) downloaded in folder (default download_dir()).

find_asset

find_asset(dandiset_id, version, path, api_url=API_URL)

{asset_id, path, size, ...} of the asset at exactly that path in that version.

is_reference

is_reference(ref)

True if ref is an address NWBSession opens through this module (no network call).

list_assets

list_assets(dandiset_id, version, path_prefix='', nwb_only=True, api_url=API_URL, page_size=1000)

Files of a dandiset version, sorted by path: list of {path, size, asset_id, address} (address: version-pinned dandi:// address). Only .nwb files unless nwb_only=False. path_prefix restricts to a folder (prefix match, as the API does).

open_nwb

open_nwb(ref, mode=None, max_download_gb=DEFAULT_MAX_DOWNLOAD_GB, folder=None, download_folder=None, api_url=API_URL)

Resolve and open a DandiRef.

folder: cache folder of the stream chunks (in its "chunks" folder); default see stream_cache_dir(). download_folder: where downloads go (e.g. chosen by the user in the GUI); default download_dir().

Returns (io, handles, source): io is the NWBHDF5IO to read, handles the objects to close after it (h5py file and remfile object when streaming), source the dict of resolve() with mode and local_path filled in.

open_stream

open_stream(url, folder=None)

(remfile file, h5py file, NWBHDF5IO) reading the file at url; chunks are cached on disk.

parse_dandiset_reference

parse_dandiset_reference(ref)

Return a DandisetRef for an address of a whole dandiset or of a folder in it, None for anything else (single files included: use parse_reference for those).

parse_reference

parse_reference(ref)

Return a DandiRef for a DANDI address or a file URL, None for anything else (e.g. local paths).

resolve

resolve(ref, api_url=API_URL)

Look up a DandiRef on the archive.

Returns the source dict kept by NWBSession as session.source: kind ('dandi' / 'asset' / 'url'), address (version-pinned address to record), dandiset_id, version, asset_path, asset_id, size (bytes), sha256, url (file address streamed or downloaded), mode and local_path (set when opened).

resolve_version

resolve_version(dandiset_id, version=None, api_url=API_URL)

The version asked for, else the most recent published version, else 'draft' (with a warning).

stream_cache_dir

stream_cache_dir(folder=None)

Folder of the stream chunk cache (pieces of streamed files, reused while they are kept): /chunks, else the environment variable CICADA_DANDI_CHUNKS, else /chunks.