iBOTS
Dask
Python library for parallel and out-of-core computing with familiar NumPy, Pandas, and xarray-style interfaces.
Our View
Dask helps researchers scale analyses beyond a single in-memory array or table while staying close to the scientific Python tools they already use.
Provisional positions are editorial proposals awaiting group review.
Teaching — Recorded in topic offerings; consult the dated event links for availability.
2 topics with recorded teaching · 1 past offerings · 1 upcoming offerings
Related themes: Scientific Computing
Related topics: Arrays and Numerical ComputingDataframes and StatisticsRobust and Reusable Software
Common use cases
- Process arrays and tables in chunks
- Work with datasets that are larger than available memory
- Parallelize computational workflows on local machines or clusters
Getting started
Dask extends the scientific Python ecosystem with lazy, chunked, and parallel computation. It can wrap array and table operations so they are planned first and executed later, often in pieces that fit into memory.
Why It Matters in Research Workflows
Large imaging, electrophysiology, and behavioral datasets can make otherwise clear analysis code fail because the whole dataset no longer fits in memory. Dask gives researchers a way to keep the analysis structure recognizable while changing how work is scheduled and loaded.
In Our Teaching
We introduce Dask in the context of high-performance Python and memory-aware data analysis. The course material on chunk-based analysis with xarray, HDF5, and Dask shows how chunking changes the memory profile of an analysis pipeline.