Conference site ยป Proceedings

Dask: Parallel Computation with Blocked algorithms and Task Scheduling

Matthew Rocklin
Continuum Analytics

Abstract

Dask enables parallel and out-of-core computation. We couple blocked algorithms with dynamic and memory aware task scheduling to achieve a parallel and out-of-core NumPy clone. We show how this extends the effective scale of modern hardware to larger datasets and discuss how these ideas can be more broadly applied to other parallel collections.

Keywords

parallelism, NumPy, scheduling

Bibtex entry

Full text PDF