• ⸻ 2026-07-10

Scaling anndata training to the terabyte scale with annbatch

• Felix Fischer, Ilan Gold, Fabian Theis, Alex Wolf

The demand for AI in omics has grown at an unprecedented rate, with state-of-the-art models now routinely trained on datasets exceeding the terabyte scale. To make that process more efficient, we developed annbatch,[1] a high-performance data loader built on anndata that enables loading speeds of 60k samples/second and more, at least a factor of 3 higher than the fastest recent alternatives.

• ⸻ 2024-04-03

MappedCollection: Weighted random sampling from large collections of scRNA-seq datasets

• Sergei Rybakov, Felix Fischer, Maciek Wiatrak, Ilan Gold, Yanay Rosen, Sunny Sun, Chaichontat Sriworarat, Fabian Theis, Jeremie Kalfon, Alex Wolf

A few labs and companies now train models on large-scale scRNA-seq count matrices and related data modalities. But unlike for many other data types, there isn’t yet a playbook for data scales that don’t fit into memory.