colstore

A memory-mapped columnar binary format for fast, memory-efficient random-access I/O on structured arrays.

colstore writes a tabular dataset to a single .cstore file — in one shot or streamed across many records — and loads arbitrary row/column subsets back without materializing the rest. Reads are memory-mapped, so process memory stays bounded by the output you ask for, never the file size; fancy-index and boolean gathers run through a parallel C++/OpenMP kernel bound via Cython.

Zero-decode reads

The on-disk column is raw NumPy bytes, so a gather is a parallel memcpy of the pages you touch — no decode, no decompress.

Bounded memory

Reads are np.memmap; a 1% slice of a file larger than RAM costs a few pages, not the whole frame.

Zero-copy views

A compacted, native-byte-order store hands back read-only ndarrays aliasing the page cache (copy=False).

Multi-file & shards

Open many same-schema files — or a growable directory of shards — as one logical table.

Lazy edits & queries

reader.edit() derives a new file through a deferred column-expression graph, without touching the source.

Format interop

A zero-copy Arrow bridge plus Parquet / Feather / JSON / HDF5 / NPZ / ROOT converters.

Install

pip install colstore

Quick start

import colstore

ds = colstore.store(df, "data.cstore")     # one-shot write + open
ds = colstore.open("data.cstore")          # or open an existing file

ds["price"]                                 # lazy ColumnView, nothing read yet
ds[[1, 5, 9], ["price", "qty"]].dict()      # materialize a fancy gather
ds[100:200].dict(copy=False)                # zero-copy views over the mmap