colstore.ColStoreReader

class colstore.ColStoreReader(path, *, madvise='__default__', mlock=False, backend=None, max_workers=None)[source]

Bases: _ReaderBase

Memory-mapped columnar store with lazy, NumPy-style indexing.

Opening a store reads its header, creates one np.memmap per column, and applies any requested kernel hints. Reads are performed through __getitem__, which returns a lazy view: either a ColumnView (single-column) or a TableView (multi-column). The view materializes when one of its array / dict / recarray / frame methods is called.

Parameters:

  • path (str or pathlib.Path) – Path to a .cstore file produced by colstore.store() or ColStoreWriter.

  • madvise (str or None, optional) – Kernel access-pattern hint applied to every column memmap. One of "normal", "sequential", "random", "willneed", "dontneed", or None. Defaults to the package-wide setting.

  • mlock (bool, optional) – If True, attempt to lock every column’s pages in RAM. Failures emit a warning rather than raising. Defaults to False.

  • backend (str or None, optional) – Gather backend used for fancy-index reads ("cpp" or "numpy"). None uses the package-wide default. Applies to single-record stores; multi-record stores require the compiled C++ extension and always use it for fancy-index reads.

  • max_workers (int or None, optional) – Override the package-wide thread-pool size for multi-column reads. None uses the global setting (physical core count by default).

Examples

>>> ds = colstore.store(df, "data.cstore")
>>> ds['price']                      # ColumnView -> array()
>>> ds[100:200, ['price', 'qty']]    # TableView -> dict / recarray / frame
property path: Path

Filesystem path the store was opened from.

property paths: tuple[Path, ...]

a one-element tuple of this reader’s path.

Type:

The backing files

property n_rows: int

Number of rows in every column.

property backend: str

Effective gather backend on this instance.

Governs single-record fancy-index reads; multi-record fancy reads always use the C++ extension (see the backend parameter note).

property max_workers: int

Effective thread-pool size for multi-column reads on this instance.

close()[source]

Release all memmaps. Subsequent reads will fail.

Return type:

None