colstore.ColStoreDataset

class colstore.ColStoreDataset(sources=None, *, on_mismatch='strict', **reader_kwargs)

Bases: _ReaderBase

One or more same-schema .cstore files presented as one logical table.

Construct from any mix of paths and open readers/datasets, or empty:

ColStoreDataset()                 # empty; grow later with append()/|=
ColStoreDataset(path)             # open and own one file
ColStoreDataset([path])           # same
ColStoreDataset(reader)           # borrow one open reader
ColStoreDataset([p, reader, ds])  # paths owned, readers/datasets borrowed
ColStoreDataset("run_*.cstore")   # glob: own every match, numeric order
ColStoreDataset("trades/")        # a directory: its .cstore shards, in order

Paths are opened and owned (closed on close()); readers and datasets are borrowed (left open). Datasets are flattened to their children. A path string may be a glob (*, ?, [; ** recursive), expanded to its matches in numeric order; a pattern matching no files raises FileNotFoundError. A path naming a directory expands to its .cstore shards in numeric order – the managed dataset append() / appender() write to – so a directory and a list of files compose the same way (an empty directory is an empty dataset). The public read surface matches ColStoreReader.

The native multi-file gather’s per-column segment table (_native_segment_table()) depends only on the children and the column, not on the rows a read requests, so it is memoized per column in _segment_table_cache and reused across reads. Rebuilding it dominates the per-read cost of the small, repeated fancy reads that index sampling over many files issues – the table-build is O(n_files) while the kernel touches only the sampled rows – so the memo is what keeps that workload fast. The cache is cleared whenever the children change (_rebuild_offsets()), so it never serves a stale table.

Whether that table is a uniform grid – every segment the same row count except possibly the global-last – is likewise a property of the children, not the read, so it too is memoized (_uniform_grid_rows, reset on the same child change). On a uniform grid the unsorted gather divides instead of searching (_uniform_segment_grid()).

Parameters:

• sources (Source | Sequence[Source] | None)

• on_mismatch (OnMismatch)

• reader_kwargs (Any)

append(source, **reader_kwargs)

Grow the dataset in place; return self so calls can be chained.

source is anything the constructor accepts: a path – a file, a glob, or a directory of shards – (opened and owned), a reader or dataset (borrowed), or a list/tuple mixing them.

Under the default on_mismatch='strict' the first child establishes the schema and later children must match it (same column names and dtypes; column order may differ, since reads are by name), or a ValueError leaves the dataset unchanged and closes anything this call opened. Under on_mismatch='drop' the dataset instead exposes only the columns common to every file with one consistent dtype, warning about the rest; it raises only if no column is common to all.

Parameters:

• source (Source | Sequence[Source])

• reader_kwargs (Any)

Return type:

ColStoreDataset

property n_rows: int

Total number of rows across all files (0 when empty).

property paths: tuple[Path, ...]

The child files’ paths, in order (empty for borrowed-only/empty).

property backend: str

Gather backend, taken from the first file, else the configured default.

property max_workers: int

Multi-column thread-pool size, from the first file, else the default.

property needs_compaction: bool

True while the data is spread across more than one file.

close()

Close the dataset, closing only the children it opened (owns).

Return type:

None