colstore.concat

colstore.concat(sources: Sequence[str | PathLike[str] | ColStoreReader | ColStoreDataset], *, out: None = None, memory_budget: int | None = None, **reader_kwargs: Any) → ColStoreDataset[source]
colstore.concat(sources: Sequence[str | PathLike[str] | ColStoreReader | ColStoreDataset], *, out: str | PathLike[str], memory_budget: int | None = None, **reader_kwargs: Any) → ColStoreReader

Combine several same-schema sources, lazily or into one written file.

sources is a list or tuple mixing file paths and already-open readers or datasets; all must share one schema. Paths are opened (and owned by the result); readers and datasets are borrowed and left open. A path string may be a glob (e.g. "run_*.cstore"), expanded to its matches in numeric order.

Parameters:

  • sources (sequence of path or reader or dataset) – The inputs, combined in the given order.

  • out (str or os.PathLike, optional) – Destination. If None (the default), returns a lazy ColStoreDataset spanning the sources without copying – equivalent to open() of the same list. If given, streams the combined data to a new .cstore at out in bounded memory and returns a ColStoreReader opened on it; out must be a new path, not one of the sources.

  • memory_budget (int, optional) – Peak bytes for the streaming write (out given only); None uses the configured default.

  • **reader_kwargs – Forwarded to ColStoreReader when opening any source paths.

Returns:

ColStoreDataset or ColStoreReader – The lazy dataset when out is None; otherwise a reader on the written file.

Return type:

ColStoreReader | ColStoreDataset