colstore.convert

colstore.convert(source, dest=None, *, format=None, columns=None, dtypes=None, batch_size=None, compact=True, rename=None, output_dir=None, overwrite=False, on_mismatch='strict', max_workers=None, **kwargs)[source]

Convert files between colstore’s format and another, one endpoint being .cstore.

source is a single path, a glob, or a list of them; direction is inferred from the extensions (one endpoint must be .cstore, or ValueError is raised):

  • foreign -> .cstore (import) returns the opened ColStoreReader; .cstore -> foreign (export) returns the output Path; .cstore -> .cstore copies (or, across many inputs, merges) and returns a reader.

dest selects how the outputs are written:

  • omitted – each input is converted one-to-one, auto-named by swapping its extension (convert("data.h5") -> data.cstore; convert("*.h5") -> one .cstore per file).

  • a literal path – every input is merged into that one file (convert("*.h5", "all.cstore")).

  • a template with {index} / {stem} / {name} / {parent} – one-to-one with custom names (convert("*.h5", "run_{index}.cstore")).

rename overrides the output name per input (one-to-one): a callable stem -> new_stem or a mapping {stem: new_stem} (a stem absent from the mapping keeps its name). output_dir writes the outputs into that directory. overwrite replaces existing outputs (off by default, so an existing output raises FileExistsError). on_mismatch reconciles schemas when merging (see open()). format overrides the foreign endpoint’s format, and dtypes (import only) coerces columns as they are read (see open() for the rules). columns converts only the named columns, in either direction (projected as the foreign file is read on import, or selected from the store on export).

batch_size streams the conversion in bounded memory in either direction – an int row count or a "256 MiB" per-batch byte budget (peak memory is a few times the budget, from decode and conversion buffers, not an exact ceiling); the default None works whole-file. On import it reads the foreign file in row-batches when its schema is fixed-width numeric / temporal with no nulls (ROOT, Parquet, Feather, HDF5); a file that cannot stream stably (a variable-width string or null column, a pandas fixed-format HDF5, or JSON / NPZ) falls back to a whole-file read with a warning. On export it writes the foreign file in row-batches (Parquet row groups, Feather record batches, resizable HDF5 datasets, a ROOT Snapshot, or a .cstore editing-frame write); JSON / NPZ, the pandas HDF5 backend, and Feather with write options have no appendable path and are written whole with a warning. compact (default True, import only) collapses the streamed multi-record .cstore into a single record; pass False to keep it multi-record and skip the rewrite.

max_workers converts multiple files concurrently (across a glob / list one-to-one, or a merge’s per-file imports): None (default) is sequential, an int is a thread-pool worker count, and "auto" is min(config.get_convert_auto_workers(), n_files) (the per-host throughput plateau, default 8). Threads parallelize the per-file work (the heavy stages release the GIL), and they share one pyarrow pool so they do not oversubscribe. Peak memory scales with the worker count – each worker holds one file’s working set – so combine max_workers with batch_size to bound the total (roughly max_workers times the per-file peak). A single-file conversion ignores it (nothing to parallelize).

Returns a single result for a single source path or a merge, and a list (one per input) for a glob or list converted one-to-one.

Parameters:
Return type:

ColStoreReader | Path | list[ColStoreReader | Path]