colstore.compact

colstore.compact(path, *, out=None, show_progress=True)[source]

Collapse a multi-record file into a single-record file.

Streamed writers produce one record per write() call, and unsorted-fancy reads degrade as records accumulate. Compaction concatenates every record’s column bytes into one contiguous block per column, after which all reads take the single-record fast path. The byte splice runs in bounded memory regardless of file size (os.sendfile on Linux, shutil.copyfileobj elsewhere; see colstore.compaction), so files much larger than RAM compact fine.

Parameters:

  • path (str or os.PathLike) – Source file. Must be a valid colstore file.

  • out (str or os.PathLike, optional) – Destination. If None (the default), compaction is done in-place via a sibling temp file and an atomic os.replace(); the original is overwritten on success and untouched on failure. If given and different from path, the compacted result is written there and the original is left as-is. If equal to path, behaves as in-place.

  • show_progress (bool, default True) – Whether to display a tqdm progress bar during the copy.

Returns:

pathlib.Path – The path the compacted file was written to (path for in-place; out for out-of-place).

Return type:

Path

Notes

Already-compact files (n_records <= 1) are a no-op when out is None or the same file, and a byte-for-byte copy when out points elsewhere. An advisory lock is held on the source for the duration (see colstore._lock): a concurrent colstore.update() writer is blocked, while readers are unaffected (they don’t take the lock, and on POSIX they continue reading the unlinked inode after the rename).