colstore.append

colstore.append(directory, data, *, name='shard_{index:05d}.cstore', statistics=False, batch_size='auto', show_progress=False)[source]

Append data to the dataset at directory as one new shard.

The directory is created if needed and its .cstore shards form the dataset (read with colstore.open(directory)). data is anything colstore.store() accepts – a {name: array} dict, a structured array, or a DataFrame – or an open reader / dataset, or a path to a .cstore file (or directory of them). It must match the existing shards’ schema. The shard is named from name (a template whose {index} is the next free shard number, or a literal filename; it must end in .cstore) and committed atomically. Returns the new shard’s path. statistics=True records per-record statistics in the shard (see colstore.store()).

A colstore source (path, reader, or dataset) is written without materializing it into memory: its columns are streamed into the shard in bounded memory, so a file far larger than RAM can be appended. statistics=True uses the materializing path (the streaming writer records no footer).

Raises OSError if another writer holds the directory lock, and ValueError if name lacks the .cstore extension, data does not match the existing schema, or data is the dataset’s own directory (a self-append would duplicate every existing row).

Parameters:
Return type:

Path