colstore.ColStoreFrame

class colstore.ColStoreFrame(store, columns=None, rows=None, predicate=None)[source]

Bases: object

A deferred editing view over an opened store’s columns.

Created by edit() on an opened reader or dataset. A frame holds an ordered mapping of output column name to an expression (Expr); opening one seeds it with a native-passthrough leaf per source column. Indexing by name returns the expression for a column, which composes with operators and elementwise NumPy ufuncs to build transformations; a frame does not slice or index rows or columns (that is the reader’s role). The edit methods (assign / with_columns / drop / rename / astype / select) return a new frame by default – pass inplace=True to edit this one – so edits branch cheaply off a shared base; frame[name] = ... and del always edit in place. Nothing is read until you materialize: array() / dict() / recarray() evaluate columns into memory, and write() streams the result to a new file and returns a reader for it. The source store is never modified.

A frame may also carry a row selection – a concrete index set (e.g. from ds[idx].edit()) and/or pending where() predicates – applied to every column alike when the frame is materialized.

Assignment adds a column as an expression over the frame’s columns, which co-filters with any selection. An external array may be attached only on an unfiltered frame, where its length is checked against the source row count; it is held by reference unless copy=True is passed to assign().

Parameters:

  • store (_ReaderBase)

  • columns (list[str] | None)

  • rows (NDArray[Any] | None)

  • predicate (_Expr | None)

property n_rows: int

the source count, or the size of a concrete selection (e.g. from ds[idx].edit()).

A pending where() predicate is evaluated on access – an O(n) scan – so the read is cheap only when the selection is already concrete or absent.

Type:

Number of rows the frame selects

report(show=None)[source]

Cutflow of the where / filter pipeline: per-cut survivor counts.

An action that evaluates each cut in order and returns a CutflowReport of how many rows enter and pass it (named by where()’s label, else by position), plus the summed weight over those rows when a cut carries a where weight. show picks what the report prints – "raw", "weighted", or "both" (the default shows both when any weight is in effect, else raw). A frame with no cuts gives an empty report; a concrete base selection (ds[idx].edit()) is the first cut’s entering count.

Parameters:

show (str | None)

Return type:

CutflowReport

property columns: list[str]

Output column names, in order.

__getitem__(key)[source]

Column key as a FrameColumn – its expression bound to this frame.

It composes like a lazy expression (frame["a"] * 2, frame.assign(x= frame["a"] + frame["b"])) and, because it knows its owning frame, also offers pandas-style terminals over the frame’s current row selection: frame["a"].mean() / .sum() / .array() / np.asarray(frame["a"]). A frame still does not slice or index rows or columns – filter with where(), project columns with select(), and for positional or fancy indexing write() the frame and index the returned reader, where the data is contiguous. (The reader / dataset is for viewing; the frame is for editing.)

Parameters:

key (Any)

Return type:

FrameColumn

apply(func, *cols, out_dtype=None)[source]

Derive a column by running func on the named columns’ arrays.

func receives each of cols – column names, col() expressions, or frame expressions – as a NumPy array (the whole batch) and returns a 1-D array of the same length: the escape hatch for a column that cannot be written as a NumPy expression over cf[...]. Any NumPy is allowed (the function is run, not traced). The result dtype is inferred by running func once on empty inputs (reads no store data, but does execute func, so an effectful one should be pure) – pass out_dtype when func cannot run on a zero-length input, when its output dtype depends on the data, or to skip that build-time call. The returned expression composes and is attached with cf[name] = ... or assign().

Because batching only bounds memory, each batch is the whole array to func; a function that mixes rows (a running total, a sort) sees each batch on its own, while elementwise (per-row) functions – the common case – are unaffected.

Parameters:
Return type:

Expr

copy()[source]

An independent copy of the frame – a cheap branch point.

The expression nodes are immutable and shared; only the column mapping is duplicated, so editing the copy never affects this frame. This is the default result of every edit method below (see inplace).

Return type:

ColStoreFrame

assign(*, copy=False, inplace=False, **new_columns)[source]

Add or replace columns from keyword arguments.

Returns a new frame, leaving this one unchanged, unless inplace=True. Each value may be an expression, array, or scalar; arrays are held by reference unless copy=True. (Columns literally named copy or inplace must be set with frame[name] = ... instead.)

Parameters:
Return type:

ColStoreFrame

with_columns(*, copy=False, inplace=False, **new_columns)[source]

Add or replace columns from keyword arguments; an alias for assign().

Parameters:
Return type:

ColStoreFrame

drop(*names, inplace=False)[source]

Remove one or more columns.

Returns a new frame unless inplace=True. Raises KeyError for a name that is not in the frame, before changing anything.

Parameters:
Return type:

ColStoreFrame

rename(columns, inplace=False)[source]

Rename columns, resolving all mappings simultaneously.

Returns a new frame unless inplace=True. A swap such as {"a": "b", "b": "a"} exchanges the two names in one step. Renames change output names only – the underlying data, and any expression already referencing a column, are unaffected. Raises if a source name is missing or the result would contain duplicate names.

Parameters:
Return type:

ColStoreFrame

astype(dtypes, inplace=False)[source]

Cast the named columns to new dtypes (deferred).

Returns a new frame unless inplace=True. Each value is anything numpy.dtype accepts; the cast is lazy – evaluated by array / dict / recarray / write. Raises KeyError for an unknown name and validates every dtype before changing anything.

Parameters:
Return type:

ColStoreFrame

select(*names, inplace=False)[source]

Project the frame to names in the given order, dropping the rest.

Returns a new frame by default; pass inplace=True to edit this one. An unknown name raises KeyError and a repeated name ValueError. Only the column mapping is narrowed – no data is read, and the row selection is preserved.

Parameters:
Return type:

ColStoreFrame

where(predicate, label=None, *, weight=None, params=None, inplace=False)[source]

Keep the rows where predicate is true; returns a new frame by default.

Lazy: the predicate – a col() expression or a query string (the grammar of query()) – is validated up front (an unknown column or a non-boolean condition raises QueryError, reading no data) but only evaluated when the frame is materialized. Names resolve against the frame’s columns in order, so a column derived earlier is a valid target and a dropped or renamed-away one is not. Successive where calls compose (AND); reading n_rows, like any terminal, resolves them.

Pass label to name this cut in the report() cutflow, and weight (a column name or expression) to sum a per-row weight entering and passing the cut in the weighted cutflow. A weight is sticky – it carries to later cuts until another is given. filter is an alias. Pass inplace=True to edit this frame.

Parameters:

  • predicate (str | _Expr)

  • label (str | None)

  • weight (str | _Expr | Expr | None)

  • params (dict[str, Any] | None)

  • inplace (bool)

Return type:

ColStoreFrame

filter(predicate, label=None, *, weight=None, params=None, inplace=False)[source]

Alias of where() – keep the rows where predicate is true.

Parameters:

  • predicate (str | _Expr)

  • label (str | None)

  • weight (str | _Expr | Expr | None)

  • params (dict[str, Any] | None)

  • inplace (bool)

Return type:

ColStoreFrame

array(name)[source]

Materialize one column as a 1-D array over the selected rows; writes no file.

The single-column counterpart of dict() / recarray() – resolves any pending where() predicate, then evaluates column name over the selected rows. Equivalent to frame[name].array().

Parameters:

name (str)

Return type:

NDArray[Any]

dict()[source]

Compute every column into memory as a name -> array mapping; writes no file.

Resolves any pending where() predicate, then evaluates each column over the selected rows (one shared memo, so a subexpression used by several columns runs once). The in-memory analogue of write().

Return type:

dict[str, NDArray[Any]]

recarray()[source]

Compute every column into one structured (record) ndarray; writes no file.

Columns are interleaved into the record layout by the same parallel interleave_records kernel the reader’s recarray() uses.

Return type:

NDArray[Any]

frame()[source]

Compute every column into a pandas DataFrame; writes no file.

The pandas analogue of dict() / recarray(): resolves any pending where() predicate, evaluates each column over the selected rows, and assembles them into a DataFrame in column order with the computed dtypes. Requires pandas.

Return type:

pd.DataFrame

count()[source]

Number of selected rows (resolves any pending where()).

Return type:

int

sum(column)[source]

Sum of column over the selected rows – a full pass, returning a scalar.

column is a name, a col() expression, or a frame expression. The column is streamed in bounded-memory batches and the per-batch sums added. Integer sums widen to int64; floating sums accumulate in float64 for a stable, batch-size-independent result (so a float32 column sums to float64). A NaN in the data propagates to the result (NumPy semantics, not NaN-skipping); an empty selection sums to zero.

Parameters:

column (Any)

Return type:

Any

mean(column)[source]

Mean of column over the selected rows – a full pass, returning a float.

Streams the per-batch sum (a float64 accumulator) and the row count, dividing once at the end. A NaN in the data propagates; an empty selection gives nan.

Parameters:

column (Any)

Return type:

Any

min(column)[source]

Minimum of column over the selected rows – a full pass.

A NaN in the data propagates to the result (NumPy semantics); an empty selection returns a float nan regardless of the column’s dtype.

Parameters:

column (Any)

Return type:

Any

max(column)[source]

Maximum of column over the selected rows – a full pass.

A NaN in the data propagates to the result (NumPy semantics); an empty selection returns a float nan regardless of the column’s dtype.

Parameters:

column (Any)

Return type:

Any

var(column, ddof=0)[source]

Variance of column over the selected rows – a full bounded-memory pass.

Uses a numerically stable one-pass variance in float64: each batch’s mean and squared deviations are combined with Chan’s parallel update, so it never forms the cancellation-prone “mean of squares minus square of mean”. ddof is the delta degrees of freedom – ddof=0 the population variance, ddof=1 the sample variance. A NaN in the data propagates; a selection with ddof or fewer rows gives nan.

Parameters:
Return type:

float

std(column, ddof=0)[source]

Standard deviation of column over the selected rows – sqrt of var().

Same stable one-pass computation and ddof convention as var(). A NaN propagates; a selection with ddof or fewer rows gives nan.

Parameters:
Return type:

float

iter_batches(batch_size=None, *, copy=True)[source]

Yield the selected rows as materialized frames, bounded in memory.

The streaming counterpart of recarray(): the selection is resolved once, then each batch of the selected rows is evaluated with a fresh memo into a materialized, in-memory ColStoreFrame – its columns are concrete arrays detached from the source store – so peak memory is one batch rather than the whole frame, and the caller picks the format (dict() / recarray() / write(), or further edits) instead of a fixed array type. batch_size sizes each batch – int rows, str an IEC memory budget per batch (e.g. "256 MiB") converted from the per-row byte size, None the configured default budget. A frame with no columns or no selected rows yields nothing.

copy=False is a fast path for read-only streaming consumers that finish with each batch before drawing the next (accumulating a reduction, writing each batch out, feeding a model): a batch column is a read-only zero-copy view of the source where available (a stored column over a contiguous range of a single-record native store), or a gather into a per-column buffer reused across batches (an interleaved multi-record column, or a filtered/fancy selection) so the gather allocates nothing per batch; a transform is freshly computed. Because the views and buffers are reused, a batch is valid only until the next is drawn – do not mutate or hold it. Keep the default copy=True for owning arrays safe to mutate or hold after the store closes.

Parameters:
Return type:

Iterator[ColStoreFrame]

write(path, *, memory_budget=None)[source]

Stream the edited columns to a new .cstore and return a reader.

Evaluates every column one batch at a time into the new file (see colstore.format.write_dataset_streaming()); peak memory is bounded by memory_budget (bytes; None uses the configured default). A frame carrying a selection (an index set, or a resolved where()) streams its selected rows the same way – each batch gathers the selected source rows rather than a contiguous range. The source store is not modified; writing a frame with no columns is an error.

Parameters:
Return type:

ColStoreReader