Skip to content

Model Selection

Position-based splitting for ordered observations. These utilities work with forecasting, temporal classification, ordered regression, or any workflow where future rows must not train a model evaluated on earlier rows.

Ordering Is Explicit

Temporal splitters never guess a time column and never reorder data. Establish the canonical order first, then use the same positional membership for features, targets, weights, identities, and reporting axes.

ordered = data.sort_values("event_time", kind="mergesort").reset_index(drop=True)
origins = ordered["event_time"].tolist()

For forecasting data, ForecastFrame already establishes stable entity/time order. ForecastBacktester synchronizes panel rows by unique origin before it calls a splitter.

Final Ordered Holdout

from bitbullet.model_selection import ordered_holdout_split

split = ordered_holdout_split(
    ordered,
    test_size=0.2,
    gap=7,
    min_train_size=30,
    origins=origins,
)

X_train, X_gap, X_test = split.partition(X)
y_train, y_gap, y_test = split.partition(y)
print(split.metadata.to_dict())

The holdout is always the final test_size positions. An integer is an exact row count; a fraction is rounded up. gap positions immediately before the holdout belong to neither training nor test. max_train_size can retain only the most recent pre-gap history.

Expanding Windows

from bitbullet.model_selection import ExpandingWindowSplitter

splitter = ExpandingWindowSplitter(
    n_splits=4,
    test_size=14,
    gap=7,
    min_train_size=60,
    step_size=14,
)

for split in splitter.split_with_metadata(ordered, origins=origins):
    X_train, X_gap, X_valid = split.partition(X)
    y_train, y_gap, y_valid = split.partition(y)

An expanding window begins at position zero unless max_train_size is set. Validation windows are anchored so the final split ends at the final row. step_size controls the distance between validation starts; it defaults to the validation size.

Rolling Windows

from bitbullet.model_selection import RollingWindowSplitter

splitter = RollingWindowSplitter(
    max_train_size=180,
    n_splits=4,
    test_size=14,
    gap=7,
    step_size=14,
)

A rolling window caps retained training history. By default its minimum training size equals max_train_size, producing fixed-width windows. Set a smaller min_train_size to permit a shorter earliest window.

Gap Semantics

A temporal gap removes positions immediately before each validation boundary. It is useful when labels mature after their feature row, when windows overlap, or when operational latency means recent examples would not yet be trainable.

The splitter treats the gap as a row count. It does not infer cadence, target lead, embargo length, or label availability. Choose the gap from the data construction and prediction contract. Forecast backtesting adds a second safety check by removing framed samples whose target time is later than the validation boundary.

Shared Positional Membership

TemporalSplit.partition_many() applies one immutable membership definition to multiple aligned objects:

(X_parts, y_parts, weight_parts, id_parts) = split.partition_many(
    X,
    y,
    sample_weight,
    row_identity,
)

Each result is (train, gap, test). Pandas inputs retain index and type; NumPy arrays are taken along axis zero; lists and tuples retain their container type.

For scikit-learn-compatible consumers, split() and as_cv_indices() return only (train_indices, validation_indices). Gap membership remains available on the full TemporalSplit.

Auditable Metadata

All boundaries are half-open. For example, train_start=2 and train_end=7 describe positions [2, 7).

SplitMetadata records:

  • train, gap, and test boundaries and sizes;
  • discarded prefix and unused suffix counts;
  • validation origin position;
  • optional first and last origin values for every partition.

Origin values enrich metadata only. They never change membership or trigger a sort.

metadata = splitter.get_split_metadata(ordered, origins=origins)
for fold in metadata:
    print(fold.split_number, fold.test_origin_start, fold.test_origin_end)

API Reference

bitbullet.model_selection.temporal.ordered_holdout_split(data_or_count, *, test_size, gap=0, min_train_size=1, max_train_size=None, origins=None)

Create one final ordered holdout with an optional preceding gap.

Parameters:

Name Type Description Default
data_or_count Any

Any sized object, or its positive integer row count.

 required
test_size SizeLike

Positive row count or fraction in (0, 1). Fractional sizes round up, matching common held-out test-size semantics.

 required
gap int

Number of positions excluded immediately before the holdout.

 0
min_train_size int

Minimum retained training positions.

 1
max_train_size Optional[int]

Optional cap retaining only the most recent training positions before the gap.

 None
origins Optional[Sequence[Any]]

Optional values aligned one-to-one with rows. They enrich metadata but never reorder data.

 None

bitbullet.model_selection.temporal.ExpandingWindowSplitter

Bases: _BaseWindowSplitter

Expanding temporal validation windows with optional history cap.

Validation windows are anchored so the final split ends at the final input row. Earlier validation starts are separated by step_size (defaulting to the resolved validation size). With max_train_size=None, every split begins training at position zero.

bitbullet.model_selection.temporal.RollingWindowSplitter

Bases: _BaseWindowSplitter

Fixed/restricted-history temporal validation windows.

max_train_size is required and caps every training window. By default min_train_size equals that cap, producing fixed-width training windows. A smaller explicit minimum permits a shorter earliest window.

bitbullet.model_selection.temporal.TemporalSplit dataclass

Positional train/gap/test indices plus their exact metadata.

as_cv_indices()

Return the sklearn-style (train, validation) pair.

Gap positions remain excluded from both arrays and are available through :attr:gap_indices and :attr:metadata.

partition(values)

Apply this split to one positionally aligned object.

Pandas inputs retain their index and type, NumPy arrays are taken along axis zero, and lists/tuples retain their container type.

partition_many(*values)

Apply the same positional membership to multiple aligned objects.

bitbullet.model_selection.temporal.SplitMetadata dataclass

Exact boundaries and optional origin values for one temporal split.

origin_position is the first validation/test position. When callers supply origins, origin_value is the value at that same position. The module records origins for audit and reporting only; origin values never alter membership or trigger an implicit sort.

prefix_discarded property

Rows before a capped/rolling training window.

suffix_unused property

Rows after this split's validation window.

validation_end property

Alias for the exclusive test/validation end.

validation_start property

Alias for the first test/validation position.

to_dict()

Return flat, JSON-oriented boundary metadata.

Date-like origin values are intentionally retained as their original objects so callers can choose their own ISO/string serialization policy.