BitBullet:Lessons: Forecasting¶
Forecasting is regression with a deadline. The model may only use information that existed at the moment the forecast was made, the evaluation must respect the order of time, and the answer is judged against a future that had not happened yet. Break any of those and you get a model that scores beautifully and fails in production.
This lesson builds one complete forecasting workflow on real retail data: 942 days of daily sales from a European drugstore chain, including the closures, promotions and holidays that make real series awkward.
What You Will Build¶
| Step | Concept |
|---|---|
| 1-2 | Load a multi-store daily panel and understand its structure |
| 3 | Declare an explicit forecast contract and validate cadence |
| 4 | One series on its calendar grid: period inference, a horizon that never crosses a hole, splits in periods, and a baseline to beat |
| 5 | Decide what to do about closed days |
| 6 | Establish a seasonal baseline worth beating |
| 7 | Build features that cannot see the future |
| 8 | Direct multi-horizon forecasting |
| 9 | Gap-aware rolling-origin backtesting |
| 10 | Read error by horizon, store and target |
| 11 | Compare recursive against direct |
| 12 | Global versus local panel models |
| 13 | Joint multi-target, multi-horizon output |
| 14 | Conformal prediction intervals with honest coverage |
| 15 | Reconcile a three-level store hierarchy |
| 16 | Save, load and replay a fitted forecaster |
| 17 | Optional native statistical models |
Dataset: Rossmann Store Sales
Source: Kaggle - Rossmann Store Sales
Files: train.csv, store.csv, test.csv
Before running this notebook, download the three files above and place them next to it, or point
DATA_DIRat the folder holding them. The competition requires a signed-in Kaggle account and acceptance of its rules.
Configure a guided forecasting lifecycle instead of writing the surrounding orchestration. BitBullet Platform centralises datasets, managed compute, storage, configurations, and results in one governed environment. Set up forecasting experiments with chronological validation and candidate models through guided controls or AI-assisted draft preparation, review completed diagnostics, then export the fitted result with preprocessing, metadata, and generated inference code. This notebook teaches the underlying workflow directly with the BitBullet SDK.
1. Environment and Imports¶
import os
import sys
from pathlib import Path
from tempfile import TemporaryDirectory
import numpy as np
import pandas as pd
from sklearn.linear_model import Ridge
# Prefer the local SDK when this notebook runs from the lessons directory.
sdk_root = os.path.abspath('..')
if sdk_root not in sys.path:
sys.path.insert(0, sdk_root)
from bitbullet.forecast import (
ConformalIntervalCalibrator,
ForecastBacktester,
ForecastConfig,
ForecastFeatureBuilder,
ForecastFrame,
ForecastHorizon,
ForecastSchema,
HierarchicalReconciler,
Hierarchy,
RollingFeature,
SeasonalNaiveForecaster,
TabularForecaster,
evaluate_interval_coverage,
)
from bitbullet.model_selection import ExpandingWindowSplitter
pd.set_option('display.max_columns', 30)
print('Forecasting imports are ready.')
2. Load the Rossmann Store Panel¶
train.csv holds one row per store per day: Sales, Customers, whether the
store was Open, whether a Promo ran, and the school and state holiday flags.
store.csv adds attributes that do not change over time. test.csv is the
genuine future window - it carries the calendar and promotion plan for the 48
days after training ends, but no outcomes, because those had not happened.
Two design choices, both deliberate.
Six stores, not 1,115. The mechanics are identical at any width, and six keeps every cell fast. They are chosen deterministically: the three lowest-numbered stores of the two commonest store types, restricted to stores with a complete 942-day history and a presence in the future window. 181 of the 1,115 stores have month-long refurbishment gaps; excluding them here is a convenience, not a claim that irregular series can be ignored.
Two targets. Sales and Customers are both real outcomes, and neither
appears in test.csv precisely because neither is knowable in advance. That
makes them honest forecasting targets rather than features in disguise.
DATA_DIR = Path(os.environ.get('ROSSMANN_DIR', '.'))
STORES = [3, 7, 8, 15, 23, 29]
sales_raw = pd.read_csv(DATA_DIR / 'train.csv', low_memory=False, parse_dates=['Date'])
store_meta = pd.read_csv(DATA_DIR / 'store.csv', low_memory=False)
future_raw = pd.read_csv(DATA_DIR / 'test.csv', low_memory=False, parse_dates=['Date'])
STATIC = ['Store', 'StoreType', 'Assortment', 'CompetitionDistance']
def tidy(frame):
out = frame[frame['Store'].isin(STORES)].copy()
# StateHoliday mixes the integer 0 with the codes a/b/c. Collapse to a flag;
# the distinction between holiday types is not needed here.
out['is_state_holiday'] = (
out['StateHoliday'].astype(str).ne('0')
& out['StateHoliday'].astype(str).ne('none')
).astype(float)
out['store_id'] = 'store_' + out['Store'].astype(str)
return out.merge(store_meta[STATIC], on='Store', how='left')
PANEL = ['store_id', 'Date', 'Sales', 'Customers', 'Open', 'Promo',
'SchoolHoliday', 'is_state_holiday', 'DayOfWeek', 'CompetitionDistance']
history_df = (
tidy(sales_raw)[PANEL].sort_values(['store_id', 'Date']).reset_index(drop=True)
)
future_df = (
tidy(future_raw)[[c for c in PANEL if c not in ('Sales', 'Customers')]]
.sort_values(['store_id', 'Date']).reset_index(drop=True)
)
print('History:', history_df.shape, '| Future inputs:', future_df.shape)
print('History window:', history_df.Date.min().date(), '->', history_df.Date.max().date())
print('Future window :', future_df.Date.min().date(), '->', future_df.Date.max().date())
display(history_df.head(3))
display(future_df.head(3))
3. Declare and Validate the Forecast Contract¶
ForecastSchema states which column is time, which are targets, which identify
a series, and - the part that matters most - which covariates will be known at
the forecast origin.
Open, Promo, SchoolHoliday, is_state_holiday and DayOfWeek are all
future covariates: the chain plans its promotions and knows the calendar in
advance, and test.csv proves it by carrying exactly these columns for dates
that had not occurred. CompetitionDistance is static - one value per store.
There are no historic-only covariates in this dataset. Everything measured after the fact is a target. That is worth noticing rather than glossing over: it is what makes recursive forecasting viable in section 11, and it is unusual. Where a dataset does contain them - a temperature actual, a realised stock level - they may only enter as lags, and a recursive strategy cannot use them at all.
schema = ForecastSchema(
time='Date',
targets=('Sales', 'Customers'),
entities='store_id',
static_covariates='CompetitionDistance',
future_covariates=('Open', 'Promo', 'SchoolHoliday', 'is_state_holiday', 'DayOfWeek'),
cadence='D',
)
history = ForecastFrame(history_df, schema)
print('Series:', len(history.groups))
print('Regular cadence:', history.cadence_diagnostics.is_regular)
display(pd.DataFrame(history.cadence_diagnostics.to_dict()['series']))
4. One Prepared Series on Its Calendar Grid¶
Section 3 validated the panel's cadence through its declared schema. The
building blocks in bitbullet.forecast.cadence make that validation concrete
for the other common shape — one prepared series: an observed target, a time
axis, and features an analyst has already built — and show mechanically what
a regular calendar buys and what a hole in it does, before the rest of the
lesson scales the same practice up to the panel. Horizons, withheld labels, holdouts and
folds are all counted in periods, and a row count only equals a period
count when consecutive rows are exactly one period apart. The grid makes that
true even when periods are missing.
Store 13 makes the point on real data: it closed for refurbishment for 184 days in 2014, so its daily axis has a hole a sixth the length of its history.
infer_periodscores candidate calendars by how much of each grid the observed dates cover. Under the default 90% coverage rule the store is refused as ambiguous — the honest answer, since a naive row-based shift would attach labels from after the closure to origins from before it. In an interactive analysis we can lower the threshold consciously and inspect what the grid then does; a production pipeline should keep the refusal.place_on_gridmaps every row to its slot on the daily calendar; empty slots are never rows.shift_on_gridattaches the labelHORIZONperiods ahead by slot, so an origin whose target slot is empty getsNaNrather than a value from the far side of the hole;lag_on_gridbuilds level features the same way.grid_holdout_splitandGridExpandingWindowSplitterdecide the holdout and the folds in periods and return ordinary positional memberships over rows — identical to the row splitters on a complete calendar, and injectable as a scikit-learncv.naive_forecasts_on_gridandnaive_scale_on_gridgive the naive and seasonal-naive forecasts for the same horizon and the MASE denominator, so the model is judged against the baseline it has to beat. The horizon here is three days: at a horizon equal to the season, seasonal naive is the last observed value, and the two baselines coincide.
These are the primitives to reach for when the input is a single prepared table rather than a panel. Section 6 establishes the panel's seasonal baseline and section 10 reads the panel's error the same way; the multi-horizon, interval and hierarchy sections that follow are this practice applied at larger scope.
from sklearn.model_selection import cross_val_score
from bitbullet.forecast import (
GridExpandingWindowSplitter,
grid_holdout_split,
infer_period,
lag_on_grid,
naive_forecasts_on_grid,
naive_scale_on_grid,
place_on_grid,
seasonal_period_for,
shift_on_grid,
)
HORIZON = 3
# Store 13 is not one of the six panel stores above; take it straight from the raw file.
store13 = (
sales_raw[sales_raw['Store'].eq(13)][['Date', 'Sales', 'Open', 'Promo']]
.sort_values('Date').reset_index(drop=True)
)
# 1. Period inference: the default coverage rule refuses the axis, and says why.
refused = infer_period(store13['Date'])
print('Default rule:', refused.status, '-', refused.reason)
# Lower the threshold deliberately to look at the grid the closure leaves behind.
inferred = infer_period(store13['Date'], min_coverage=0.75)
period = inferred.period
print(f'Relaxed rule: {inferred.status} - {period.alias}, coverage {period.coverage:.1%}, '
f'{period.missing_slots} missing periods')
# 2. Grid, horizon-shifted label, and level features looked up by slot.
grid = place_on_grid(store13['Date'], period)
sales = store13['Sales'].to_numpy(dtype=float)
y_ahead = shift_on_grid(sales, grid, HORIZON)
features = pd.DataFrame({
'sales_now': sales,
'sales_lag_7': lag_on_grid(sales, grid, 7),
'sales_lag_14': lag_on_grid(sales, grid, 14),
# Future-known covariates are looked up at the target time, exactly as the
# feature builder in section 7 will do for the panel.
'open_ahead': shift_on_grid(store13['Open'].to_numpy(dtype=float), grid, HORIZON),
'promo_ahead': shift_on_grid(store13['Promo'].to_numpy(dtype=float), grid, HORIZON),
})
labelled = ~np.isnan(y_ahead) & features.notna().all(axis=1).to_numpy()
print(f'Origins with a label {HORIZON} periods ahead: {labelled.sum()} of {len(store13)} '
f'({(~labelled).sum()} have no target on the calendar or no history)')
# 3. Holdout and folds decided in periods; rows follow the slot boundaries.
outer = grid_holdout_split(grid, test_size=0.2, gap=HORIZON - 1)
train_rows = outer.rows.train_indices[labelled[outer.rows.train_indices]]
test_rows = outer.rows.test_indices[labelled[outer.rows.test_indices]]
print(f'Holdout: final {outer.slot_metadata.test_size} periods, {HORIZON - 1} withheld before it; '
f'{len(train_rows)} training and {len(test_rows)} evaluation origins')
train_grid = place_on_grid(store13['Date'].iloc[train_rows], period)
folds = GridExpandingWindowSplitter(train_grid, n_splits=3, gap=HORIZON - 1)
X_train, y_train = features.iloc[train_rows], y_ahead[train_rows]
X_test, y_test = features.iloc[test_rows], y_ahead[test_rows]
fold_scores = -cross_val_score(Ridge(alpha=1.0), X_train, y_train, cv=folds,
scoring='neg_mean_absolute_error')
print('Fold MAE (periods withheld before every validation block):', np.round(fold_scores, 1))
display(pd.DataFrame([{
'fold': meta.split_number + 1,
'train_periods': meta.train_size, 'withheld_periods': meta.gap_size,
'validation_periods': meta.test_size,
'validation_start': meta.test_origin_start,
} for meta in folds.last_slot_metadata]))
# 4. The model against the baselines it has to beat, all at the same horizon.
model = Ridge(alpha=1.0).fit(X_train, y_train)
predicted = model.predict(X_test)
season = seasonal_period_for(period)
baselines = naive_forecasts_on_grid(sales, grid, horizon=HORIZON, seasonal_period=season)
scale = naive_scale_on_grid(sales[train_rows], train_grid)
def mae(values):
mask = np.isfinite(values)
return float(np.mean(np.abs(y_test[mask] - values[mask])))
report = pd.DataFrame({
'mae': [mae(predicted), mae(baselines['naive'][test_rows]), mae(baselines['seasonal_naive'][test_rows])],
}, index=['ridge', 'naive (last value)', f'seasonal naive ({season})'])
report['mase'] = report['mae'] / scale
report['skill_vs_seasonal_naive'] = 1 - report['mae'] / report.loc[f'seasonal naive ({season})', 'mae']
display(report.round(3))
5. Closed Days Are Structure, Not Noise¶
Roughly 17% of rows have Sales == 0, almost all of them Sundays. A forecaster
has three options and only one of them is honest here.
Dropping closed days would break the daily cadence that every lag and every
window depends on: lag_7 would silently stop meaning "one week ago". Imputing
them would invent revenue that never existed. Keeping them and giving the model
Open lets it learn the closure rule directly - which is exactly why Open
appears in the future window.
The cost is that error metrics average over days the store could not trade, which flatters them. Section 10 looks at error by store, where that shows up.
closed = history_df.Open.eq(0)
print(f'Closed rows: {closed.sum()} of {len(history_df)} ({closed.mean():.1%})')
print(f'Zero sales while open: {int((history_df.Sales.eq(0) & ~closed).sum())}')
by_dow = history_df.groupby('DayOfWeek').agg(
mean_sales=('Sales', 'mean'),
mean_customers=('Customers', 'mean'),
open_rate=('Open', 'mean'),
)
by_dow.index = ['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun']
display(by_dow.round(1))
promo_lift = history_df[history_df.Open.eq(1)].groupby('Promo').Sales.mean()
print(f'Mean sales while open - no promo {promo_lift.loc[0]:,.0f}, promo {promo_lift.loc[1]:,.0f} '
f'({promo_lift.loc[1] / promo_lift.loc[0] - 1:+.1%})')
6. Establish a Seasonal Baseline¶
Weekly rhythm dominates this series, so seasonal naive - "same weekday last week" - is a strong baseline and a fair one. It carries the closure pattern for free, since last Sunday was also closed. Any learned model that cannot beat it is not earning its complexity.
baseline = SeasonalNaiveForecaster(seasonal_period=7).fit(
history.data,
target_columns=history.schema.target_columns,
time_column=history.schema.time,
entity_columns=history.schema.entity_columns,
)
baseline_predictions = baseline.predict(ForecastHorizon.up_to(7))
print('Baseline rows:', len(baseline_predictions))
display(baseline_predictions.head(8))
7. Build Leakage-Conscious Features¶
Every feature must be computable at the origin. ForecastFeatureBuilder
enforces that by construction:
- target lags at 1, 7 and 14 days - yesterday, last week, the week before
- rolling means with
lag=1, so a 7-day window ends the day before the origin and never touches the day being predicted - calendar terms as sine/cosine pairs, which are pure functions of the date and therefore always knowable
Future covariates need no lag at all - Promo for next Tuesday is already
known. That asymmetry is the whole point of separating future from historic
covariates in the schema.
feature_builder = ForecastFeatureBuilder(
target_lags=(1, 7, 14),
rolling_features=(
RollingFeature('Sales', window=7, statistic='mean', lag=1),
RollingFeature('Customers', window=7, statistic='mean', lag=1),
),
calendar_features=('day_of_week_sin', 'day_of_week_cos', 'month_sin', 'month_cos'),
)
one_week = feature_builder.make_supervised(history, ForecastHorizon.up_to(7))
print('Supervised rows:', len(one_week))
print('Feature columns:', list(one_week.X.columns))
display(one_week.to_frame().head())
8. Direct Multi-Horizon Forecasting¶
A direct strategy fits one model per horizon and target. Day 1 and day 7 are different problems - by day 7 the most recent lag is a week stale - so letting each fit its own coefficients is usually stronger than asking one model to serve both.
Three horizons and two targets gives six models. panel_strategy='global' fits
across all six stores together, sharing what they have in common.
HORIZONS = (1, 3, 7)
def make_direct_forecaster():
return TabularForecaster(
Ridge(alpha=1.0),
config=ForecastConfig(
horizons=HORIZONS,
strategy='direct',
panel_strategy='global',
),
feature_builder=feature_builder,
)
direct_forecaster = make_direct_forecaster().fit(history)
direct_future = direct_forecaster.forecast(future_df)
print('Fitted models:', len(direct_forecaster.state.models))
display(direct_future.head(12))
9. Gap-Aware Rolling-Origin Backtesting¶
A single holdout gives one estimate of forecast error against one recent stretch, and recent stretches differ. Rolling-origin backtesting repeats the forecast at several historical cutoffs and reports the distribution instead.
The gap is the part most tooling omits. Suppose sales are consolidated weekly,
so at any origin the last seven days of outcomes are not yet available. Training
on them would use labels the forecaster could not have had. gap=7 withholds
exactly those rows from every fold - and note it is a label availability
question, not a feature availability one.
purged_label_samples reports how many rows each fold dropped for that reason.
REPORTING_GAP = 7
splitter = ExpandingWindowSplitter(
n_splits=3,
test_size=28,
gap=REPORTING_GAP,
min_train_size=365,
)
backtest = ForecastBacktester(splitter).run(
history,
forecaster_factory=make_direct_forecaster,
)
fold_summary = pd.DataFrame([{
'fold': fold.fold,
'train_origins': fold.split.train_size,
'gap_origins': fold.split.gap_size,
'validation_origins': fold.validation_origins,
'training_samples': fold.training_samples,
'unavailable_labels_removed': fold.purged_label_samples,
'validation_start': fold.split.test_origin_start,
'validation_end': fold.split.test_origin_end,
} for fold in backtest.folds])
display(fold_summary)
10. Read Overall and Grouped Metrics¶
One number hides the shape of the error. Four things in this output are worth more than the headline.
Error barely grows with horizon. Day 1 gives wape 14.2%, day 7 gives 14.9%. The common heuristic says forecast error should climb steeply with distance, and flat error should make you suspect leakage - but that heuristic assumes predictability comes from recent momentum. Here it does not. Weekly rhythm, promotions and the closure calendar dominate, and all of them are known exactly at every horizon. The lag features decay as the horizon lengthens; the calendar and promotion terms do not. When your signal lives in known-in-advance drivers, horizon costs you little. That is a property of the problem, not a bug.
sMAPE is roughly 45% while MAPE is roughly 12%. This is the cost of the
closed days from section 5 arriving on the invoice. Percentage errors divide by
the actual, and a Sunday actual is zero. sMAPE's symmetric denominator does not
rescue it. On any series with legitimate zeros, read wape - which divides by
the total rather than per row - and treat sMAPE as uninformative.
Stores differ by more than the horizon does. wape ranges from 12.7% to 17.8% across six stores, a wider spread than anything horizon produces. The weakest store also carries by far the largest negative bias. That is a modelling signal, not noise: it says a global model is leaving something store-specific on the table, which is exactly the question section 12 asks.
Compare scale-free metrics across targets, never rmse. Sales shows rmse
1245 against Customers at 104, purely because one is money and the other is
people. Their wape values, 15.0% and 13.1%, are comparable and tell you the
truthful story: the two targets are forecast about equally well.
bias is signed and negative almost everywhere here, meaning systematic
under-forecasting. In retail that means stockouts. mase and rmsse are absent
because both need an in-sample seasonal error scale to divide by, which this
report was not given.
print('Overall')
display(pd.Series(backtest.metrics.metrics, name='value').to_frame().round(3))
print('By horizon')
display(pd.DataFrame(backtest.metrics.grouped_metrics['horizon']).round(3))
print('By store')
display(pd.DataFrame(backtest.metrics.grouped_metrics['entity']).round(3))
print('By target')
display(pd.DataFrame(backtest.metrics.grouped_metrics['target']).round(3))
11. Compare Recursive Forecasting¶
A recursive strategy fits one model for a single step and applies it repeatedly, feeding each prediction back in as the next lag. It is compact, and it needs every feature to be available at every step - which holds here only because this dataset has no historic-only covariates.
Its weakness is error compounding: a day-1 mistake becomes an input to day 2. Compare the two strategies rather than assuming one wins.
recursive_forecaster = TabularForecaster(
Ridge(alpha=1.0),
config=ForecastConfig(horizons=HORIZONS, strategy='recursive'),
feature_builder=ForecastFeatureBuilder(
target_lags=(1, 7, 14),
calendar_features=('day_of_week_sin', 'day_of_week_cos'),
),
).fit(history)
recursive_future = recursive_forecaster.forecast(future_df)
comparison = direct_future.merge(
recursive_future,
on=['origin', 'time', 'horizon', 'entity', 'target'],
suffixes=('_direct', '_recursive'),
)
display(
comparison[comparison.target.eq('Sales')][
['entity', 'time', 'horizon', 'y_pred_direct', 'y_pred_recursive']
].round(1).head(12)
)
12. Global Versus Local Panel Models¶
A global model pools every store and learns one set of coefficients per horizon and target. A local model fits each store separately.
Global borrows strength - a quiet store benefits from patterns visible in busier ones - and is the better default when series are few or short. Local wins when stores genuinely behave differently. Here, six stores across two store types is exactly the regime where it is worth measuring rather than guessing.
global_one_step = TabularForecaster(
Ridge(alpha=1.0),
config=ForecastConfig(horizons=(1,), panel_strategy='global'),
feature_builder=feature_builder,
).fit(history)
local_one_step = TabularForecaster(
Ridge(alpha=1.0),
config=ForecastConfig(horizons=(1,), panel_strategy='local'),
feature_builder=feature_builder,
).fit(history)
print('Global model count:', len(global_one_step.state.models))
print('Local model count :', len(local_one_step.state.models))
13. Joint Multi-Target, Multi-Horizon Forecasting¶
Sales and Customers are not independent - footfall drives revenue. A
multioutput strategy fits one estimator that emits every target and horizon
together, which keeps their relationship intact instead of letting two separate
models drift apart.
Capability is checked before use: estimators that cannot emit multiple outputs are rejected rather than silently wrapped.
multioutput_forecaster = TabularForecaster(
Ridge(alpha=1.0),
config=ForecastConfig(
horizons=(1, 7),
strategy='multioutput',
panel_strategy='global',
),
feature_builder=feature_builder,
).fit(history)
joint_future = multioutput_forecaster.forecast(future_df, horizons=(1, 7))
print('Joint target/horizon outputs:', sorted(
set(zip(joint_future['target'], joint_future['horizon']))
))
display(joint_future.head(8))
14. Add Conformal Prediction Intervals¶
A point forecast says nothing about how wrong it might be. Conformal calibration turns held-out residuals into intervals with a target coverage level, without assuming a distribution and without retraining.
Grouping by ('horizon', 'target') matters: day-7 uncertainty is wider than
day-1, and Sales is on a different scale from Customers. One pooled interval
would be too wide for the easy cases and too narrow for the hard ones.
Calibrate on earlier folds, measure coverage on a later one. Coverage close to the nominal level is the result you want; well above it means intervals are wastefully wide, and below it means they are lying.
calibration_predictions = backtest.predictions[backtest.predictions['fold'].lt(2)]
interval_validation = backtest.predictions[backtest.predictions['fold'].eq(2)]
calibrator = ConformalIntervalCalibrator(
alpha=0.2, # target 80% coverage
group_by=('horizon', 'target'),
min_group_size=20,
).fit(calibration_predictions)
coverage_report = evaluate_interval_coverage(calibrator.transform(interval_validation))
future_intervals = calibrator.transform(direct_future)
print('Nominal coverage: 0.80')
display(pd.Series(coverage_report.metrics, name='value').to_frame().round(3))
display(
future_intervals[future_intervals.target.eq('Sales')][
['entity', 'time', 'horizon', 'y_pred', 'lower', 'upper']
].round(1).head(9)
)
15. Reconcile the Store Hierarchy¶
Forecasts are usually needed at several levels at once: per store for staffing, per store type for buying, chain-wide for finance. Forecast each level independently and the totals will not agree, which is indefensible when the same numbers appear in two reports.
Reconciliation enforces coherence. bottom_up sums the store forecasts into
their type and then the chain, so every parent equals the sum of its children by
construction.
store_type = store_meta.set_index('Store').StoreType.to_dict()
parent_map = {f'store_{s}': f'type_{store_type[s]}' for s in STORES}
parent_map.update({f'type_{t}': 'chain' for t in {store_type[s] for s in STORES}})
parent_map['chain'] = None
hierarchy = Hierarchy.from_parent_map(parent_map)
reconciler = HierarchicalReconciler(hierarchy, method='bottom_up')
bottom_sales = direct_future[
direct_future['target'].eq('Sales') & direct_future['horizon'].eq(1)
]
coherent_sales = reconciler.reconcile_frame(bottom_sales, actual_col=None)
print('Hierarchy levels: store -> store type -> chain')
print('Coherent:', hierarchy.is_coherent(coherent_sales['y_pred'].to_numpy()))
display(coherent_sales.round(1))
16. Save, Load, and Replay the Forecast¶
A forecaster is only useful if it produces identical output tomorrow. Saving captures the estimator, the fitted feature contract and the schema together, so a restored artifact reproduces predictions exactly rather than approximately.
The assertion below is the one that matters. If it ever fails, something stateful was left out of the artifact.
with TemporaryDirectory() as artifact_dir:
artifact_path = Path(artifact_dir) / 'rossmann_forecast.pkl'
direct_forecaster.save(artifact_path)
restored = TabularForecaster.load(artifact_path)
replayed = restored.forecast(future_df)
pd.testing.assert_frame_equal(direct_future, replayed)
print('Artifact replay is exactly equivalent.')
print(restored.to_model_metadata(name='rossmann_forecast').inference_contract)
17. Optional Native Statistical Models¶
Classical statistical models remain competitive on short, regular, univariate series, and are often the right answer for a single well-behaved series. The adapter is optional and imported lazily, so it costs nothing unless installed:
pip install "bitbullet[forecast-statistical]"
from bitbullet.forecast import StatsForecastAdapter
Reach for it when a series is univariate and seasonal. Prefer the tabular forecasters when covariates matter, when many series share structure, or when you need one model to serve a panel.
What You Built¶
A complete forecasting workflow on real retail data: an explicit contract with future covariates separated from outcomes, one series placed on its calendar grid with a horizon that respects every missing period, a deliberate decision about closed days, a seasonal baseline and the MASE to beat it, leakage-conscious features, direct and recursive multi-horizon strategies, global and local panel models, joint multi-target output, gap-aware rolling-origin evaluation, calibrated intervals with measured coverage, a coherent three-level hierarchy, and an artifact that replays exactly.
Forecasting Checklist¶
| Question | Where it was answered |
|---|---|
| Which columns are known at the origin? | Section 3, future versus static covariates |
| Does the cadence support lags and windows? | Section 3, cadence diagnostics |
| Does a horizon in rows mean a horizon in periods? | Section 4, period inference and the calendar grid |
| What do structural zeros mean? | Section 5 |
| What am I trying to beat? | Section 6, seasonal naive |
| Can any feature see the future? | Section 7, lagged windows and calendar terms |
| Does error grow with horizon? | Section 10, grouped metrics |
| Were unavailable labels withheld? | Section 9, gap and purged samples |
| How wrong might this be? | Section 14, measured coverage |
| Do the levels add up? | Section 15, reconciliation |
| Will it reproduce tomorrow? | Section 16, exact replay |
| Is the model earning its complexity? | Sections 4 and 6, seasonal naive and MASE |