{ "cells": [ { "cell_type": "markdown", "id": "ce28fa5b", "metadata": {}, "source": [ "# BitBullet:Lessons: Forecasting\n", "\n", "Forecasting is regression with a deadline. The model may only use information\n", "that existed at the moment the forecast was made, the evaluation must respect\n", "the order of time, and the answer is judged against a future that had not\n", "happened yet. Break any of those and you get a model that scores beautifully and\n", "fails in production.\n", "\n", "This lesson builds one complete forecasting workflow on real retail data: 942\n", "days of daily sales from a European drugstore chain, including the closures,\n", "promotions and holidays that make real series awkward.\n", "\n", "### What You Will Build\n", "\n", "| Step | Concept |\n", "| --- | --- |\n", "| 1-2 | Load a multi-store daily panel and understand its structure |\n", "| 3 | Declare an explicit forecast contract and validate cadence |\n", "| 4 | One series on its calendar grid: period inference, a horizon that never crosses a hole, splits in periods, and a baseline to beat |\n", "| 5 | Decide what to do about closed days |\n", "| 6 | Establish a seasonal baseline worth beating |\n", "| 7 | Build features that cannot see the future |\n", "| 8 | Direct multi-horizon forecasting |\n", "| 9 | Gap-aware rolling-origin backtesting |\n", "| 10 | Read error by horizon, store and target |\n", "| 11 | Compare recursive against direct |\n", "| 12 | Global versus local panel models |\n", "| 13 | Joint multi-target, multi-horizon output |\n", "| 14 | Conformal prediction intervals with honest coverage |\n", "| 15 | Reconcile a three-level store hierarchy |\n", "| 16 | Save, load and replay a fitted forecaster |\n", "| 17 | Optional native statistical models |\n", "\n", "**Dataset:** Rossmann Store Sales\n", "\n", "**Source:** [Kaggle - Rossmann Store Sales](https://www.kaggle.com/competitions/rossmann-store-sales/data)\n", "\n", "**Files:** `train.csv`, `store.csv`, `test.csv`\n", "\n", "> Before running this notebook, download the three files above and place them\n", "> next to it, or point `DATA_DIR` at the folder holding them. The competition\n", "> requires a signed-in Kaggle account and acceptance of its rules." ] }, { "cell_type": "markdown", "id": "4a84853e", "metadata": {}, "source": [ "> **Configure a guided forecasting lifecycle instead of writing the surrounding orchestration.**\n", "> [BitBullet Platform](https://bitbullet.co.uk/platform/regression) centralises datasets, managed compute, storage, configurations, and results in one governed environment. Set up forecasting experiments with chronological validation and candidate models through guided controls or AI-assisted draft preparation, review completed diagnostics, then export the fitted result with preprocessing, metadata, and generated inference code. This notebook teaches the underlying workflow directly with the BitBullet SDK." ] }, { "cell_type": "markdown", "id": "5720d975", "metadata": {}, "source": [ "## 1. Environment and Imports" ] }, { "cell_type": "code", "execution_count": null, "id": "4c07fed2", "metadata": {}, "outputs": [], "source": [ "import os\n", "import sys\n", "from pathlib import Path\n", "from tempfile import TemporaryDirectory\n", "\n", "import numpy as np\n", "import pandas as pd\n", "from sklearn.linear_model import Ridge\n", "\n", "# Prefer the local SDK when this notebook runs from the lessons directory.\n", "sdk_root = os.path.abspath('..')\n", "if sdk_root not in sys.path:\n", " sys.path.insert(0, sdk_root)\n", "\n", "from bitbullet.forecast import (\n", " ConformalIntervalCalibrator,\n", " ForecastBacktester,\n", " ForecastConfig,\n", " ForecastFeatureBuilder,\n", " ForecastFrame,\n", " ForecastHorizon,\n", " ForecastSchema,\n", " HierarchicalReconciler,\n", " Hierarchy,\n", " RollingFeature,\n", " SeasonalNaiveForecaster,\n", " TabularForecaster,\n", " evaluate_interval_coverage,\n", ")\n", "from bitbullet.model_selection import ExpandingWindowSplitter\n", "\n", "pd.set_option('display.max_columns', 30)\n", "print('Forecasting imports are ready.')" ] }, { "cell_type": "markdown", "id": "3f64cf3e", "metadata": {}, "source": [ "## 2. Load the Rossmann Store Panel\n", "\n", "`train.csv` holds one row per store per day: `Sales`, `Customers`, whether the\n", "store was `Open`, whether a `Promo` ran, and the school and state holiday flags.\n", "`store.csv` adds attributes that do not change over time. `test.csv` is the\n", "genuine future window - it carries the calendar and promotion plan for the 48\n", "days after training ends, but no outcomes, because those had not happened.\n", "\n", "Two design choices, both deliberate.\n", "\n", "**Six stores, not 1,115.** The mechanics are identical at any width, and six\n", "keeps every cell fast. They are chosen deterministically: the three\n", "lowest-numbered stores of the two commonest store types, restricted to stores\n", "with a complete 942-day history and a presence in the future window. 181 of the\n", "1,115 stores have month-long refurbishment gaps; excluding them here is a\n", "convenience, not a claim that irregular series can be ignored.\n", "\n", "**Two targets.** `Sales` and `Customers` are both real outcomes, and neither\n", "appears in `test.csv` precisely because neither is knowable in advance. That\n", "makes them honest forecasting targets rather than features in disguise." ] }, { "cell_type": "code", "execution_count": null, "id": "033c1220", "metadata": {}, "outputs": [], "source": [ "DATA_DIR = Path(os.environ.get('ROSSMANN_DIR', '.'))\n", "STORES = [3, 7, 8, 15, 23, 29]\n", "\n", "sales_raw = pd.read_csv(DATA_DIR / 'train.csv', low_memory=False, parse_dates=['Date'])\n", "store_meta = pd.read_csv(DATA_DIR / 'store.csv', low_memory=False)\n", "future_raw = pd.read_csv(DATA_DIR / 'test.csv', low_memory=False, parse_dates=['Date'])\n", "\n", "STATIC = ['Store', 'StoreType', 'Assortment', 'CompetitionDistance']\n", "\n", "def tidy(frame):\n", " out = frame[frame['Store'].isin(STORES)].copy()\n", " # StateHoliday mixes the integer 0 with the codes a/b/c. Collapse to a flag;\n", " # the distinction between holiday types is not needed here.\n", " out['is_state_holiday'] = (\n", " out['StateHoliday'].astype(str).ne('0')\n", " & out['StateHoliday'].astype(str).ne('none')\n", " ).astype(float)\n", " out['store_id'] = 'store_' + out['Store'].astype(str)\n", " return out.merge(store_meta[STATIC], on='Store', how='left')\n", "\n", "PANEL = ['store_id', 'Date', 'Sales', 'Customers', 'Open', 'Promo',\n", " 'SchoolHoliday', 'is_state_holiday', 'DayOfWeek', 'CompetitionDistance']\n", "\n", "history_df = (\n", " tidy(sales_raw)[PANEL].sort_values(['store_id', 'Date']).reset_index(drop=True)\n", ")\n", "future_df = (\n", " tidy(future_raw)[[c for c in PANEL if c not in ('Sales', 'Customers')]]\n", " .sort_values(['store_id', 'Date']).reset_index(drop=True)\n", ")\n", "\n", "print('History:', history_df.shape, '| Future inputs:', future_df.shape)\n", "print('History window:', history_df.Date.min().date(), '->', history_df.Date.max().date())\n", "print('Future window :', future_df.Date.min().date(), '->', future_df.Date.max().date())\n", "display(history_df.head(3))\n", "display(future_df.head(3))" ] }, { "cell_type": "markdown", "id": "a3b7352b", "metadata": {}, "source": [ "## 3. Declare and Validate the Forecast Contract\n", "\n", "`ForecastSchema` states which column is time, which are targets, which identify\n", "a series, and - the part that matters most - which covariates will be known at\n", "the forecast origin.\n", "\n", "`Open`, `Promo`, `SchoolHoliday`, `is_state_holiday` and `DayOfWeek` are all\n", "**future covariates**: the chain plans its promotions and knows the calendar in\n", "advance, and `test.csv` proves it by carrying exactly these columns for dates\n", "that had not occurred. `CompetitionDistance` is **static** - one value per store.\n", "\n", "There are no historic-only covariates in this dataset. Everything measured after\n", "the fact is a target. That is worth noticing rather than glossing over: it is\n", "what makes recursive forecasting viable in section 11, and it is unusual. Where\n", "a dataset does contain them - a temperature actual, a realised stock level - they\n", "may only enter as lags, and a recursive strategy cannot use them at all." ] }, { "cell_type": "code", "execution_count": null, "id": "182ce35a", "metadata": {}, "outputs": [], "source": [ "schema = ForecastSchema(\n", " time='Date',\n", " targets=('Sales', 'Customers'),\n", " entities='store_id',\n", " static_covariates='CompetitionDistance',\n", " future_covariates=('Open', 'Promo', 'SchoolHoliday', 'is_state_holiday', 'DayOfWeek'),\n", " cadence='D',\n", ")\n", "history = ForecastFrame(history_df, schema)\n", "\n", "print('Series:', len(history.groups))\n", "print('Regular cadence:', history.cadence_diagnostics.is_regular)\n", "display(pd.DataFrame(history.cadence_diagnostics.to_dict()['series']))" ] }, { "cell_type": "markdown", "id": "68478290", "metadata": {}, "source": [ "## 4. One Prepared Series on Its Calendar Grid\n", "\n", "Section 3 validated the panel's cadence through its declared schema. The\n", "building blocks in `bitbullet.forecast.cadence` make that validation concrete\n", "for the other common shape — one prepared series: an observed target, a time\n", "axis, and features an analyst has already built — and show mechanically what\n", "a regular calendar buys and what a hole in it does, before the rest of the\n", "lesson scales the same practice up to the panel. Horizons, withheld labels, holdouts and\n", "folds are all counted in **periods**, and a row count only equals a period\n", "count when consecutive rows are exactly one period apart. The grid makes that\n", "true even when periods are missing.\n", "\n", "Store 13 makes the point on real data: it closed for refurbishment for 184 days\n", "in 2014, so its daily axis has a hole a sixth the length of its history.\n", "\n", "- `infer_period` scores candidate calendars by how much of each grid the\n", " observed dates cover. Under the default 90% coverage rule the store is\n", " refused as *ambiguous* — the honest answer, since a naive row-based shift\n", " would attach labels from after the closure to origins from before it. In an\n", " interactive analysis we can lower the threshold consciously and inspect what\n", " the grid then does; a production pipeline should keep the refusal.\n", "- `place_on_grid` maps every row to its slot on the daily calendar; empty slots\n", " are never rows.\n", "- `shift_on_grid` attaches the label `HORIZON` periods ahead by *slot*, so an\n", " origin whose target slot is empty gets `NaN` rather than a value from the\n", " far side of the hole; `lag_on_grid` builds level features the same way.\n", "- `grid_holdout_split` and `GridExpandingWindowSplitter` decide the holdout and\n", " the folds in periods and return ordinary positional memberships over rows —\n", " identical to the row splitters on a complete calendar, and injectable as a\n", " scikit-learn `cv`.\n", "- `naive_forecasts_on_grid` and `naive_scale_on_grid` give the naive and\n", " seasonal-naive forecasts for the same horizon and the MASE denominator, so\n", " the model is judged against the baseline it has to beat. The horizon here is\n", " three days: at a horizon equal to the season, seasonal naive *is* the last\n", " observed value, and the two baselines coincide.\n", "\n", "These are the primitives to reach for when the input is a single prepared\n", "table rather than a panel. Section 6 establishes the panel's seasonal baseline\n", "and section 10 reads the panel's error the same way; the multi-horizon,\n", "interval and hierarchy sections that follow are this practice applied at\n", "larger scope.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "6ba5375d", "metadata": {}, "outputs": [], "source": [ "from sklearn.model_selection import cross_val_score\n", "\n", "from bitbullet.forecast import (\n", " GridExpandingWindowSplitter,\n", " grid_holdout_split,\n", " infer_period,\n", " lag_on_grid,\n", " naive_forecasts_on_grid,\n", " naive_scale_on_grid,\n", " place_on_grid,\n", " seasonal_period_for,\n", " shift_on_grid,\n", ")\n", "\n", "HORIZON = 3\n", "# Store 13 is not one of the six panel stores above; take it straight from the raw file.\n", "store13 = (\n", " sales_raw[sales_raw['Store'].eq(13)][['Date', 'Sales', 'Open', 'Promo']]\n", " .sort_values('Date').reset_index(drop=True)\n", ")\n", "\n", "# 1. Period inference: the default coverage rule refuses the axis, and says why.\n", "refused = infer_period(store13['Date'])\n", "print('Default rule:', refused.status, '-', refused.reason)\n", "\n", "# Lower the threshold deliberately to look at the grid the closure leaves behind.\n", "inferred = infer_period(store13['Date'], min_coverage=0.75)\n", "period = inferred.period\n", "print(f'Relaxed rule: {inferred.status} - {period.alias}, coverage {period.coverage:.1%}, '\n", " f'{period.missing_slots} missing periods')\n", "\n", "# 2. Grid, horizon-shifted label, and level features looked up by slot.\n", "grid = place_on_grid(store13['Date'], period)\n", "sales = store13['Sales'].to_numpy(dtype=float)\n", "y_ahead = shift_on_grid(sales, grid, HORIZON)\n", "features = pd.DataFrame({\n", " 'sales_now': sales,\n", " 'sales_lag_7': lag_on_grid(sales, grid, 7),\n", " 'sales_lag_14': lag_on_grid(sales, grid, 14),\n", " # Future-known covariates are looked up at the target time, exactly as the\n", " # feature builder in section 7 will do for the panel.\n", " 'open_ahead': shift_on_grid(store13['Open'].to_numpy(dtype=float), grid, HORIZON),\n", " 'promo_ahead': shift_on_grid(store13['Promo'].to_numpy(dtype=float), grid, HORIZON),\n", "})\n", "labelled = ~np.isnan(y_ahead) & features.notna().all(axis=1).to_numpy()\n", "print(f'Origins with a label {HORIZON} periods ahead: {labelled.sum()} of {len(store13)} '\n", " f'({(~labelled).sum()} have no target on the calendar or no history)')\n", "\n", "# 3. Holdout and folds decided in periods; rows follow the slot boundaries.\n", "outer = grid_holdout_split(grid, test_size=0.2, gap=HORIZON - 1)\n", "train_rows = outer.rows.train_indices[labelled[outer.rows.train_indices]]\n", "test_rows = outer.rows.test_indices[labelled[outer.rows.test_indices]]\n", "print(f'Holdout: final {outer.slot_metadata.test_size} periods, {HORIZON - 1} withheld before it; '\n", " f'{len(train_rows)} training and {len(test_rows)} evaluation origins')\n", "\n", "train_grid = place_on_grid(store13['Date'].iloc[train_rows], period)\n", "folds = GridExpandingWindowSplitter(train_grid, n_splits=3, gap=HORIZON - 1)\n", "X_train, y_train = features.iloc[train_rows], y_ahead[train_rows]\n", "X_test, y_test = features.iloc[test_rows], y_ahead[test_rows]\n", "fold_scores = -cross_val_score(Ridge(alpha=1.0), X_train, y_train, cv=folds,\n", " scoring='neg_mean_absolute_error')\n", "print('Fold MAE (periods withheld before every validation block):', np.round(fold_scores, 1))\n", "display(pd.DataFrame([{\n", " 'fold': meta.split_number + 1,\n", " 'train_periods': meta.train_size, 'withheld_periods': meta.gap_size,\n", " 'validation_periods': meta.test_size,\n", " 'validation_start': meta.test_origin_start,\n", "} for meta in folds.last_slot_metadata]))\n", "\n", "# 4. The model against the baselines it has to beat, all at the same horizon.\n", "model = Ridge(alpha=1.0).fit(X_train, y_train)\n", "predicted = model.predict(X_test)\n", "season = seasonal_period_for(period)\n", "baselines = naive_forecasts_on_grid(sales, grid, horizon=HORIZON, seasonal_period=season)\n", "scale = naive_scale_on_grid(sales[train_rows], train_grid)\n", "\n", "def mae(values):\n", " mask = np.isfinite(values)\n", " return float(np.mean(np.abs(y_test[mask] - values[mask])))\n", "\n", "report = pd.DataFrame({\n", " 'mae': [mae(predicted), mae(baselines['naive'][test_rows]), mae(baselines['seasonal_naive'][test_rows])],\n", "}, index=['ridge', 'naive (last value)', f'seasonal naive ({season})'])\n", "report['mase'] = report['mae'] / scale\n", "report['skill_vs_seasonal_naive'] = 1 - report['mae'] / report.loc[f'seasonal naive ({season})', 'mae']\n", "display(report.round(3))\n" ] }, { "cell_type": "markdown", "id": "31f064b0", "metadata": {}, "source": [ "## 5. Closed Days Are Structure, Not Noise\n", "\n", "Roughly 17% of rows have `Sales == 0`, almost all of them Sundays. A forecaster\n", "has three options and only one of them is honest here.\n", "\n", "Dropping closed days would break the daily cadence that every lag and every\n", "window depends on: `lag_7` would silently stop meaning \"one week ago\". Imputing\n", "them would invent revenue that never existed. Keeping them and giving the model\n", "`Open` lets it learn the closure rule directly - which is exactly why `Open`\n", "appears in the future window.\n", "\n", "The cost is that error metrics average over days the store could not trade,\n", "which flatters them. Section 10 looks at error by store, where that shows up." ] }, { "cell_type": "code", "execution_count": null, "id": "97420312", "metadata": {}, "outputs": [], "source": [ "closed = history_df.Open.eq(0)\n", "print(f'Closed rows: {closed.sum()} of {len(history_df)} ({closed.mean():.1%})')\n", "print(f'Zero sales while open: {int((history_df.Sales.eq(0) & ~closed).sum())}')\n", "\n", "by_dow = history_df.groupby('DayOfWeek').agg(\n", " mean_sales=('Sales', 'mean'),\n", " mean_customers=('Customers', 'mean'),\n", " open_rate=('Open', 'mean'),\n", ")\n", "by_dow.index = ['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun']\n", "display(by_dow.round(1))\n", "\n", "promo_lift = history_df[history_df.Open.eq(1)].groupby('Promo').Sales.mean()\n", "print(f'Mean sales while open - no promo {promo_lift.loc[0]:,.0f}, promo {promo_lift.loc[1]:,.0f} '\n", " f'({promo_lift.loc[1] / promo_lift.loc[0] - 1:+.1%})')" ] }, { "cell_type": "markdown", "id": "9334a605", "metadata": {}, "source": [ "## 6. Establish a Seasonal Baseline\n", "\n", "Weekly rhythm dominates this series, so seasonal naive - \"same weekday last\n", "week\" - is a strong baseline and a fair one. It carries the closure pattern for\n", "free, since last Sunday was also closed. Any learned model that cannot beat it\n", "is not earning its complexity." ] }, { "cell_type": "code", "execution_count": null, "id": "3f6d64e2", "metadata": {}, "outputs": [], "source": [ "baseline = SeasonalNaiveForecaster(seasonal_period=7).fit(\n", " history.data,\n", " target_columns=history.schema.target_columns,\n", " time_column=history.schema.time,\n", " entity_columns=history.schema.entity_columns,\n", ")\n", "baseline_predictions = baseline.predict(ForecastHorizon.up_to(7))\n", "print('Baseline rows:', len(baseline_predictions))\n", "display(baseline_predictions.head(8))" ] }, { "cell_type": "markdown", "id": "1f6807a0", "metadata": {}, "source": [ "## 7. Build Leakage-Conscious Features\n", "\n", "Every feature must be computable at the origin. `ForecastFeatureBuilder`\n", "enforces that by construction:\n", "\n", "- **target lags** at 1, 7 and 14 days - yesterday, last week, the week before\n", "- **rolling means** with `lag=1`, so a 7-day window ends the day *before* the\n", " origin and never touches the day being predicted\n", "- **calendar terms** as sine/cosine pairs, which are pure functions of the date\n", " and therefore always knowable\n", "\n", "Future covariates need no lag at all - `Promo` for next Tuesday is already\n", "known. That asymmetry is the whole point of separating future from historic\n", "covariates in the schema." ] }, { "cell_type": "code", "execution_count": null, "id": "9418669b", "metadata": {}, "outputs": [], "source": [ "feature_builder = ForecastFeatureBuilder(\n", " target_lags=(1, 7, 14),\n", " rolling_features=(\n", " RollingFeature('Sales', window=7, statistic='mean', lag=1),\n", " RollingFeature('Customers', window=7, statistic='mean', lag=1),\n", " ),\n", " calendar_features=('day_of_week_sin', 'day_of_week_cos', 'month_sin', 'month_cos'),\n", ")\n", "\n", "one_week = feature_builder.make_supervised(history, ForecastHorizon.up_to(7))\n", "print('Supervised rows:', len(one_week))\n", "print('Feature columns:', list(one_week.X.columns))\n", "display(one_week.to_frame().head())" ] }, { "cell_type": "markdown", "id": "af0513c6", "metadata": {}, "source": [ "## 8. Direct Multi-Horizon Forecasting\n", "\n", "A *direct* strategy fits one model per horizon and target. Day 1 and day 7 are\n", "different problems - by day 7 the most recent lag is a week stale - so letting\n", "each fit its own coefficients is usually stronger than asking one model to serve\n", "both.\n", "\n", "Three horizons and two targets gives six models. `panel_strategy='global'` fits\n", "across all six stores together, sharing what they have in common." ] }, { "cell_type": "code", "execution_count": null, "id": "2bbe7fbb", "metadata": {}, "outputs": [], "source": [ "HORIZONS = (1, 3, 7)\n", "\n", "def make_direct_forecaster():\n", " return TabularForecaster(\n", " Ridge(alpha=1.0),\n", " config=ForecastConfig(\n", " horizons=HORIZONS,\n", " strategy='direct',\n", " panel_strategy='global',\n", " ),\n", " feature_builder=feature_builder,\n", " )\n", "\n", "direct_forecaster = make_direct_forecaster().fit(history)\n", "direct_future = direct_forecaster.forecast(future_df)\n", "\n", "print('Fitted models:', len(direct_forecaster.state.models))\n", "display(direct_future.head(12))" ] }, { "cell_type": "markdown", "id": "55a7035e", "metadata": {}, "source": [ "## 9. Gap-Aware Rolling-Origin Backtesting\n", "\n", "A single holdout gives one estimate of forecast error against one recent\n", "stretch, and recent stretches differ. Rolling-origin backtesting repeats the\n", "forecast at several historical cutoffs and reports the distribution instead.\n", "\n", "The `gap` is the part most tooling omits. Suppose sales are consolidated weekly,\n", "so at any origin the last seven days of outcomes are not yet available. Training\n", "on them would use labels the forecaster could not have had. `gap=7` withholds\n", "exactly those rows from every fold - and note it is a *label availability*\n", "question, not a feature availability one.\n", "\n", "`purged_label_samples` reports how many rows each fold dropped for that reason." ] }, { "cell_type": "code", "execution_count": null, "id": "20a057c9", "metadata": {}, "outputs": [], "source": [ "REPORTING_GAP = 7\n", "\n", "splitter = ExpandingWindowSplitter(\n", " n_splits=3,\n", " test_size=28,\n", " gap=REPORTING_GAP,\n", " min_train_size=365,\n", ")\n", "backtest = ForecastBacktester(splitter).run(\n", " history,\n", " forecaster_factory=make_direct_forecaster,\n", ")\n", "\n", "fold_summary = pd.DataFrame([{\n", " 'fold': fold.fold,\n", " 'train_origins': fold.split.train_size,\n", " 'gap_origins': fold.split.gap_size,\n", " 'validation_origins': fold.validation_origins,\n", " 'training_samples': fold.training_samples,\n", " 'unavailable_labels_removed': fold.purged_label_samples,\n", " 'validation_start': fold.split.test_origin_start,\n", " 'validation_end': fold.split.test_origin_end,\n", "} for fold in backtest.folds])\n", "display(fold_summary)" ] }, { "cell_type": "markdown", "id": "2af32529", "metadata": {}, "source": [ "## 10. Read Overall and Grouped Metrics\n", "\n", "One number hides the shape of the error. Four things in this output are worth\n", "more than the headline.\n", "\n", "**Error barely grows with horizon.** Day 1 gives wape 14.2%, day 7 gives 14.9%.\n", "The common heuristic says forecast error should climb steeply with distance, and\n", "flat error should make you suspect leakage - but that heuristic assumes\n", "predictability comes from recent momentum. Here it does not. Weekly rhythm,\n", "promotions and the closure calendar dominate, and all of them are known exactly\n", "at every horizon. The lag features decay as the horizon lengthens; the calendar\n", "and promotion terms do not. When your signal lives in known-in-advance drivers,\n", "horizon costs you little. That is a property of the problem, not a bug.\n", "\n", "**sMAPE is roughly 45% while MAPE is roughly 12%.** This is the cost of the\n", "closed days from section 5 arriving on the invoice. Percentage errors divide by\n", "the actual, and a Sunday actual is zero. sMAPE's symmetric denominator does not\n", "rescue it. On any series with legitimate zeros, read `wape` - which divides by\n", "the total rather than per row - and treat sMAPE as uninformative.\n", "\n", "**Stores differ by more than the horizon does.** wape ranges from 12.7% to 17.8%\n", "across six stores, a wider spread than anything horizon produces. The weakest\n", "store also carries by far the largest negative bias. That is a modelling signal,\n", "not noise: it says a global model is leaving something store-specific on the\n", "table, which is exactly the question section 12 asks.\n", "\n", "**Compare scale-free metrics across targets, never rmse.** `Sales` shows rmse\n", "1245 against `Customers` at 104, purely because one is money and the other is\n", "people. Their wape values, 15.0% and 13.1%, are comparable and tell you the\n", "truthful story: the two targets are forecast about equally well.\n", "\n", "`bias` is signed and negative almost everywhere here, meaning systematic\n", "under-forecasting. In retail that means stockouts. `mase` and `rmsse` are absent\n", "because both need an in-sample seasonal error scale to divide by, which this\n", "report was not given." ] }, { "cell_type": "code", "execution_count": null, "id": "77a204d1", "metadata": {}, "outputs": [], "source": [ "print('Overall')\n", "display(pd.Series(backtest.metrics.metrics, name='value').to_frame().round(3))\n", "\n", "print('By horizon')\n", "display(pd.DataFrame(backtest.metrics.grouped_metrics['horizon']).round(3))\n", "\n", "print('By store')\n", "display(pd.DataFrame(backtest.metrics.grouped_metrics['entity']).round(3))\n", "\n", "print('By target')\n", "display(pd.DataFrame(backtest.metrics.grouped_metrics['target']).round(3))" ] }, { "cell_type": "markdown", "id": "bb86a5a9", "metadata": {}, "source": [ "## 11. Compare Recursive Forecasting\n", "\n", "A *recursive* strategy fits one model for a single step and applies it\n", "repeatedly, feeding each prediction back in as the next lag. It is compact, and\n", "it needs every feature to be available at every step - which holds here only\n", "because this dataset has no historic-only covariates.\n", "\n", "Its weakness is error compounding: a day-1 mistake becomes an input to day 2.\n", "Compare the two strategies rather than assuming one wins." ] }, { "cell_type": "code", "execution_count": null, "id": "ca870371", "metadata": {}, "outputs": [], "source": [ "recursive_forecaster = TabularForecaster(\n", " Ridge(alpha=1.0),\n", " config=ForecastConfig(horizons=HORIZONS, strategy='recursive'),\n", " feature_builder=ForecastFeatureBuilder(\n", " target_lags=(1, 7, 14),\n", " calendar_features=('day_of_week_sin', 'day_of_week_cos'),\n", " ),\n", ").fit(history)\n", "recursive_future = recursive_forecaster.forecast(future_df)\n", "\n", "comparison = direct_future.merge(\n", " recursive_future,\n", " on=['origin', 'time', 'horizon', 'entity', 'target'],\n", " suffixes=('_direct', '_recursive'),\n", ")\n", "display(\n", " comparison[comparison.target.eq('Sales')][\n", " ['entity', 'time', 'horizon', 'y_pred_direct', 'y_pred_recursive']\n", " ].round(1).head(12)\n", ")" ] }, { "cell_type": "markdown", "id": "a53a05e0", "metadata": {}, "source": [ "## 12. Global Versus Local Panel Models\n", "\n", "A **global** model pools every store and learns one set of coefficients per\n", "horizon and target. A **local** model fits each store separately.\n", "\n", "Global borrows strength - a quiet store benefits from patterns visible in busier\n", "ones - and is the better default when series are few or short. Local wins when\n", "stores genuinely behave differently. Here, six stores across two store types is\n", "exactly the regime where it is worth measuring rather than guessing." ] }, { "cell_type": "code", "execution_count": null, "id": "b37b1715", "metadata": {}, "outputs": [], "source": [ "global_one_step = TabularForecaster(\n", " Ridge(alpha=1.0),\n", " config=ForecastConfig(horizons=(1,), panel_strategy='global'),\n", " feature_builder=feature_builder,\n", ").fit(history)\n", "local_one_step = TabularForecaster(\n", " Ridge(alpha=1.0),\n", " config=ForecastConfig(horizons=(1,), panel_strategy='local'),\n", " feature_builder=feature_builder,\n", ").fit(history)\n", "\n", "print('Global model count:', len(global_one_step.state.models))\n", "print('Local model count :', len(local_one_step.state.models))" ] }, { "cell_type": "markdown", "id": "04e708d1", "metadata": {}, "source": [ "## 13. Joint Multi-Target, Multi-Horizon Forecasting\n", "\n", "`Sales` and `Customers` are not independent - footfall drives revenue. A\n", "`multioutput` strategy fits one estimator that emits every target and horizon\n", "together, which keeps their relationship intact instead of letting two separate\n", "models drift apart.\n", "\n", "Capability is checked before use: estimators that cannot emit multiple outputs\n", "are rejected rather than silently wrapped." ] }, { "cell_type": "code", "execution_count": null, "id": "9d6353a2", "metadata": {}, "outputs": [], "source": [ "multioutput_forecaster = TabularForecaster(\n", " Ridge(alpha=1.0),\n", " config=ForecastConfig(\n", " horizons=(1, 7),\n", " strategy='multioutput',\n", " panel_strategy='global',\n", " ),\n", " feature_builder=feature_builder,\n", ").fit(history)\n", "joint_future = multioutput_forecaster.forecast(future_df, horizons=(1, 7))\n", "\n", "print('Joint target/horizon outputs:', sorted(\n", " set(zip(joint_future['target'], joint_future['horizon']))\n", "))\n", "display(joint_future.head(8))" ] }, { "cell_type": "markdown", "id": "5dcd535a", "metadata": {}, "source": [ "## 14. Add Conformal Prediction Intervals\n", "\n", "A point forecast says nothing about how wrong it might be. Conformal calibration\n", "turns held-out residuals into intervals with a target coverage level, without\n", "assuming a distribution and without retraining.\n", "\n", "Grouping by `('horizon', 'target')` matters: day-7 uncertainty is wider than\n", "day-1, and `Sales` is on a different scale from `Customers`. One pooled interval\n", "would be too wide for the easy cases and too narrow for the hard ones.\n", "\n", "Calibrate on earlier folds, measure coverage on a later one. Coverage close to\n", "the nominal level is the result you want; well above it means intervals are\n", "wastefully wide, and below it means they are lying." ] }, { "cell_type": "code", "execution_count": null, "id": "82a50e7f", "metadata": {}, "outputs": [], "source": [ "calibration_predictions = backtest.predictions[backtest.predictions['fold'].lt(2)]\n", "interval_validation = backtest.predictions[backtest.predictions['fold'].eq(2)]\n", "\n", "calibrator = ConformalIntervalCalibrator(\n", " alpha=0.2, # target 80% coverage\n", " group_by=('horizon', 'target'),\n", " min_group_size=20,\n", ").fit(calibration_predictions)\n", "\n", "coverage_report = evaluate_interval_coverage(calibrator.transform(interval_validation))\n", "future_intervals = calibrator.transform(direct_future)\n", "\n", "print('Nominal coverage: 0.80')\n", "display(pd.Series(coverage_report.metrics, name='value').to_frame().round(3))\n", "display(\n", " future_intervals[future_intervals.target.eq('Sales')][\n", " ['entity', 'time', 'horizon', 'y_pred', 'lower', 'upper']\n", " ].round(1).head(9)\n", ")" ] }, { "cell_type": "markdown", "id": "02bce05d", "metadata": {}, "source": [ "## 15. Reconcile the Store Hierarchy\n", "\n", "Forecasts are usually needed at several levels at once: per store for staffing,\n", "per store type for buying, chain-wide for finance. Forecast each level\n", "independently and the totals will not agree, which is indefensible when the same\n", "numbers appear in two reports.\n", "\n", "Reconciliation enforces coherence. `bottom_up` sums the store forecasts into\n", "their type and then the chain, so every parent equals the sum of its children by\n", "construction." ] }, { "cell_type": "code", "execution_count": null, "id": "d013a15c", "metadata": {}, "outputs": [], "source": [ "store_type = store_meta.set_index('Store').StoreType.to_dict()\n", "\n", "parent_map = {f'store_{s}': f'type_{store_type[s]}' for s in STORES}\n", "parent_map.update({f'type_{t}': 'chain' for t in {store_type[s] for s in STORES}})\n", "parent_map['chain'] = None\n", "\n", "hierarchy = Hierarchy.from_parent_map(parent_map)\n", "reconciler = HierarchicalReconciler(hierarchy, method='bottom_up')\n", "\n", "bottom_sales = direct_future[\n", " direct_future['target'].eq('Sales') & direct_future['horizon'].eq(1)\n", "]\n", "coherent_sales = reconciler.reconcile_frame(bottom_sales, actual_col=None)\n", "\n", "print('Hierarchy levels: store -> store type -> chain')\n", "print('Coherent:', hierarchy.is_coherent(coherent_sales['y_pred'].to_numpy()))\n", "display(coherent_sales.round(1))" ] }, { "cell_type": "markdown", "id": "de7c3d72", "metadata": {}, "source": [ "## 16. Save, Load, and Replay the Forecast\n", "\n", "A forecaster is only useful if it produces identical output tomorrow. Saving\n", "captures the estimator, the fitted feature contract and the schema together, so\n", "a restored artifact reproduces predictions exactly rather than approximately.\n", "\n", "The assertion below is the one that matters. If it ever fails, something stateful\n", "was left out of the artifact." ] }, { "cell_type": "code", "execution_count": null, "id": "ceeb8a73", "metadata": {}, "outputs": [], "source": [ "with TemporaryDirectory() as artifact_dir:\n", " artifact_path = Path(artifact_dir) / 'rossmann_forecast.pkl'\n", " direct_forecaster.save(artifact_path)\n", " restored = TabularForecaster.load(artifact_path)\n", " replayed = restored.forecast(future_df)\n", "\n", "pd.testing.assert_frame_equal(direct_future, replayed)\n", "print('Artifact replay is exactly equivalent.')\n", "print(restored.to_model_metadata(name='rossmann_forecast').inference_contract)" ] }, { "cell_type": "markdown", "id": "4f64da2b", "metadata": {}, "source": [ "## 17. Optional Native Statistical Models\n", "\n", "Classical statistical models remain competitive on short, regular, univariate\n", "series, and are often the right answer for a single well-behaved series. The\n", "adapter is optional and imported lazily, so it costs nothing unless installed:\n", "\n", "```bash\n", "pip install \"bitbullet[forecast-statistical]\"\n", "```\n", "\n", "```python\n", "from bitbullet.forecast import StatsForecastAdapter\n", "```\n", "\n", "Reach for it when a series is univariate and seasonal. Prefer the tabular\n", "forecasters when covariates matter, when many series share structure, or when\n", "you need one model to serve a panel.\n" ] }, { "cell_type": "markdown", "id": "16f4ddb8", "metadata": {}, "source": [ "## What You Built\n", "\n", "A complete forecasting workflow on real retail data: an explicit contract with\n", "future covariates separated from outcomes, one series placed on its calendar\n", "grid with a horizon that respects every missing period, a deliberate decision\n", "about closed days, a seasonal baseline and the MASE to beat it, leakage-conscious\n", "features, direct and recursive multi-horizon strategies, global and local panel\n", "models, joint multi-target output, gap-aware rolling-origin evaluation,\n", "calibrated intervals with measured coverage, a coherent three-level hierarchy,\n", "and an artifact that replays exactly.\n", "\n", "### Forecasting Checklist\n", "\n", "| Question | Where it was answered |\n", "| --- | --- |\n", "| Which columns are known at the origin? | Section 3, future versus static covariates |\n", "| Does the cadence support lags and windows? | Section 3, cadence diagnostics |\n", "| Does a horizon in rows mean a horizon in periods? | Section 4, period inference and the calendar grid |\n", "| What do structural zeros mean? | Section 5 |\n", "| What am I trying to beat? | Section 6, seasonal naive |\n", "| Can any feature see the future? | Section 7, lagged windows and calendar terms |\n", "| Does error grow with horizon? | Section 10, grouped metrics |\n", "| Were unavailable labels withheld? | Section 9, gap and purged samples |\n", "| How wrong might this be? | Section 14, measured coverage |\n", "| Do the levels add up? | Section 15, reconciliation |\n", "| Will it reproduce tomorrow? | Section 16, exact replay |\n", "| Is the model earning its complexity? | Sections 4 and 6, seasonal naive and MASE |" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.11" } }, "nbformat": 4, "nbformat_minor": 5 }