{ "cells": [ { "cell_type": "markdown", "id": "md59cd2519", "metadata": {}, "source": [ "# BitBullet:Lessons: Model Management\n", "\n", "You trained a great model. Now what?\n", "\n", "In most teams, that question is answered with a folder called `models/` containing files named `final_v3_USE_THIS.pkl`, a Slack message with the test AUC, and a vague memory of what preprocessing was applied. Six months later, no one can reproduce the result, the model is silently serving stale predictions, and rolling back to the previous version means digging through git blame.\n", "\n", "**This is model rot. And it is entirely preventable.**\n", "\n", "**Dataset**: `default_of_credit_card_clients.xls` — same dataset as Lesson 04. \n", "**Source**: [UCI ML Repository — Default of Credit Card Clients](https://archive.uci.edu/dataset/350/default+of+credit+card+clients)\n", "\n", "---\n", "\n", "### What This Tutorial Covers\n", "\n", "| Section | Component | What You Will Learn |\n", "|---------|-----------|---------------------|\n", "| 1 | Setup | Imports and lightweight training run |\n", "| 2 | `ModelMetadata` | Capturing identity, hyperparameters, metrics, and provenance |\n", "| 3 | `DatasetMetadata` | Recording what you trained on — including cryptographic hashing |\n", "| 4 | `metadata.summary()` | Human-readable model cards |\n", "| 5 | `metadata.to_dict()` | JSON-serialisable output for logging |\n", "| 6 | `ModelSerializer.save()` | Full packages — one artefact, everything inside |\n", "| 7 | `load_metadata()` | Fast metadata inspection without loading the model |\n", "| 8 | `ModelSerializer.load()` | Reconstructing the full package from disk |\n", "| 9 | `save_model_only()` | Lean model-only artefacts |\n", "| 10 | `ModelRegistry` | A typed catalogue of every model architecture |\n", "| 11 | `@ModelRegistry.register` | Extending BitBullet with custom model wrappers |\n", "| 12 | Version management | Semantic versioning patterns |\n", "| 13 | V1 vs V2 comparison | Loading, diffing, and visualising metric deltas |\n", "| 14 | Production checklist | What to verify before promoting a model |" ] }, { "cell_type": "markdown", "id": "9e6b0ecf", "metadata": {}, "source": [ "> **BitBullet Platform handles this lifecycle under the hood.**\n", "> [BitBullet Platform](https://bitbullet.co.uk) centralises projects, datasets, managed compute, storage, resolved configurations, fitted artefacts, preprocessing, metrics, and export material in a guided, repeatable workflow. Design and compare modelling configurations, inspect completed evidence, then export the trained result with metadata and generated inference code. This lesson shows how to manage those artefacts explicitly with the BitBullet SDK." ] }, { "cell_type": "markdown", "id": "md0da86e79", "metadata": {}, "source": [ "---\n", "## 1. Environment Setup & Imports" ] }, { "cell_type": "code", "execution_count": 1, "id": "cd7ab0b1a3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "bitbullet : vdev\n", "All imports successful.\n" ] } ], "source": [ "import sys\n", "import os\n", "import json\n", "import shutil\n", "import warnings\n", "import time\n", "from datetime import datetime\n", "from pathlib import Path\n", "\n", "import pandas as pd\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import matplotlib.ticker as mticker\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.metrics import roc_auc_score, f1_score, accuracy_score\n", "\n", "warnings.filterwarnings('ignore')\n", "\n", "# When running this notebook from bitbullet/lessons in a local clone, prefer the local SDK source.\n", "sdk_root = os.path.abspath(\"..\")\n", "if sdk_root not in sys.path:\n", " sys.path.insert(0, sdk_root)\n", "\n", "import bitbullet\n", "print(f\"bitbullet : v{getattr(bitbullet, '__version__', 'dev')}\")\n", "\n", "from bitbullet.model import (\n", " ModelMetadata,\n", " DatasetMetadata,\n", " ModelSerializer,\n", " ModelRegistry,\n", " BaseModel,\n", ")\n", "from bitbullet.transform import TransformPipeline\n", "from bitbullet.train import TrainConfig, OptunaTrainer\n", "\n", "print(\"All imports successful.\")" ] }, { "cell_type": "markdown", "id": "mdd108c418", "metadata": {}, "source": [ "---\n", "## 2. Quick Training Run\n", "\n", "**Dataset:** Default of Credit Card Clients Dataset \n", "**Source:** [UCI ML Repository — Default of Credit Card Clients](https://archive.uci.edu/dataset/350/default+of+credit+card+clients) \n", "**File:** `default_of_credit_card_clients.xls`\n", "\n", "> **Before running this cell:** download `default_of_credit_card_clients.xls` from the link above and\n", "> place it in the **same directory as this notebook**. If you store it elsewhere,\n", "> update `data_path` in the code cell below to match your chosen location." ] }, { "cell_type": "code", "execution_count": null, "id": "cde567e468", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Dataset loaded — shape: (30000, 24)\n", "Default rate — 22.1%\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
LIMIT_BALSEXEDUCATIONMARRIAGEAGEPAY_0PAY_2PAY_3PAY_4PAY_5...BILL_AMT4BILL_AMT5BILL_AMT6PAY_AMT1PAY_AMT2PAY_AMT3PAY_AMT4PAY_AMT5PAY_AMT6default_payment
020000FemaleUniversityMarried2422-1-1-2...000068900001
1120000FemaleUniversitySingle26-12000...3272345532610100010001000020001
290000FemaleUniversitySingle3400000...1433114948155491518150010001000100050000
\n", "

3 rows × 24 columns

\n", "
" ], "text/plain": [ " LIMIT_BAL SEX EDUCATION MARRIAGE AGE PAY_0 PAY_2 PAY_3 PAY_4 \\\n", "0 20000 Female University Married 24 2 2 -1 -1 \n", "1 120000 Female University Single 26 -1 2 0 0 \n", "2 90000 Female University Single 34 0 0 0 0 \n", "\n", " PAY_5 ... BILL_AMT4 BILL_AMT5 BILL_AMT6 PAY_AMT1 PAY_AMT2 PAY_AMT3 \\\n", "0 -2 ... 0 0 0 0 689 0 \n", "1 0 ... 3272 3455 3261 0 1000 1000 \n", "2 0 ... 14331 14948 15549 1518 1500 1000 \n", "\n", " PAY_AMT4 PAY_AMT5 PAY_AMT6 default_payment \n", "0 0 0 0 1 \n", "1 1000 0 2000 1 \n", "2 1000 1000 5000 0 \n", "\n", "[3 rows x 24 columns]" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Update this path if you stored the file in a different location.\n", "data_path = \"default_of_credit_card_clients.xls\"\n", "df_raw = pd.read_excel(data_path, header=1)\n", "df = df_raw.drop(columns=['ID']).rename(columns={'default payment next month': 'default_payment'})\n", "\n", "df['SEX'] = df['SEX'].map({1: 'Male', 2: 'Female'})\n", "df['EDUCATION'] = df['EDUCATION'].map({1: 'Graduate', 2: 'University', 3: 'High_School',\n", " 4: 'Other', 5: 'Other', 6: 'Other', 0: 'Other'})\n", "df['MARRIAGE'] = df['MARRIAGE'].map({1: 'Married', 2: 'Single', 3: 'Other', 0: 'Other'})\n", "\n", "target_col = 'default_payment'\n", "X = df.drop(columns=[target_col])\n", "y = df[target_col]\n", "\n", "print(f\"Dataset loaded — shape: {df.shape}\")\n", "print(f\"Default rate — {y.mean():.1%}\")\n", "display(df.head(3))" ] }, { "cell_type": "code", "execution_count": 6, "id": "cdb6a13d65", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Training : 24,000 samples (default rate: 22.1%)\n", "Test : 6,000 samples (default rate: 22.1%)\n" ] } ], "source": [ "X_train, X_test, y_train, y_test = train_test_split(\n", " X, y, test_size=0.20, random_state=42, stratify=y\n", ")\n", "print(f\"Training : {X_train.shape[0]:,} samples (default rate: {y_train.mean():.1%})\")\n", "print(f\"Test : {X_test.shape[0]:,} samples (default rate: {y_test.mean():.1%})\")" ] }, { "cell_type": "code", "execution_count": 7, "id": "cd97c17df3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Transform applied — 23 raw features → 26 engineered\n" ] } ], "source": [ "pay_amt_cols = [f'PAY_AMT{i}' for i in range(1, 7)]\n", "bill_amt_cols = [f'BILL_AMT{i}' for i in range(1, 7)]\n", "pay_cols = ['PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6']\n", "\n", "pipeline_v1 = (\n", " TransformPipeline(name=\"credit_default_pipeline_v1\")\n", " .add(\"numerical\", \"log1p\", columns=pay_amt_cols + ['LIMIT_BAL'])\n", " .add(\"numerical\", \"yeo_johnson\", columns=bill_amt_cols)\n", " .add(\"numerical\", \"standard_scale\", columns=[\"AGE\"])\n", " .add(\"numerical\", \"robust_scale\", columns=pay_cols)\n", " .add(\"categorical\", \"label_encode\", columns=['SEX', 'MARRIAGE'],\n", " params={\"unknown_strategy\": \"use_encoded_value\", \"unknown_value\": -1})\n", " .add(\"categorical\", \"onehot_encode\", columns=['EDUCATION'],\n", " params={\"sparse\": False, \"drop\": None})\n", ")\n", "\n", "X_train_t = pipeline_v1.fit_transform(X_train)\n", "X_test_t = pipeline_v1.transform(X_test)\n", "print(f\"Transform applied — {X_train.shape[1]} raw features \\u2192 {X_train_t.shape[1]} engineered\")" ] }, { "cell_type": "code", "execution_count": 8, "id": "cd2ce9067d", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[I 2026-05-06 14:28:45,367] A new study created in memory with name: credit_default_lgbm_v1\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "======================================================================\n", "BitBullet Train - credit_default_lgbm_v1\n", "======================================================================\n", "\n", "Training samples: 24000\n", "Features: 26\n", "Class distribution: {0: 18691, 1: 5309}\n", "\n", "Starting hyperparameter optimization (optuna)...\n", "Trials: 10, CV Folds: 3\n", "\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[I 2026-05-06 14:28:49,397] Trial 4 finished with value: 0.782487674922033 and parameters: {'num_leaves': 159, 'max_depth': 3, 'min_child_samples': 10, 'lambda_l1': 1.0469669874479804e-05, 'lambda_l2': 9.142555405712056e-06, 'min_gain_to_split': 0.34314851045805117, 'feature_fraction': 0.5878176146192433, 'bagging_fraction': 0.975979532434263, 'bagging_freq': 6, 'learning_rate': 0.0658812570668321, 'max_bin': 255}. Best is trial 4 with value: 0.782487674922033.\n", "[I 2026-05-06 14:28:52,481] Trial 7 finished with value: 0.7837924046279956 and parameters: {'num_leaves': 109, 'max_depth': 12, 'min_child_samples': 96, 'lambda_l1': 3.5285162658627813, 'lambda_l2': 0.0019118667106049142, 'min_gain_to_split': 0.32810293195939566, 'feature_fraction': 0.6071973854234938, 'bagging_fraction': 0.6829336275747605, 'bagging_freq': 1, 'learning_rate': 0.22938974425733039, 'max_bin': 159}. Best is trial 7 with value: 0.7837924046279956.\n", "[I 2026-05-06 14:28:53,259] Trial 0 finished with value: 0.782487220249224 and parameters: {'num_leaves': 167, 'max_depth': 5, 'min_child_samples': 52, 'lambda_l1': 0.012244544992846699, 'lambda_l2': 3.807906909338432e-06, 'min_gain_to_split': 0.12483790206820222, 'feature_fraction': 0.9836098221122052, 'bagging_fraction': 0.5360854270191966, 'bagging_freq': 5, 'learning_rate': 0.037154923360773084, 'max_bin': 63}. Best is trial 7 with value: 0.7837924046279956.\n", "[I 2026-05-06 14:28:53,918] Trial 3 finished with value: 0.780966030345489 and parameters: {'num_leaves': 110, 'max_depth': 7, 'min_child_samples': 7, 'lambda_l1': 0.8973210675155582, 'lambda_l2': 0.0682781033300667, 'min_gain_to_split': 0.18895290807712162, 'feature_fraction': 0.6036009337046335, 'bagging_fraction': 0.9853566704657207, 'bagging_freq': 6, 'learning_rate': 0.16851528691592105, 'max_bin': 191}. Best is trial 7 with value: 0.7837924046279956.\n", "[I 2026-05-06 14:28:57,320] Trial 8 finished with value: 0.7800509573551696 and parameters: {'num_leaves': 95, 'max_depth': 11, 'min_child_samples': 6, 'lambda_l1': 9.826480059552244e-07, 'lambda_l2': 3.5891969341014676, 'min_gain_to_split': 0.019956836311096127, 'feature_fraction': 0.5668757933312176, 'bagging_fraction': 0.6721811735214078, 'bagging_freq': 7, 'learning_rate': 0.1344044915382327, 'max_bin': 95}. Best is trial 7 with value: 0.7837924046279956.\n", "[I 2026-05-06 14:28:59,336] Trial 9 finished with value: 0.7804075427177111 and parameters: {'num_leaves': 144, 'max_depth': 9, 'min_child_samples': 23, 'lambda_l1': 4.032669945433137e-05, 'lambda_l2': 0.00017840705367659724, 'min_gain_to_split': 0.23520980412204784, 'feature_fraction': 0.9323885741820045, 'bagging_fraction': 0.8762152516768509, 'bagging_freq': 4, 'learning_rate': 0.054954409425528324, 'max_bin': 127}. Best is trial 7 with value: 0.7837924046279956.\n", "[I 2026-05-06 14:29:00,084] Trial 6 finished with value: 0.7844659064489745 and parameters: {'num_leaves': 190, 'max_depth': 5, 'min_child_samples': 8, 'lambda_l1': 4.854780435874212e-08, 'lambda_l2': 0.0003385218306665211, 'min_gain_to_split': 0.38641319697198695, 'feature_fraction': 0.8697100967282957, 'bagging_fraction': 0.773404252227638, 'bagging_freq': 4, 'learning_rate': 0.01686610263980842, 'max_bin': 127}. Best is trial 6 with value: 0.7844659064489745.\n", "[I 2026-05-06 14:29:00,382] Trial 2 finished with value: 0.7840050071347134 and parameters: {'num_leaves': 248, 'max_depth': 3, 'min_child_samples': 5, 'lambda_l1': 1.0197907205515458e-08, 'lambda_l2': 0.0006846440275151191, 'min_gain_to_split': 0.7171143191373233, 'feature_fraction': 0.7409280316047595, 'bagging_fraction': 0.5122644701055994, 'bagging_freq': 2, 'learning_rate': 0.009678552464226966, 'max_bin': 63}. Best is trial 6 with value: 0.7844659064489745.\n", "[I 2026-05-06 14:29:03,905] Trial 5 finished with value: 0.7853619152285397 and parameters: {'num_leaves': 289, 'max_depth': 5, 'min_child_samples': 13, 'lambda_l1': 4.791304580577613e-05, 'lambda_l2': 0.026538461705427185, 'min_gain_to_split': 0.7288215433160152, 'feature_fraction': 0.6082480512336117, 'bagging_fraction': 0.6018969670529957, 'bagging_freq': 1, 'learning_rate': 0.010181453780052564, 'max_bin': 127}. Best is trial 5 with value: 0.7853619152285397.\n", "[I 2026-05-06 14:29:08,684] Trial 1 finished with value: 0.7836721253189375 and parameters: {'num_leaves': 244, 'max_depth': 12, 'min_child_samples': 53, 'lambda_l1': 1.8175764466996019e-06, 'lambda_l2': 2.1529752202441412e-07, 'min_gain_to_split': 0.13464133779050647, 'feature_fraction': 0.7641059146439713, 'bagging_fraction': 0.5748102098291443, 'bagging_freq': 3, 'learning_rate': 0.018348297643719154, 'max_bin': 223}. Best is trial 5 with value: 0.7853619152285397.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "======================================================================\n", "Optuna Optimization Summary\n", "======================================================================\n", "Number of finished trials: 10\n", "Best trial: 5\n", "Best value: 0.785362\n", "\n", "Best hyperparameters:\n", " num_leaves: 289\n", " max_depth: 5\n", " min_child_samples: 13\n", " lambda_l1: 0.000048\n", " lambda_l2: 0.026538\n", " min_gain_to_split: 0.728822\n", " feature_fraction: 0.608248\n", " bagging_fraction: 0.601897\n", " bagging_freq: 1\n", " learning_rate: 0.010181\n", " max_bin: 127\n", "\n", "Additional attributes:\n", " cv_scores: [0.7865111521841548, 0.7797120729838307, 0.789862520517634]\n", " cv_std: 0.004222831235812959\n", " mean_training_loss: 0.38680237181227256\n", " mean_validation_loss: 0.42566816947721814\n", " best_iteration: 536\n", "======================================================================\n", "\n", "\n", "Optimization complete! Best score: 0.7854\n", "Time: 23.4s\n", "\n", "Training final model with best parameters...\n", "[LightGBM] [Warning] feature_fraction is set=0.6082480512336117, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.6082480512336117\n", "[LightGBM] [Warning] lambda_l2 is set=0.026538461705427185, reg_lambda=0.0 will be ignored. Current value: lambda_l2=0.026538461705427185\n", "[LightGBM] [Warning] min_gain_to_split is set=0.7288215433160152, min_split_gain=0.0 will be ignored. Current value: min_gain_to_split=0.7288215433160152\n", "[LightGBM] [Warning] lambda_l1 is set=4.791304580577613e-05, reg_alpha=0.0 will be ignored. Current value: lambda_l1=4.791304580577613e-05\n", "[LightGBM] [Warning] bagging_fraction is set=0.6018969670529957, subsample=1.0 will be ignored. Current value: bagging_fraction=0.6018969670529957\n", "[LightGBM] [Warning] bagging_freq is set=1, subsample_freq=0 will be ignored. Current value: bagging_freq=1\n", "[LightGBM] [Warning] feature_fraction is set=0.6082480512336117, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.6082480512336117\n", "[LightGBM] [Warning] lambda_l2 is set=0.026538461705427185, reg_lambda=0.0 will be ignored. Current value: lambda_l2=0.026538461705427185\n", "[LightGBM] [Warning] min_gain_to_split is set=0.7288215433160152, min_split_gain=0.0 will be ignored. Current value: min_gain_to_split=0.7288215433160152\n", "[LightGBM] [Warning] lambda_l1 is set=4.791304580577613e-05, reg_alpha=0.0 will be ignored. Current value: lambda_l1=4.791304580577613e-05\n", "[LightGBM] [Warning] bagging_fraction is set=0.6018969670529957, subsample=1.0 will be ignored. Current value: bagging_fraction=0.6018969670529957\n", "[LightGBM] [Warning] bagging_freq is set=1, subsample_freq=0 will be ignored. Current value: bagging_freq=1\n", "[LightGBM] [Info] Number of positive: 5309, number of negative: 18691\n", "[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.003791 seconds.\n", "You can set `force_col_wise=true` to remove the overhead.\n", "[LightGBM] [Info] Total Bins 1727\n", "[LightGBM] [Info] Number of data points in the train set: 24000, number of used features: 26\n", "[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.221208 -> initscore=-1.258639\n", "[LightGBM] [Info] Start training from score -1.258639\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "[LightGBM] [Warning] No further splits with positive gain, best gain: -inf\n", "\n", "Optimizing classification threshold...\n", "[LightGBM] [Warning] feature_fraction is set=0.6082480512336117, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.6082480512336117\n", "[LightGBM] [Warning] lambda_l2 is set=0.026538461705427185, reg_lambda=0.0 will be ignored. Current value: lambda_l2=0.026538461705427185\n", "[LightGBM] [Warning] min_gain_to_split is set=0.7288215433160152, min_split_gain=0.0 will be ignored. Current value: min_gain_to_split=0.7288215433160152\n", "[LightGBM] [Warning] lambda_l1 is set=4.791304580577613e-05, reg_alpha=0.0 will be ignored. Current value: lambda_l1=4.791304580577613e-05\n", "[LightGBM] [Warning] bagging_fraction is set=0.6018969670529957, subsample=1.0 will be ignored. Current value: bagging_fraction=0.6018969670529957\n", "[LightGBM] [Warning] bagging_freq is set=1, subsample_freq=0 will be ignored. Current value: bagging_freq=1\n", "Optimal threshold: 0.216\n", "\n", "======================================================================\n", "Training Complete!\n", "======================================================================\n", "=== Training Summary ===\n", "Model: LGBMClassifier\n", "Best Score: 0.7854\n", "CV Score: 0.0000 ± 0.0000\n", "Features: 26/26\n", "Optimal Threshold: 0.216\n", "Training Time: 27.7s\n", "Optimization Time: 23.4s\n", "\n", "Best Hyperparameters:\n", " num_leaves: 289\n", " max_depth: 5\n", " min_child_samples: 13\n", " lambda_l1: 0.0000\n", " lambda_l2: 0.0265\n", " min_gain_to_split: 0.7288\n", " feature_fraction: 0.6082\n", " bagging_fraction: 0.6019\n", " bagging_freq: 1\n", " learning_rate: 0.0102\n", " max_bin: 127\n", " n_estimators: 536\n", "======================================================================\n", "\n", "\n", "Training complete in 27.7s\n", "=== Training Summary ===\n", "Model: LGBMClassifier\n", "Best Score: 0.7854\n", "CV Score: 0.0000 ± 0.0000\n", "Features: 26/26\n", "Optimal Threshold: 0.216\n", "Training Time: 27.7s\n", "Optimization Time: 23.4s\n", "\n", "Best Hyperparameters:\n", " num_leaves: 289\n", " max_depth: 5\n", " min_child_samples: 13\n", " lambda_l1: 0.0000\n", " lambda_l2: 0.0265\n", " min_gain_to_split: 0.7288\n", " feature_fraction: 0.6082\n", " bagging_fraction: 0.6019\n", " bagging_freq: 1\n", " learning_rate: 0.0102\n", " max_bin: 127\n", " n_estimators: 536\n" ] } ], "source": [ "# n_trials=10, cv_folds=3 — fast demo run. Production: use >=100 trials, 5 folds.\n", "config_v1 = TrainConfig(\n", " name=\"credit_default_lgbm_v1\",\n", " model_type=\"lgbm\",\n", " task=\"binary_classification\",\n", " optimization_metric=\"roc_auc\",\n", " optuna_sampler=\"tpe\",\n", " n_trials=10,\n", " cv_folds=3,\n", " optimize_threshold=True,\n", " threshold_optimization_method=\"youden\",\n", " save_feature_importance=True,\n", " generate_shap=False,\n", " verbose=True,\n", " optuna_show_progress=False,\n", " random_state=42\n", ")\n", "\n", "t0 = time.time()\n", "trainer_v1 = OptunaTrainer(config=config_v1)\n", "trainer_v1.fit(X_train_t, y_train)\n", "state_v1 = trainer_v1.state\n", "\n", "print(f\"\\nTraining complete in {time.time() - t0:.1f}s\")\n", "print(state_v1.summary())" ] }, { "cell_type": "code", "execution_count": 9, "id": "cdf8043ec2", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[LightGBM] [Warning] feature_fraction is set=0.6082480512336117, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.6082480512336117\n", "[LightGBM] [Warning] lambda_l2 is set=0.026538461705427185, reg_lambda=0.0 will be ignored. Current value: lambda_l2=0.026538461705427185\n", "[LightGBM] [Warning] min_gain_to_split is set=0.7288215433160152, min_split_gain=0.0 will be ignored. Current value: min_gain_to_split=0.7288215433160152\n", "[LightGBM] [Warning] lambda_l1 is set=4.791304580577613e-05, reg_alpha=0.0 will be ignored. Current value: lambda_l1=4.791304580577613e-05\n", "[LightGBM] [Warning] bagging_fraction is set=0.6018969670529957, subsample=1.0 will be ignored. Current value: bagging_fraction=0.6018969670529957\n", "[LightGBM] [Warning] bagging_freq is set=1, subsample_freq=0 will be ignored. Current value: bagging_freq=1\n", "V1 Test Metrics:\n", " ROC AUC : 0.7801\n", " F1 Score : 0.5350\n", " Accuracy : 0.7552\n", " Threshold: 0.2164\n" ] } ], "source": [ "y_proba_v1 = state_v1.model.predict_proba(X_test_t)[:, 1]\n", "threshold_v1 = state_v1.optimal_threshold or 0.5\n", "y_pred_v1 = (y_proba_v1 >= threshold_v1).astype(int)\n", "\n", "test_report_v1 = evaluate_classification(\n", " y_true=y_test,\n", " y_pred_proba=y_proba_v1,\n", " threshold=threshold_v1,\n", " labels=[0, 1],\n", ")\n", "\n", "test_roc_auc_v1 = test_report_v1.metrics[\"roc_auc\"]\n", "test_f1_v1 = test_report_v1.metrics[\"f1_score\"]\n", "test_accuracy_v1 = test_report_v1.metrics[\"accuracy\"]\n", "\n", "print(f\"V1 Test Metrics:\")\n", "print(f\" ROC AUC : {test_roc_auc_v1:.4f}\")\n", "print(f\" F1 Score : {test_f1_v1:.4f}\")\n", "print(f\" Accuracy : {test_accuracy_v1:.4f}\")\n", "print(f\" Threshold: {threshold_v1:.4f}\")" ] }, { "cell_type": "markdown", "id": "md1bf5eb02", "metadata": {}, "source": [ "---\n", "\n", "## 3. `ModelMetadata`: Your Model's Identity Card\n", "\n", "`ModelMetadata` is a structured dataclass that captures everything needed for **reproducibility, auditing, lineage tracking, and exact inference replay** in data-science environments.\n", "\n", "| Field | Purpose |\n", "|-------|---------|\n", "| `name` + `version` | Unique identity — use semantic versioning |\n", "| `model_type` + `framework` | What architecture, what library |\n", "| `task` | `binary_classification`, `multiclass_classification` |\n", "| `feature_names` + `n_features` | Exact contract with the inference pipeline |\n", "| `hyperparameters` | Full reproducibility — never lose these |\n", "| `metrics` | Test-set performance, locked in before shipping |\n", "| `optimal_threshold` | The decision boundary found during training |\n", "| `notes` | Free-form context for the human reading this six months later |\n", "| `feature_schema` | Exact feature names, dtypes, and feature count |\n", "| `preprocessing` | Pointer to fitted transform pipeline and fit scope |\n", "| `search_metadata` | Optimizer, sampler, trial budget, CV setup, and best params |\n", "| `inference_contract` | Minimal replay instructions: feature order, threshold, pipeline dependency |\n", "| `artifact_metadata` | Structured reports such as `evaluate_classification(...).to_dict()` |" ] }, { "cell_type": "code", "execution_count": 10, "id": "cd36b1f009", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "ModelMetadata created with schema, preprocessing, search, and inference metadata.\n", " Name : credit_default_predictor v1.0.0\n", " Framework : lightgbm / LGBMClassifier\n", " Features : 26\n" ] } ], "source": [ "metadata_v1 = ModelMetadata(\n", " name=\"credit_default_predictor\",\n", " version=\"1.0.0\",\n", " model_type=state_v1.model.model_type,\n", " framework=state_v1.model.framework,\n", " task=\"binary_classification\",\n", " trained_at=datetime.now(),\n", " feature_names=list(X_train_t.columns),\n", " n_features=X_train_t.shape[1],\n", " hyperparameters=state_v1.best_params,\n", " classes=[0, 1],\n", " optimal_threshold=threshold_v1,\n", " training_time_seconds=state_v1.training_time_seconds,\n", " tags=[\"binary_classification\", \"credit_default\", \"lightgbm\", \"uci\"],\n", " notes=(\n", " \"Credit card default predictor v1.0. \"\n", " \"Trained on 30k UCI records. \"\n", " \"Youden-J threshold optimised during training.\"\n", " )\n", ")\n", "\n", "metadata_v1.add_feature_schema(X_train_t)\n", "metadata_v1.preprocessing = {\n", " \"pipeline_name\": pipeline_v1.name,\n", " \"pipeline_path\": \"pipelines/credit_default_pipeline_v1.pkl\",\n", " \"fit_scope\": \"training_data_only\",\n", "}\n", "metadata_v1.search_metadata = {\n", " \"optimizer\": config_v1.optimizer,\n", " \"optuna_sampler\": config_v1.optuna_sampler,\n", " \"n_trials\": config_v1.n_trials,\n", " \"cv_folds\": config_v1.cv_folds,\n", " \"optimization_metric\": config_v1.optimization_metric,\n", " \"best_params\": state_v1.best_params,\n", "}\n", "metadata_v1.target_mapping = {\"negative_class\": 0, \"positive_class\": 1}\n", "metadata_v1.inference_contract = {\n", " \"required_pipeline_artifact\": \"pipelines/credit_default_pipeline_v1.pkl\",\n", " \"required_feature_order\": list(X_train_t.columns),\n", " \"probability_column\": 1,\n", " \"decision_threshold\": threshold_v1,\n", "}\n", "metadata_v1.add_artifact_metadata(classification_metrics=test_report_v1.to_dict())\n", "\n", "print(\"ModelMetadata created with schema, preprocessing, search, and inference metadata.\")\n", "print(f\" Name : {metadata_v1.name} v{metadata_v1.version}\")\n", "print(f\" Framework : {metadata_v1.framework} / {metadata_v1.model_type}\")\n", "print(f\" Features : {metadata_v1.n_features}\")" ] }, { "cell_type": "markdown", "id": "md925182da", "metadata": {}, "source": [ "---\n", "## 4. `add_metric()` and `add_cv_scores()`" ] }, { "cell_type": "code", "execution_count": 11, "id": "cdba9d8598", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Metrics recorded:\n", " test_roc_auc : 0.7801\n", " test_f1 : 0.5350\n", " test_accuracy : 0.7552\n", "\n", "CV scores (3 folds):\n", " roc_auc: [0.7865, 0.7797, 0.7899]\n", " mean=0.7854 std=0.0042\n" ] } ], "source": [ "metadata_v1.add_metric(\"test_roc_auc\", test_roc_auc_v1)\n", "metadata_v1.add_metric(\"test_f1\", test_f1_v1)\n", "metadata_v1.add_metric(\"test_accuracy\", test_accuracy_v1)\n", "\n", "if state_v1.study:\n", " fold_scores = state_v1.study.best_trial.user_attrs.get('cv_scores', [state_v1.best_score])\n", "else:\n", " fold_scores = state_v1.cv_scores if state_v1.cv_scores else [state_v1.best_score]\n", "\n", "metadata_v1.add_cv_scores(\"roc_auc\", fold_scores)\n", "\n", "print(\"Metrics recorded:\")\n", "for name, val in metadata_v1.metrics.items():\n", " print(f\" {name:<20s}: {val:.4f}\")\n", "\n", "print(f\"\\nCV scores ({metadata_v1.cv_folds} folds):\")\n", "for metric, scores in metadata_v1.cv_scores.items():\n", " print(f\" {metric}: {[round(s, 4) for s in scores]}\")\n", " print(f\" mean={np.mean(scores):.4f} std={np.std(scores):.4f}\")" ] }, { "cell_type": "markdown", "id": "md960f3632", "metadata": {}, "source": [ "---\n", "## 5. `DatasetMetadata`: Knowing Exactly What You Trained On" ] }, { "cell_type": "code", "execution_count": 12, "id": "cd5152177e", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "=======================================================\n", " TRAIN DATASET: credit_default_train_v1\n", "=======================================================\n", " Samples : 24,000\n", " Features : 26\n", " Default rate : {0: 18691, 1: 5309}\n", " Data hash : 5d3b35fff8b0da10a6ed3a35b064f769\n", " Missing values: none\n", "\n", "Dataset provenance attached to model metadata.\n" ] } ], "source": [ "train_meta = DatasetMetadata.from_dataframe(\n", " X_train_t, y_train,\n", " name=\"credit_default_train_v1\",\n", " compute_hash=True\n", ")\n", "\n", "test_meta = DatasetMetadata.from_dataframe(\n", " X_test_t, y_test,\n", " name=\"credit_default_test_v1\",\n", " compute_hash=True\n", ")\n", "\n", "print(\"=\" * 55)\n", "print(f\" TRAIN DATASET: {train_meta.name}\")\n", "print(\"=\" * 55)\n", "print(f\" Samples : {train_meta.n_samples:,}\")\n", "print(f\" Features : {train_meta.n_features}\")\n", "print(f\" Default rate : {train_meta.class_distribution}\")\n", "print(f\" Data hash : {train_meta.data_hash}\")\n", "\n", "missing = {k: v for k, v in train_meta.statistics['missing_values'].items() if v > 0}\n", "print(f\" Missing values: {'none' if not missing else missing}\")\n", "\n", "metadata_v1.add_dataset(X_train_t, y_train, \"train\")\n", "metadata_v1.add_dataset(X_test_t, y_test, \"test\")\n", "print(\"\\nDataset provenance attached to model metadata.\")" ] }, { "cell_type": "markdown", "id": "md7458ff8f", "metadata": {}, "source": [ "---\n", "## 6. `metadata.summary()` — Human-Readable Model Cards" ] }, { "cell_type": "code", "execution_count": 13, "id": "cd0611929f", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model: credit_default_predictor (v1.0.0)\n", "Type: LGBMClassifier (lightgbm)\n", "Task: binary_classification\n", "Features: 26\n", "Trained: 2026-05-06 14:29\n", "\n", "Performance Metrics:\n", " test_roc_auc: 0.7801\n", " test_f1: 0.5350\n", " test_accuracy: 0.7552\n", "\n", "Cross-Validation:\n", " roc_auc: 0.7854 ± 0.0042\n", "\n", "Training Data: 24000 samples\n", "Test Data: 6000 samples\n" ] } ], "source": [ "print(metadata_v1.summary())" ] }, { "cell_type": "markdown", "id": "md2729c0ba", "metadata": {}, "source": [ "---\n", "## 7. `metadata.to_dict()` — JSON-Serialisable Output" ] }, { "cell_type": "code", "execution_count": 14, "id": "cdf9c5684c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Top-level keys:\n", " name : credit_default_predictor\n", " version : 1.0.0\n", " model_type : LGBMClassifier\n", " framework : lightgbm\n", " task : binary_classification\n", " created_at : 2026-05-06T14:29:13.165864\n", " trained_at : 2026-05-06T14:29:13.165791\n", " hyperparameters : {...} (12 keys)\n", " feature_names : [...] (26 items)\n", " n_features : 26\n", " classes : [0, 1]\n", " metrics : {...} (3 keys)\n", " training_time_seconds : 27.72658348083496\n", " cv_folds : 3\n", " cv_scores : {...} (1 keys)\n", " optimal_threshold : 0.21636717755711785\n", " training_dataset : {...} (12 keys)\n", " validation_dataset : None\n", " test_dataset : {...} (12 keys)\n", " environment : {...} (0 keys)\n", " tags : [...] (4 items)\n", " notes : Credit card default predictor v1.0. Trained on 30k UCI records. Youden-J threshold optimised during training.\n", "\n", "JSON serialisation: OK (7,742 characters)\n" ] } ], "source": [ "meta_dict = metadata_v1.to_dict()\n", "\n", "print(\"Top-level keys:\")\n", "for k, v in meta_dict.items():\n", " if isinstance(v, dict):\n", " print(f\" {k:<28s}: {{...}} ({len(v)} keys)\")\n", " elif isinstance(v, list) and len(v) > 3:\n", " print(f\" {k:<28s}: [...] ({len(v)} items)\")\n", " else:\n", " print(f\" {k:<28s}: {v}\")\n", "\n", "try:\n", " json_str = json.dumps(meta_dict, default=str, indent=2)\n", " print(f\"\\nJSON serialisation: OK ({len(json_str):,} characters)\")\n", "except (TypeError, ValueError) as e:\n", " print(f\"Serialisation failed: {e}\")" ] }, { "cell_type": "markdown", "id": "md2ecbf756", "metadata": {}, "source": [ "---\n", "## 8. `ModelSerializer`: Save Once, Reproduce Anywhere" ] }, { "cell_type": "code", "execution_count": 15, "id": "cd705e3b54", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Files written:\n", " model.pkl 8583.9 KB\n", " model_metadata.json 7.6 KB\n" ] } ], "source": [ "models_dir = Path(\"models\")\n", "models_dir.mkdir(exist_ok=True)\n", "\n", "v1_dir = models_dir / \"credit_default_predictor\" / \"1.0.0\"\n", "v1_dir.mkdir(parents=True, exist_ok=True)\n", "v1_path = str(v1_dir / \"model.pkl\")\n", "\n", "ModelSerializer.save(\n", " model=state_v1.model,\n", " path=v1_path,\n", " metadata=metadata_v1,\n", " train_data=(X_train_t, y_train),\n", " test_data=(X_test_t, y_test),\n", " include_datasets=True\n", ")\n", "\n", "print(\"Files written:\")\n", "for f in sorted(v1_dir.glob(\"*\")):\n", " size_kb = f.stat().st_size / 1024\n", " print(f\" {f.name:<35s} {size_kb:>8.1f} KB\")" ] }, { "cell_type": "markdown", "id": "mdd2b99c79", "metadata": {}, "source": [ "---\n", "## 9. `load_metadata()` — Fast Inspection Without Loading the Model" ] }, { "cell_type": "code", "execution_count": 16, "id": "cd98e8a039", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Metadata loaded (JSON sidecar only — no model deserialisation):\n", " Model name : credit_default_predictor v1.0.0\n", " Task : binary_classification\n", " Framework : lightgbm\n", " Features : 26\n", " Trained at : 2026-05-06T14:29:13.165791\n", "\n", " Recorded metrics:\n", " test_roc_auc : 0.7801\n", " test_f1 : 0.5350\n", " test_accuracy : 0.7552\n" ] } ], "source": [ "fast_meta = ModelSerializer.load_metadata(v1_path)\n", "\n", "print(\"Metadata loaded (JSON sidecar only — no model deserialisation):\")\n", "print(f\" Model name : {fast_meta.get('name')} v{fast_meta.get('version')}\")\n", "print(f\" Task : {fast_meta.get('task')}\")\n", "print(f\" Framework : {fast_meta.get('framework')}\")\n", "print(f\" Features : {fast_meta.get('n_features')}\")\n", "print(f\" Trained at : {fast_meta.get('trained_at')}\")\n", "print()\n", "print(\" Recorded metrics:\")\n", "for metric, val in (fast_meta.get('metrics') or {}).items():\n", " print(f\" {metric:<22s}: {val:.4f}\")" ] }, { "cell_type": "markdown", "id": "md2d279c4c", "metadata": {}, "source": [ "---\n", "## 10. `ModelSerializer.load()` — Full Package Reconstruction" ] }, { "cell_type": "code", "execution_count": 17, "id": "cdaf7622dc", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model: credit_default_predictor (v1.0.0)\n", "Type: LGBMClassifier (lightgbm)\n", "Task: binary_classification\n", "Features: 26\n", "Trained: 2026-05-06 14:29\n", "\n", "Performance Metrics:\n", " test_roc_auc: 0.7801\n", " test_f1: 0.5350\n", " test_accuracy: 0.7552\n", "\n", "Cross-Validation:\n", " roc_auc: 0.7854 ± 0.0042\n", "\n", "Training Data: 24000 samples\n", "Test Data: 6000 samples\n", "\n", "Training data included: 24000 samples\n", "Test data included: 6000 samples\n", "\n", "Package components:\n", " package.model : LGBMClassifierWrapper(fitted, n_features=26)\n", " package.metadata : ModelMetadata(name='credit_default_predictor')\n", " package.train_data : X=(24000, 26) y=(24000,)\n", " package.test_data : X=(6000, 26) y=(6000,)\n" ] } ], "source": [ "package_v1 = ModelSerializer.load(v1_path)\n", "\n", "print(package_v1.summary())\n", "print(\"\\nPackage components:\")\n", "print(f\" package.model : {package_v1.model}\")\n", "print(f\" package.metadata : ModelMetadata(name={package_v1.metadata.name!r})\")\n", "print(f\" package.train_data : X={package_v1.train_data['X'].shape} y={package_v1.train_data['y'].shape}\")\n", "print(f\" package.test_data : X={package_v1.test_data['X'].shape} y={package_v1.test_data['y'].shape}\")" ] }, { "cell_type": "code", "execution_count": 18, "id": "cd4dcda7f9", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[LightGBM] [Warning] feature_fraction is set=0.6082480512336117, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.6082480512336117\n", "[LightGBM] [Warning] lambda_l2 is set=0.026538461705427185, reg_lambda=0.0 will be ignored. Current value: lambda_l2=0.026538461705427185\n", "[LightGBM] [Warning] min_gain_to_split is set=0.7288215433160152, min_split_gain=0.0 will be ignored. Current value: min_gain_to_split=0.7288215433160152\n", "[LightGBM] [Warning] lambda_l1 is set=4.791304580577613e-05, reg_alpha=0.0 will be ignored. Current value: lambda_l1=4.791304580577613e-05\n", "[LightGBM] [Warning] bagging_fraction is set=0.6018969670529957, subsample=1.0 will be ignored. Current value: bagging_fraction=0.6018969670529957\n", "[LightGBM] [Warning] bagging_freq is set=1, subsample_freq=0 will be ignored. Current value: bagging_freq=1\n", "[LightGBM] [Warning] feature_fraction is set=0.6082480512336117, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.6082480512336117\n", "[LightGBM] [Warning] lambda_l2 is set=0.026538461705427185, reg_lambda=0.0 will be ignored. Current value: lambda_l2=0.026538461705427185\n", "[LightGBM] [Warning] min_gain_to_split is set=0.7288215433160152, min_split_gain=0.0 will be ignored. Current value: min_gain_to_split=0.7288215433160152\n", "[LightGBM] [Warning] lambda_l1 is set=4.791304580577613e-05, reg_alpha=0.0 will be ignored. Current value: lambda_l1=4.791304580577613e-05\n", "[LightGBM] [Warning] bagging_fraction is set=0.6018969670529957, subsample=1.0 will be ignored. Current value: bagging_fraction=0.6018969670529957\n", "[LightGBM] [Warning] bagging_freq is set=1, subsample_freq=0 will be ignored. Current value: bagging_freq=1\n", "Round-trip prediction delta (max): 0.00e+00\n", "Prediction integrity: PASS\n" ] } ], "source": [ "proba_original = state_v1.model.predict_proba(X_test_t)[:, 1]\n", "proba_from_disk = package_v1.model.predict_proba(package_v1.test_data['X'])[:, 1]\n", "max_delta = np.abs(proba_original - proba_from_disk).max()\n", "print(f\"Round-trip prediction delta (max): {max_delta:.2e}\")\n", "print(f\"Prediction integrity: {'PASS' if max_delta < 1e-10 else 'FAIL'}\")" ] }, { "cell_type": "markdown", "id": "md6ba2721c", "metadata": {}, "source": [ "---\n", "## 11. `save_model_only()` + `load_model_only()` — Lean Model-Only Artefacts" ] }, { "cell_type": "code", "execution_count": 19, "id": "cd580324b4", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "File size comparison:\n", " Full package (model + data) : 8583.9 KB\n", " Deploy artefact (model only): 1547.9 KB\n", " Size reduction : 82.0%\n", "[LightGBM] [Warning] feature_fraction is set=0.6082480512336117, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.6082480512336117\n", "[LightGBM] [Warning] lambda_l2 is set=0.026538461705427185, reg_lambda=0.0 will be ignored. Current value: lambda_l2=0.026538461705427185\n", "[LightGBM] [Warning] min_gain_to_split is set=0.7288215433160152, min_split_gain=0.0 will be ignored. Current value: min_gain_to_split=0.7288215433160152\n", "[LightGBM] [Warning] lambda_l1 is set=4.791304580577613e-05, reg_alpha=0.0 will be ignored. Current value: lambda_l1=4.791304580577613e-05\n", "[LightGBM] [Warning] bagging_fraction is set=0.6018969670529957, subsample=1.0 will be ignored. Current value: bagging_fraction=0.6018969670529957\n", "[LightGBM] [Warning] bagging_freq is set=1, subsample_freq=0 will be ignored. Current value: bagging_freq=1\n", "\n", "Sample default probabilities: [0.1483 0.1437 0.1475 0.1676 0.0394]\n" ] } ], "source": [ "deploy_path = str(v1_dir / \"model_deploy.pkl\")\n", "ModelSerializer.save_model_only(model=state_v1.model, path=deploy_path)\n", "\n", "full_size = Path(v1_path).stat().st_size / 1024\n", "deploy_size = Path(deploy_path).stat().st_size / 1024\n", "\n", "print(\"File size comparison:\")\n", "print(f\" Full package (model + data) : {full_size:>8.1f} KB\")\n", "print(f\" Deploy artefact (model only): {deploy_size:>8.1f} KB\")\n", "print(f\" Size reduction : {(1 - deploy_size/full_size)*100:.1f}%\")\n", "\n", "inference_model = ModelSerializer.load_model_only(deploy_path)\n", "sample_scores = inference_model.predict_proba(X_test_t.iloc[:5])[:, 1]\n", "print(f\"\\nSample default probabilities: {sample_scores.round(4)}\")" ] }, { "cell_type": "markdown", "id": "mdcf3fe859", "metadata": {}, "source": [ "---\n", "## 12. `ModelRegistry`: A Catalogue of Your ML Arsenal" ] }, { "cell_type": "code", "execution_count": 20, "id": "cda4ab06e2", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Registered model wrappers (4 total):\n", " lgbm_classifier -> bitbullet.model.wrappers.lgbm_wrapper.LGBMClassifierWrapper\n", " lgbm_regressor -> bitbullet.model.wrappers.lgbm_wrapper.LGBMRegressorWrapper\n", " xgb_classifier -> bitbullet.model.wrappers.xgb_wrapper.XGBClassifierWrapper\n", " xgb_regressor -> bitbullet.model.wrappers.xgb_wrapper.XGBRegressorWrapper\n" ] } ], "source": [ "from bitbullet.model.wrappers import lgbm_wrapper, xgb_wrapper\n", "\n", "all_models = ModelRegistry.list_models()\n", "print(f\"Registered model wrappers ({len(all_models)} total):\")\n", "for name, cls in sorted(all_models.items()):\n", " print(f\" {name:<30s} -> {cls.__module__}.{cls.__name__}\")" ] }, { "cell_type": "code", "execution_count": 21, "id": "cda19b0440", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " lgbm_classifier : registered\n", " lgbm_regressor : registered\n", " xgb_classifier : registered\n", " nonexistent_model : not found\n" ] } ], "source": [ "for name in [\"lgbm_classifier\", \"lgbm_regressor\", \"xgb_classifier\", \"nonexistent_model\"]:\n", " status = \"registered\" if ModelRegistry.is_registered(name) else \"not found\"\n", " print(f\" {name:<30s}: {status}\")" ] }, { "cell_type": "code", "execution_count": 22, "id": "cd3870ea7c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Created via factory:\n", " Type : LGBMClassifierWrapper\n", " Is fitted : False\n" ] } ], "source": [ "factory_model = ModelRegistry.create(\"lgbm_classifier\", n_estimators=100, max_depth=5, learning_rate=0.05)\n", "print(f\"Created via factory:\")\n", "print(f\" Type : {type(factory_model).__name__}\")\n", "print(f\" Is fitted : {factory_model.is_fitted}\")" ] }, { "cell_type": "markdown", "id": "md6d57227a", "metadata": {}, "source": [ "---\n", "## 13. `@ModelRegistry.register` — Custom Model Wrappers" ] }, { "cell_type": "code", "execution_count": 23, "id": "cd54dbf30c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Registered: True\n", "Registry now contains 5 model types\n", "\n", "Logistic Regression Test AUC: 0.7316\n" ] } ], "source": [ "from sklearn.linear_model import LogisticRegression\n", "from datetime import datetime as dt\n", "\n", "@ModelRegistry.register(\"logistic_regression_custom\")\n", "class LogisticRegressionWrapper(BaseModel):\n", " def __init__(self, **kwargs):\n", " super().__init__(**kwargs)\n", " self._state.model_type = \"LogisticRegression\"\n", " self._state.framework = \"sklearn\"\n", " self._lr_model = None\n", "\n", " def fit(self, X, y, **kwargs):\n", " self._state.feature_names = self._extract_feature_names(X)\n", " self._state.n_features = len(self._state.feature_names)\n", " self._lr_model = LogisticRegression(**self._hyperparameters)\n", " self._lr_model.fit(X, y)\n", " self._state.model = self._lr_model\n", " self._state.classes = self._lr_model.classes_\n", " self._state.is_fitted = True\n", " self._state.fitted_at = dt.now()\n", " return self\n", "\n", " def predict(self, X):\n", " return self._lr_model.predict(X)\n", "\n", " def predict_proba(self, X):\n", " return self._lr_model.predict_proba(X)\n", "\n", "\n", "print(f\"Registered: {ModelRegistry.is_registered('logistic_regression_custom')}\")\n", "print(f\"Registry now contains {len(ModelRegistry.list_models())} model types\")\n", "\n", "lr_model = ModelRegistry.create(\"logistic_regression_custom\", max_iter=1000, C=1.0)\n", "lr_model.fit(X_train_t, y_train)\n", "lr_auc = roc_auc_score(y_test, lr_model.predict_proba(X_test_t)[:, 1])\n", "print(f\"\\nLogistic Regression Test AUC: {lr_auc:.4f}\")" ] }, { "cell_type": "markdown", "id": "md5b436a73", "metadata": {}, "source": [ "---\n", "\n", "## 14. Version Management — Semantic Versioning Pattern\n", "\n", "```\n", "models/\n", " credit_default_predictor/\n", " 1.0.0/\n", " model.pkl\n", " model_metadata.json\n", " model_deploy.pkl\n", " 2.0.0/\n", " model.pkl\n", " ...\n", "```\n", "\n", "Each version directory is **immutable** once promoted. Rolling back means pointing your serving layer at the previous directory." ] }, { "cell_type": "markdown", "id": "mdf1eb5153", "metadata": {}, "source": [ "---\n", "## 15. Training V2 — Simulating a Model Improvement" ] }, { "cell_type": "code", "execution_count": null, "id": "cd10c0328c", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[I 2026-05-06 14:29:35,083] A new study created in memory with name: credit_default_lgbm_v2\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "======================================================================\n", "BitBullet Train - credit_default_lgbm_v2\n", "======================================================================\n", "\n", "Training samples: 24000\n", "Features: 26\n", "Class distribution: {0: 18691, 1: 5309}\n", "\n", "Starting hyperparameter optimization (optuna)...\n", "Trials: 10, CV Folds: 3\n", "\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[I 2026-05-06 14:29:39,347] Trial 5 finished with value: 0.782912773978182 and parameters: {'num_leaves': 21, 'max_depth': 12, 'min_child_samples': 51, 'lambda_l1': 3.198129938918566e-05, 'lambda_l2': 1.2146761448930005e-05, 'min_gain_to_split': 0.8902704104749739, 'feature_fraction': 0.564218275579441, 'bagging_fraction': 0.8930131995427238, 'bagging_freq': 5, 'learning_rate': 0.1838055561830655, 'max_bin': 223}. Best is trial 5 with value: 0.782912773978182.\n", "[I 2026-05-06 14:29:43,744] Trial 3 finished with value: 0.7840345942057775 and parameters: {'num_leaves': 35, 'max_depth': 7, 'min_child_samples': 5, 'lambda_l1': 1.2668443686277898e-05, 'lambda_l2': 6.667569170828315e-05, 'min_gain_to_split': 0.46512943642656523, 'feature_fraction': 0.6950636679489371, 'bagging_fraction': 0.6368559328840082, 'bagging_freq': 3, 'learning_rate': 0.05814625308392654, 'max_bin': 95}. Best is trial 3 with value: 0.7840345942057775.\n", "[I 2026-05-06 14:29:45,129] Trial 6 finished with value: 0.7835014787792075 and parameters: {'num_leaves': 185, 'max_depth': 6, 'min_child_samples': 7, 'lambda_l1': 1.7317952674829006e-06, 'lambda_l2': 0.00015569105138337706, 'min_gain_to_split': 0.5991286983102695, 'feature_fraction': 0.6197110841995219, 'bagging_fraction': 0.9557092041573234, 'bagging_freq': 1, 'learning_rate': 0.05681276382860915, 'max_bin': 255}. Best is trial 3 with value: 0.7840345942057775.\n", "[I 2026-05-06 14:29:45,368] Trial 7 finished with value: 0.7830796797480789 and parameters: {'num_leaves': 110, 'max_depth': 6, 'min_child_samples': 6, 'lambda_l1': 8.42874454972082e-05, 'lambda_l2': 0.6566089948571227, 'min_gain_to_split': 0.2818247902754588, 'feature_fraction': 0.6432105790897444, 'bagging_fraction': 0.898022314047384, 'bagging_freq': 3, 'learning_rate': 0.05522080967791649, 'max_bin': 127}. Best is trial 3 with value: 0.7840345942057775.\n", "[I 2026-05-06 14:29:57,950] Trial 4 finished with value: 0.7846914017570068 and parameters: {'num_leaves': 236, 'max_depth': 11, 'min_child_samples': 34, 'lambda_l1': 1.668755606473228, 'lambda_l2': 3.931392110336462e-06, 'min_gain_to_split': 0.7051414683324333, 'feature_fraction': 0.5932723358063668, 'bagging_fraction': 0.5993729128275531, 'bagging_freq': 5, 'learning_rate': 0.030082493445507202, 'max_bin': 159}. Best is trial 4 with value: 0.7846914017570068.\n", "[I 2026-05-06 14:30:06,118] Trial 9 finished with value: 0.7839114270594033 and parameters: {'num_leaves': 86, 'max_depth': 6, 'min_child_samples': 18, 'lambda_l1': 1.5940386030280055e-05, 'lambda_l2': 1.2289642680019808e-06, 'min_gain_to_split': 0.08463529119537327, 'feature_fraction': 0.9135673145468102, 'bagging_fraction': 0.5924639929700057, 'bagging_freq': 7, 'learning_rate': 0.007870767412293792, 'max_bin': 223}. Best is trial 4 with value: 0.7846914017570068.\n", "[I 2026-05-06 14:30:07,638] Trial 0 finished with value: 0.7829746167981749 and parameters: {'num_leaves': 112, 'max_depth': 5, 'min_child_samples': 9, 'lambda_l1': 5.850246529282143e-08, 'lambda_l2': 1.3632684578775255e-05, 'min_gain_to_split': 0.1971884367643605, 'feature_fraction': 0.954505357601366, 'bagging_fraction': 0.9024055258424614, 'bagging_freq': 4, 'learning_rate': 0.005873253028943797, 'max_bin': 95}. Best is trial 4 with value: 0.7846914017570068.\n", "[I 2026-05-06 14:30:09,042] Trial 8 finished with value: 0.7789455830066881 and parameters: {'num_leaves': 140, 'max_depth': 11, 'min_child_samples': 9, 'lambda_l1': 2.8084078692392386e-07, 'lambda_l2': 0.0019625746887455314, 'min_gain_to_split': 0.25085040380737944, 'feature_fraction': 0.9920936949061228, 'bagging_fraction': 0.8439101897417567, 'bagging_freq': 1, 'learning_rate': 0.022712821246251496, 'max_bin': 223}. Best is trial 4 with value: 0.7846914017570068.\n", "[I 2026-05-06 14:30:10,284] Trial 2 finished with value: 0.7845402427830183 and parameters: {'num_leaves': 133, 'max_depth': 9, 'min_child_samples': 35, 'lambda_l1': 2.3501673550543083e-05, 'lambda_l2': 0.0012381674477924278, 'min_gain_to_split': 0.5803657748940718, 'feature_fraction': 0.5393470052772198, 'bagging_fraction': 0.7630633112773256, 'bagging_freq': 4, 'learning_rate': 0.014261890626007459, 'max_bin': 191}. Best is trial 4 with value: 0.7846914017570068.\n" ] } ], "source": [ "config_v2 = TrainConfig(\n", " name=\"credit_default_lgbm_v2\",\n", " model_type=\"lgbm\",\n", " task=\"binary_classification\",\n", " optimization_metric=\"roc_auc\",\n", " optuna_sampler=\"tpe\",\n", " n_trials=10,\n", " cv_folds=3,\n", " optimize_threshold=True,\n", " threshold_optimization_method=\"youden\",\n", " save_feature_importance=True,\n", " generate_shap=False,\n", " verbose=True,\n", " optuna_show_progress=False,\n", " random_state=99 # Different seed — different Optuna search trajectory\n", ")\n", "\n", "t0 = time.time()\n", "trainer_v2 = OptunaTrainer(config=config_v2)\n", "trainer_v2.fit(X_train_t, y_train)\n", "state_v2 = trainer_v2.state\n", "print(f\"V2 training complete in {time.time() - t0:.1f}s\")" ] }, { "cell_type": "code", "execution_count": null, "id": "cd05ab11b0", "metadata": {}, "outputs": [], "source": [ "y_proba_v2 = state_v2.model.predict_proba(X_test_t)[:, 1]\n", "threshold_v2 = state_v2.optimal_threshold or 0.5\n", "y_pred_v2 = (y_proba_v2 >= threshold_v2).astype(int)\n", "\n", "test_roc_auc_v2 = roc_auc_score(y_test, y_proba_v2)\n", "test_f1_v2 = f1_score(y_test, y_pred_v2)\n", "test_accuracy_v2 = accuracy_score(y_test, y_pred_v2)\n", "\n", "print(f\"V2 Test Metrics:\")\n", "print(f\" ROC AUC : {test_roc_auc_v2:.4f}\")\n", "print(f\" F1 Score : {test_f1_v2:.4f}\")\n", "print(f\" Accuracy : {test_accuracy_v2:.4f}\")" ] }, { "cell_type": "code", "execution_count": null, "id": "cd1d528ae6", "metadata": {}, "outputs": [], "source": [ "metadata_v2 = ModelMetadata(\n", " name=\"credit_default_predictor\",\n", " version=\"2.0.0\",\n", " model_type=state_v2.model.model_type,\n", " framework=state_v2.model.framework,\n", " task=\"binary_classification\",\n", " trained_at=datetime.now(),\n", " feature_names=list(X_train_t.columns),\n", " n_features=X_train_t.shape[1],\n", " hyperparameters=state_v2.best_params,\n", " classes=[0, 1],\n", " optimal_threshold=threshold_v2,\n", " training_time_seconds=state_v2.training_time_seconds,\n", " tags=[\"binary_classification\", \"credit_default\", \"lightgbm\", \"uci\"],\n", " notes=\"Credit default predictor v2.0. Retrained with different random seed.\"\n", ")\n", "test_report_v2 = evaluate_classification(\n", " y_true=y_test,\n", " y_pred_proba=y_proba_v2,\n", " threshold=threshold_v2,\n", " labels=[0, 1],\n", ")\n", "metadata_v2.add_metric(\"test_roc_auc\", test_report_v2.metrics[\"roc_auc\"])\n", "metadata_v2.add_metric(\"test_f1\", test_report_v2.metrics[\"f1_score\"])\n", "metadata_v2.add_metric(\"test_accuracy\", test_report_v2.metrics[\"accuracy\"])\n", "metadata_v2.add_feature_schema(X_train_t)\n", "metadata_v2.preprocessing = metadata_v1.preprocessing.copy()\n", "metadata_v2.search_metadata = {\n", " \"optimizer\": config_v2.optimizer,\n", " \"optuna_sampler\": config_v2.optuna_sampler,\n", " \"n_trials\": config_v2.n_trials,\n", " \"cv_folds\": config_v2.cv_folds,\n", " \"optimization_metric\": config_v2.optimization_metric,\n", " \"best_params\": state_v2.best_params,\n", "}\n", "metadata_v2.target_mapping = metadata_v1.target_mapping.copy()\n", "metadata_v2.inference_contract = {\n", " **metadata_v1.inference_contract,\n", " \"decision_threshold\": threshold_v2,\n", "}\n", "metadata_v2.add_artifact_metadata(classification_metrics=test_report_v2.to_dict())\n", "\n", "if state_v2.study:\n", " fold_scores_v2 = state_v2.study.best_trial.user_attrs.get('cv_scores', [state_v2.best_score])\n", "else:\n", " fold_scores_v2 = state_v2.cv_scores if state_v2.cv_scores else [state_v2.best_score]\n", "\n", "metadata_v2.add_cv_scores(\"roc_auc\", fold_scores_v2)\n", "metadata_v2.add_dataset(X_train_t, y_train, \"train\")\n", "metadata_v2.add_dataset(X_test_t, y_test, \"test\")\n", "\n", "v2_dir = models_dir / \"credit_default_predictor\" / \"2.0.0\"\n", "v2_dir.mkdir(parents=True, exist_ok=True)\n", "v2_path = str(v2_dir / \"model.pkl\")\n", "\n", "ModelSerializer.save(\n", " model=state_v2.model,\n", " path=v2_path,\n", " metadata=metadata_v2,\n", " train_data=(X_train_t, y_train),\n", " test_data=(X_test_t, y_test),\n", " include_datasets=True\n", ")\n", "print(\"V2 saved.\")\n", "for f in sorted((models_dir / \"credit_default_predictor\").rglob(\"*\")):\n", " if f.is_file():\n", " indent = \" \" * (len(f.relative_to(models_dir).parts) - 1)\n", " print(f\" {indent}{f.name} ({f.stat().st_size / 1024:.1f} KB)\")" ] }, { "cell_type": "markdown", "id": "md7fb9bf18", "metadata": {}, "source": [ "---\n", "## 16. Comparing V1 vs V2 — Metadata-Driven Version Diffing" ] }, { "cell_type": "code", "execution_count": null, "id": "cd2cfb315e", "metadata": {}, "outputs": [], "source": [ "v1_meta_dict = ModelSerializer.load_metadata(v1_path)\n", "v2_meta_dict = ModelSerializer.load_metadata(v2_path)\n", "\n", "v1_metrics = v1_meta_dict.get('metrics', {})\n", "v2_metrics = v2_meta_dict.get('metrics', {})\n", "all_metric_names = sorted(set(v1_metrics) | set(v2_metrics))\n", "\n", "print(f\"{'Metric':<22s} {'V1':>8s} {'V2':>8s} {'Delta':>10s} Direction\")\n", "print(\"-\" * 62)\n", "for metric in all_metric_names:\n", " v1_val = v1_metrics.get(metric)\n", " v2_val = v2_metrics.get(metric)\n", " if v1_val is None or v2_val is None:\n", " continue\n", " delta = v2_val - v1_val\n", " direction = \"IMPROVED\" if delta > 0 else (\"REGRESSED\" if delta < 0 else \"UNCHANGED\")\n", " arrow = \"^\" if delta > 0 else (\"v\" if delta < 0 else \"=\")\n", " print(f\"{metric:<22s} {v1_val:>8.4f} {v2_val:>8.4f} {delta:>+10.4f} {arrow} {direction}\")" ] }, { "cell_type": "code", "execution_count": null, "id": "cda3413e4c", "metadata": {}, "outputs": [], "source": [ "v1_cv = v1_meta_dict.get('cv_scores', {}).get('roc_auc', [])\n", "v2_cv = v2_meta_dict.get('cv_scores', {}).get('roc_auc', [])\n", "\n", "if v1_cv and v2_cv:\n", " print(f\"V1 CV ({len(v1_cv)} folds): {[round(s, 4) for s in v1_cv]} mean={np.mean(v1_cv):.4f}\")\n", " print(f\"V2 CV ({len(v2_cv)} folds): {[round(s, 4) for s in v2_cv]} mean={np.mean(v2_cv):.4f}\")" ] }, { "cell_type": "markdown", "id": "md7bccbfc9", "metadata": {}, "source": [ "## 17. Visualising the V1 vs V2 Metric Comparison" ] }, { "cell_type": "code", "execution_count": null, "id": "cde58bbd29", "metadata": {}, "outputs": [], "source": [ "BLUE = \"#2563EB\"\n", "GREEN = \"#16A34A\"\n", "RED = \"#DC2626\"\n", "\n", "metric_labels = [m.replace(\"test_\", \"\").upper() for m in all_metric_names]\n", "v1_vals = [v1_metrics.get(m, 0) for m in all_metric_names]\n", "v2_vals = [v2_metrics.get(m, 0) for m in all_metric_names]\n", "n_metrics = len(all_metric_names)\n", "x = np.arange(n_metrics)\n", "bar_w = 0.35\n", "\n", "fig, axes = plt.subplots(1, 2, figsize=(13, 5))\n", "fig.suptitle(\"credit_default_predictor: V1 vs V2 Performance Comparison\",\n", " fontsize=13, fontweight=\"bold\")\n", "\n", "ax = axes[0]\n", "bars1 = ax.bar(x - bar_w/2, v1_vals, bar_w, label=\"v1.0.0\", color=BLUE, alpha=0.85)\n", "bars2 = ax.bar(x + bar_w/2, v2_vals, bar_w, label=\"v2.0.0\", color=GREEN, alpha=0.85)\n", "for bar in list(bars1) + list(bars2):\n", " ax.text(bar.get_x() + bar.get_width() / 2, bar.get_height() + 0.002,\n", " f\"{bar.get_height():.3f}\", ha='center', va='bottom', fontsize=8)\n", "ax.set_xticks(x)\n", "ax.set_xticklabels(metric_labels, fontsize=10)\n", "y_min = max(0, min(v1_vals + v2_vals) - 0.05)\n", "ax.set_ylim(y_min, min(1.05, max(v1_vals + v2_vals) + 0.07))\n", "ax.set_title(\"Test Metrics by Version\", fontweight=\"bold\")\n", "ax.set_ylabel(\"Score\")\n", "ax.legend()\n", "ax.grid(axis='y', alpha=0.3)\n", "\n", "ax = axes[1]\n", "deltas = [v2 - v1 for v1, v2 in zip(v1_vals, v2_vals)]\n", "colors = [GREEN if d >= 0 else RED for d in deltas]\n", "bars = ax.bar(x, deltas, bar_w * 2, color=colors, alpha=0.85)\n", "for bar, delta in zip(bars, deltas):\n", " va = 'bottom' if delta >= 0 else 'top'\n", " ypos = bar.get_height() + 0.0005 if delta >= 0 else bar.get_height() - 0.0005\n", " ax.text(bar.get_x() + bar.get_width() / 2, ypos,\n", " f\"{delta:+.4f}\", ha='center', va=va, fontsize=9, fontweight='bold')\n", "ax.axhline(0, color='black', lw=0.8)\n", "ax.set_xticks(x)\n", "ax.set_xticklabels(metric_labels, fontsize=10)\n", "ax.set_title(\"V2 vs V1 Delta\", fontweight=\"bold\")\n", "ax.grid(axis='y', alpha=0.3)\n", "\n", "plt.tight_layout()\n", "plt.show()\n", "\n", "auc_delta = v2_metrics.get('test_roc_auc', 0) - v1_metrics.get('test_roc_auc', 0)\n", "if auc_delta > 0.005:\n", " print(\"Recommendation: PROMOTE v2 — meaningful AUC improvement\")\n", "elif auc_delta < -0.005:\n", " print(\"Recommendation: DO NOT PROMOTE — AUC regression vs v1\")\n", "else:\n", " print(\"Recommendation: NEUTRAL — delta within noise threshold (< 0.005)\")" ] }, { "cell_type": "markdown", "id": "md40aaedc2", "metadata": {}, "source": [ "---\n", "## 18. Production Checklist" ] }, { "cell_type": "code", "execution_count": null, "id": "cdafe2b88c", "metadata": {}, "outputs": [], "source": [ "def pre_promotion_check(model_path: str, min_roc_auc: float = 0.70) -> bool:\n", " checks = {}\n", "\n", " try:\n", " meta = ModelSerializer.load_metadata(model_path)\n", " checks[\"metadata_readable\"] = True\n", " except Exception as e:\n", " print(f\" FAIL metadata_readable: {e}\")\n", " return False\n", "\n", " auc = (meta.get('metrics') or {}).get('test_roc_auc', 0)\n", " checks[\"auc_floor\"] = auc >= min_roc_auc\n", " checks[\"feature_contract\"] = bool(meta.get('feature_names'))\n", " checks[\"threshold_set\"] = meta.get('optimal_threshold') is not None\n", " checks[\"data_hash_present\"] = bool((meta.get('training_dataset') or {}).get('data_hash'))\n", " checks[\"notes_populated\"] = bool((meta.get('notes') or '').strip())\n", "\n", " try:\n", " pkg = ModelSerializer.load(model_path)\n", " if pkg.test_data:\n", " p1 = pkg.model.predict_proba(pkg.test_data['X'])[:, 1]\n", " p2 = pkg.model.predict_proba(pkg.test_data['X'])[:, 1]\n", " checks[\"round_trip\"] = np.abs(p1 - p2).max() < 1e-10\n", " else:\n", " checks[\"round_trip\"] = None\n", " except Exception as e:\n", " checks[\"round_trip\"] = False\n", "\n", " print(f\"Pre-promotion checks for: {model_path}\")\n", " print(f\" Min AUC floor: {min_roc_auc} | Actual AUC: {auc:.4f}\")\n", " print()\n", " all_pass = True\n", " for check, result in checks.items():\n", " symbol = \"SKIP\" if result is None else (\"PASS\" if result else \"FAIL\")\n", " if result is False:\n", " all_pass = False\n", " print(f\" [{symbol}] {check}\")\n", "\n", " print()\n", " print(f\" Overall: {'READY TO PROMOTE' if all_pass else 'BLOCKED — fix failures above'}\")\n", " return all_pass\n", "\n", "print(\"Running pre-promotion checks on V2 model:\\n\")\n", "result = pre_promotion_check(v2_path, min_roc_auc=0.70)" ] }, { "cell_type": "markdown", "id": "mdeab2aaba", "metadata": {}, "source": [ "---\n", "## 19. Cleanup" ] }, { "cell_type": "code", "execution_count": null, "id": "cde9b3bd3a", "metadata": {}, "outputs": [], "source": [ "shutil.rmtree(\"models\", ignore_errors=True)\n", "print(\"Demo artefacts removed.\")" ] }, { "cell_type": "markdown", "id": "mdd4eb804b", "metadata": {}, "source": [ "---\n", "\n", "## 20. Conclusion\n", "\n", "| Problem | BitBullet Solution |\n", "|---------|--------------------|\n", "| Lost model lineage | `ModelMetadata` captures identity, features, hyperparameters, metrics, dataset hashes |\n", "| Unreproducible evaluations | Test metrics locked into metadata after evaluation |\n", "| Silent data drift | `data_hash` — cryptographic fingerprint of every training dataset |\n", "| Expensive metadata scans | `_metadata.json` sidecar — compare versions without loading pickle |\n", "| Oversized experiment packages | `save_model_only()` + `load_model_only()` for lean model-only artefacts |\n", "| Architecture coupling | `ModelRegistry` — create any model by name |\n", "| No rollback | Versioned directory structure — previous versions always loadable |\n", "\n", "**Continue the Academy:**\n", "- `01_Binary_Classification.ipynb` — end-to-end binary classification with SHAP and formal reports\n", "- `02_Multi_Class_Classification.ipynb` — extending the workflow to seven-class targets\n", "- `03_Clustering.ipynb` — unsupervised wholesale customer segmentation\n", "- `04_Data_Transformations.ipynb` — deep dive into every transform in the library" ] } ], "metadata": { "kernelspec": { "display_name": "bitbullet", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.13" } }, "nbformat": 4, "nbformat_minor": 5 }