BitBullet:Lessons: Model Management¶
You trained a great model. Now what?
In most teams, that question is answered with a folder called models/ containing files named final_v3_USE_THIS.pkl, a Slack message with the test AUC, and a vague memory of what preprocessing was applied. Six months later, no one can reproduce the result, the model is silently serving stale predictions, and rolling back to the previous version means digging through git blame.
This is model rot. And it is entirely preventable.
Dataset: default_of_credit_card_clients.xls — same dataset as Lesson 04.
Source: UCI ML Repository — Default of Credit Card Clients
What This Tutorial Covers¶
| Section | Component | What You Will Learn |
|---|---|---|
| 1 | Setup | Imports and lightweight training run |
| 2 | ModelMetadata |
Capturing identity, hyperparameters, metrics, and provenance |
| 3 | DatasetMetadata |
Recording what you trained on — including cryptographic hashing |
| 4 | metadata.summary() |
Human-readable model cards |
| 5 | metadata.to_dict() |
JSON-serialisable output for logging |
| 6 | ModelSerializer.save() |
Full packages — one artefact, everything inside |
| 7 | load_metadata() |
Fast metadata inspection without loading the model |
| 8 | ModelSerializer.load() |
Reconstructing the full package from disk |
| 9 | save_model_only() |
Lean model-only artefacts |
| 10 | ModelRegistry |
A typed catalogue of every model architecture |
| 11 | @ModelRegistry.register |
Extending BitBullet with custom model wrappers |
| 12 | Version management | Semantic versioning patterns |
| 13 | V1 vs V2 comparison | Loading, diffing, and visualising metric deltas |
| 14 | Production checklist | What to verify before promoting a model |
1. Environment Setup & Imports¶
In [1]:
Copied!
import sys
import os
import json
import shutil
import warnings
import time
from datetime import datetime
from pathlib import Path
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.ticker as mticker
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score, f1_score, accuracy_score
warnings.filterwarnings('ignore')
# When running this notebook from bitbullet/lessons in a local clone, prefer the local SDK source.
sdk_root = os.path.abspath("..")
if sdk_root not in sys.path:
sys.path.insert(0, sdk_root)
import bitbullet
print(f"bitbullet : v{getattr(bitbullet, '__version__', 'dev')}")
from bitbullet.model import (
ModelMetadata,
DatasetMetadata,
ModelSerializer,
ModelRegistry,
BaseModel,
)
from bitbullet.transform import TransformPipeline
from bitbullet.train import TrainConfig, OptunaTrainer
print("All imports successful.")
import sys
import os
import json
import shutil
import warnings
import time
from datetime import datetime
from pathlib import Path
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.ticker as mticker
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score, f1_score, accuracy_score
warnings.filterwarnings('ignore')
# When running this notebook from bitbullet/lessons in a local clone, prefer the local SDK source.
sdk_root = os.path.abspath("..")
if sdk_root not in sys.path:
sys.path.insert(0, sdk_root)
import bitbullet
print(f"bitbullet : v{getattr(bitbullet, '__version__', 'dev')}")
from bitbullet.model import (
ModelMetadata,
DatasetMetadata,
ModelSerializer,
ModelRegistry,
BaseModel,
)
from bitbullet.transform import TransformPipeline
from bitbullet.train import TrainConfig, OptunaTrainer
print("All imports successful.")
bitbullet : vdev All imports successful.
2. Quick Training Run¶
Dataset: Default of Credit Card Clients Dataset
Source: UCI ML Repository — Default of Credit Card Clients
File: default_of_credit_card_clients.xls
Before running this cell: download
default_of_credit_card_clients.xlsfrom the link above and place it in the same directory as this notebook. If you store it elsewhere, updatedata_pathin the code cell below to match your chosen location.
In [ ]:
Copied!
# Update this path if you stored the file in a different location.
data_path = "default_of_credit_card_clients.xls"
df_raw = pd.read_excel(data_path, header=1)
df = df_raw.drop(columns=['ID']).rename(columns={'default payment next month': 'default_payment'})
df['SEX'] = df['SEX'].map({1: 'Male', 2: 'Female'})
df['EDUCATION'] = df['EDUCATION'].map({1: 'Graduate', 2: 'University', 3: 'High_School',
4: 'Other', 5: 'Other', 6: 'Other', 0: 'Other'})
df['MARRIAGE'] = df['MARRIAGE'].map({1: 'Married', 2: 'Single', 3: 'Other', 0: 'Other'})
target_col = 'default_payment'
X = df.drop(columns=[target_col])
y = df[target_col]
print(f"Dataset loaded — shape: {df.shape}")
print(f"Default rate — {y.mean():.1%}")
display(df.head(3))
# Update this path if you stored the file in a different location.
data_path = "default_of_credit_card_clients.xls"
df_raw = pd.read_excel(data_path, header=1)
df = df_raw.drop(columns=['ID']).rename(columns={'default payment next month': 'default_payment'})
df['SEX'] = df['SEX'].map({1: 'Male', 2: 'Female'})
df['EDUCATION'] = df['EDUCATION'].map({1: 'Graduate', 2: 'University', 3: 'High_School',
4: 'Other', 5: 'Other', 6: 'Other', 0: 'Other'})
df['MARRIAGE'] = df['MARRIAGE'].map({1: 'Married', 2: 'Single', 3: 'Other', 0: 'Other'})
target_col = 'default_payment'
X = df.drop(columns=[target_col])
y = df[target_col]
print(f"Dataset loaded — shape: {df.shape}")
print(f"Default rate — {y.mean():.1%}")
display(df.head(3))
Dataset loaded — shape: (30000, 24) Default rate — 22.1%
| LIMIT_BAL | SEX | EDUCATION | MARRIAGE | AGE | PAY_0 | PAY_2 | PAY_3 | PAY_4 | PAY_5 | ... | BILL_AMT4 | BILL_AMT5 | BILL_AMT6 | PAY_AMT1 | PAY_AMT2 | PAY_AMT3 | PAY_AMT4 | PAY_AMT5 | PAY_AMT6 | default_payment | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 20000 | Female | University | Married | 24 | 2 | 2 | -1 | -1 | -2 | ... | 0 | 0 | 0 | 0 | 689 | 0 | 0 | 0 | 0 | 1 |
| 1 | 120000 | Female | University | Single | 26 | -1 | 2 | 0 | 0 | 0 | ... | 3272 | 3455 | 3261 | 0 | 1000 | 1000 | 1000 | 0 | 2000 | 1 |
| 2 | 90000 | Female | University | Single | 34 | 0 | 0 | 0 | 0 | 0 | ... | 14331 | 14948 | 15549 | 1518 | 1500 | 1000 | 1000 | 1000 | 5000 | 0 |
3 rows × 24 columns
In [6]:
Copied!
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.20, random_state=42, stratify=y
)
print(f"Training : {X_train.shape[0]:,} samples (default rate: {y_train.mean():.1%})")
print(f"Test : {X_test.shape[0]:,} samples (default rate: {y_test.mean():.1%})")
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.20, random_state=42, stratify=y
)
print(f"Training : {X_train.shape[0]:,} samples (default rate: {y_train.mean():.1%})")
print(f"Test : {X_test.shape[0]:,} samples (default rate: {y_test.mean():.1%})")
Training : 24,000 samples (default rate: 22.1%) Test : 6,000 samples (default rate: 22.1%)
In [7]:
Copied!
pay_amt_cols = [f'PAY_AMT{i}' for i in range(1, 7)]
bill_amt_cols = [f'BILL_AMT{i}' for i in range(1, 7)]
pay_cols = ['PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6']
pipeline_v1 = (
TransformPipeline(name="credit_default_pipeline_v1")
.add("numerical", "log1p", columns=pay_amt_cols + ['LIMIT_BAL'])
.add("numerical", "yeo_johnson", columns=bill_amt_cols)
.add("numerical", "standard_scale", columns=["AGE"])
.add("numerical", "robust_scale", columns=pay_cols)
.add("categorical", "label_encode", columns=['SEX', 'MARRIAGE'],
params={"unknown_strategy": "use_encoded_value", "unknown_value": -1})
.add("categorical", "onehot_encode", columns=['EDUCATION'],
params={"sparse": False, "drop": None})
)
X_train_t = pipeline_v1.fit_transform(X_train)
X_test_t = pipeline_v1.transform(X_test)
print(f"Transform applied — {X_train.shape[1]} raw features \u2192 {X_train_t.shape[1]} engineered")
pay_amt_cols = [f'PAY_AMT{i}' for i in range(1, 7)]
bill_amt_cols = [f'BILL_AMT{i}' for i in range(1, 7)]
pay_cols = ['PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6']
pipeline_v1 = (
TransformPipeline(name="credit_default_pipeline_v1")
.add("numerical", "log1p", columns=pay_amt_cols + ['LIMIT_BAL'])
.add("numerical", "yeo_johnson", columns=bill_amt_cols)
.add("numerical", "standard_scale", columns=["AGE"])
.add("numerical", "robust_scale", columns=pay_cols)
.add("categorical", "label_encode", columns=['SEX', 'MARRIAGE'],
params={"unknown_strategy": "use_encoded_value", "unknown_value": -1})
.add("categorical", "onehot_encode", columns=['EDUCATION'],
params={"sparse": False, "drop": None})
)
X_train_t = pipeline_v1.fit_transform(X_train)
X_test_t = pipeline_v1.transform(X_test)
print(f"Transform applied — {X_train.shape[1]} raw features \u2192 {X_train_t.shape[1]} engineered")
Transform applied — 23 raw features → 26 engineered
In [8]:
Copied!
# n_trials=10, cv_folds=3 — fast demo run. Production: use >=100 trials, 5 folds.
config_v1 = TrainConfig(
name="credit_default_lgbm_v1",
model_type="lgbm",
task="binary_classification",
optimization_metric="roc_auc",
optuna_sampler="tpe",
n_trials=10,
cv_folds=3,
optimize_threshold=True,
threshold_optimization_method="youden",
save_feature_importance=True,
generate_shap=False,
verbose=True,
optuna_show_progress=False,
random_state=42
)
t0 = time.time()
trainer_v1 = OptunaTrainer(config=config_v1)
trainer_v1.fit(X_train_t, y_train)
state_v1 = trainer_v1.state
print(f"\nTraining complete in {time.time() - t0:.1f}s")
print(state_v1.summary())
# n_trials=10, cv_folds=3 — fast demo run. Production: use >=100 trials, 5 folds.
config_v1 = TrainConfig(
name="credit_default_lgbm_v1",
model_type="lgbm",
task="binary_classification",
optimization_metric="roc_auc",
optuna_sampler="tpe",
n_trials=10,
cv_folds=3,
optimize_threshold=True,
threshold_optimization_method="youden",
save_feature_importance=True,
generate_shap=False,
verbose=True,
optuna_show_progress=False,
random_state=42
)
t0 = time.time()
trainer_v1 = OptunaTrainer(config=config_v1)
trainer_v1.fit(X_train_t, y_train)
state_v1 = trainer_v1.state
print(f"\nTraining complete in {time.time() - t0:.1f}s")
print(state_v1.summary())
[I 2026-05-06 14:28:45,367] A new study created in memory with name: credit_default_lgbm_v1
======================================================================
BitBullet Train - credit_default_lgbm_v1
======================================================================
Training samples: 24000
Features: 26
Class distribution: {0: 18691, 1: 5309}
Starting hyperparameter optimization (optuna)...
Trials: 10, CV Folds: 3
[I 2026-05-06 14:28:49,397] Trial 4 finished with value: 0.782487674922033 and parameters: {'num_leaves': 159, 'max_depth': 3, 'min_child_samples': 10, 'lambda_l1': 1.0469669874479804e-05, 'lambda_l2': 9.142555405712056e-06, 'min_gain_to_split': 0.34314851045805117, 'feature_fraction': 0.5878176146192433, 'bagging_fraction': 0.975979532434263, 'bagging_freq': 6, 'learning_rate': 0.0658812570668321, 'max_bin': 255}. Best is trial 4 with value: 0.782487674922033.
[I 2026-05-06 14:28:52,481] Trial 7 finished with value: 0.7837924046279956 and parameters: {'num_leaves': 109, 'max_depth': 12, 'min_child_samples': 96, 'lambda_l1': 3.5285162658627813, 'lambda_l2': 0.0019118667106049142, 'min_gain_to_split': 0.32810293195939566, 'feature_fraction': 0.6071973854234938, 'bagging_fraction': 0.6829336275747605, 'bagging_freq': 1, 'learning_rate': 0.22938974425733039, 'max_bin': 159}. Best is trial 7 with value: 0.7837924046279956.
[I 2026-05-06 14:28:53,259] Trial 0 finished with value: 0.782487220249224 and parameters: {'num_leaves': 167, 'max_depth': 5, 'min_child_samples': 52, 'lambda_l1': 0.012244544992846699, 'lambda_l2': 3.807906909338432e-06, 'min_gain_to_split': 0.12483790206820222, 'feature_fraction': 0.9836098221122052, 'bagging_fraction': 0.5360854270191966, 'bagging_freq': 5, 'learning_rate': 0.037154923360773084, 'max_bin': 63}. Best is trial 7 with value: 0.7837924046279956.
[I 2026-05-06 14:28:53,918] Trial 3 finished with value: 0.780966030345489 and parameters: {'num_leaves': 110, 'max_depth': 7, 'min_child_samples': 7, 'lambda_l1': 0.8973210675155582, 'lambda_l2': 0.0682781033300667, 'min_gain_to_split': 0.18895290807712162, 'feature_fraction': 0.6036009337046335, 'bagging_fraction': 0.9853566704657207, 'bagging_freq': 6, 'learning_rate': 0.16851528691592105, 'max_bin': 191}. Best is trial 7 with value: 0.7837924046279956.
[I 2026-05-06 14:28:57,320] Trial 8 finished with value: 0.7800509573551696 and parameters: {'num_leaves': 95, 'max_depth': 11, 'min_child_samples': 6, 'lambda_l1': 9.826480059552244e-07, 'lambda_l2': 3.5891969341014676, 'min_gain_to_split': 0.019956836311096127, 'feature_fraction': 0.5668757933312176, 'bagging_fraction': 0.6721811735214078, 'bagging_freq': 7, 'learning_rate': 0.1344044915382327, 'max_bin': 95}. Best is trial 7 with value: 0.7837924046279956.
[I 2026-05-06 14:28:59,336] Trial 9 finished with value: 0.7804075427177111 and parameters: {'num_leaves': 144, 'max_depth': 9, 'min_child_samples': 23, 'lambda_l1': 4.032669945433137e-05, 'lambda_l2': 0.00017840705367659724, 'min_gain_to_split': 0.23520980412204784, 'feature_fraction': 0.9323885741820045, 'bagging_fraction': 0.8762152516768509, 'bagging_freq': 4, 'learning_rate': 0.054954409425528324, 'max_bin': 127}. Best is trial 7 with value: 0.7837924046279956.
[I 2026-05-06 14:29:00,084] Trial 6 finished with value: 0.7844659064489745 and parameters: {'num_leaves': 190, 'max_depth': 5, 'min_child_samples': 8, 'lambda_l1': 4.854780435874212e-08, 'lambda_l2': 0.0003385218306665211, 'min_gain_to_split': 0.38641319697198695, 'feature_fraction': 0.8697100967282957, 'bagging_fraction': 0.773404252227638, 'bagging_freq': 4, 'learning_rate': 0.01686610263980842, 'max_bin': 127}. Best is trial 6 with value: 0.7844659064489745.
[I 2026-05-06 14:29:00,382] Trial 2 finished with value: 0.7840050071347134 and parameters: {'num_leaves': 248, 'max_depth': 3, 'min_child_samples': 5, 'lambda_l1': 1.0197907205515458e-08, 'lambda_l2': 0.0006846440275151191, 'min_gain_to_split': 0.7171143191373233, 'feature_fraction': 0.7409280316047595, 'bagging_fraction': 0.5122644701055994, 'bagging_freq': 2, 'learning_rate': 0.009678552464226966, 'max_bin': 63}. Best is trial 6 with value: 0.7844659064489745.
[I 2026-05-06 14:29:03,905] Trial 5 finished with value: 0.7853619152285397 and parameters: {'num_leaves': 289, 'max_depth': 5, 'min_child_samples': 13, 'lambda_l1': 4.791304580577613e-05, 'lambda_l2': 0.026538461705427185, 'min_gain_to_split': 0.7288215433160152, 'feature_fraction': 0.6082480512336117, 'bagging_fraction': 0.6018969670529957, 'bagging_freq': 1, 'learning_rate': 0.010181453780052564, 'max_bin': 127}. Best is trial 5 with value: 0.7853619152285397.
[I 2026-05-06 14:29:08,684] Trial 1 finished with value: 0.7836721253189375 and parameters: {'num_leaves': 244, 'max_depth': 12, 'min_child_samples': 53, 'lambda_l1': 1.8175764466996019e-06, 'lambda_l2': 2.1529752202441412e-07, 'min_gain_to_split': 0.13464133779050647, 'feature_fraction': 0.7641059146439713, 'bagging_fraction': 0.5748102098291443, 'bagging_freq': 3, 'learning_rate': 0.018348297643719154, 'max_bin': 223}. Best is trial 5 with value: 0.7853619152285397.
====================================================================== Optuna Optimization Summary ====================================================================== Number of finished trials: 10 Best trial: 5 Best value: 0.785362 Best hyperparameters: num_leaves: 289 max_depth: 5 min_child_samples: 13 lambda_l1: 0.000048 lambda_l2: 0.026538 min_gain_to_split: 0.728822 feature_fraction: 0.608248 bagging_fraction: 0.601897 bagging_freq: 1 learning_rate: 0.010181 max_bin: 127 Additional attributes: cv_scores: [0.7865111521841548, 0.7797120729838307, 0.789862520517634] cv_std: 0.004222831235812959 mean_training_loss: 0.38680237181227256 mean_validation_loss: 0.42566816947721814 best_iteration: 536 ====================================================================== Optimization complete! Best score: 0.7854 Time: 23.4s Training final model with best parameters... [LightGBM] [Warning] feature_fraction is set=0.6082480512336117, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.6082480512336117 [LightGBM] [Warning] lambda_l2 is set=0.026538461705427185, reg_lambda=0.0 will be ignored. Current value: lambda_l2=0.026538461705427185 [LightGBM] [Warning] min_gain_to_split is set=0.7288215433160152, min_split_gain=0.0 will be ignored. Current value: min_gain_to_split=0.7288215433160152 [LightGBM] [Warning] lambda_l1 is set=4.791304580577613e-05, reg_alpha=0.0 will be ignored. Current value: lambda_l1=4.791304580577613e-05 [LightGBM] [Warning] bagging_fraction is set=0.6018969670529957, subsample=1.0 will be ignored. Current value: bagging_fraction=0.6018969670529957 [LightGBM] [Warning] bagging_freq is set=1, subsample_freq=0 will be ignored. Current value: bagging_freq=1 [LightGBM] [Warning] feature_fraction is set=0.6082480512336117, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.6082480512336117 [LightGBM] [Warning] lambda_l2 is set=0.026538461705427185, reg_lambda=0.0 will be ignored. Current value: lambda_l2=0.026538461705427185 [LightGBM] [Warning] min_gain_to_split is set=0.7288215433160152, min_split_gain=0.0 will be ignored. Current value: min_gain_to_split=0.7288215433160152 [LightGBM] [Warning] lambda_l1 is set=4.791304580577613e-05, reg_alpha=0.0 will be ignored. Current value: lambda_l1=4.791304580577613e-05 [LightGBM] [Warning] bagging_fraction is set=0.6018969670529957, subsample=1.0 will be ignored. Current value: bagging_fraction=0.6018969670529957 [LightGBM] [Warning] bagging_freq is set=1, subsample_freq=0 will be ignored. Current value: bagging_freq=1 [LightGBM] [Info] Number of positive: 5309, number of negative: 18691 [LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.003791 seconds. You can set `force_col_wise=true` to remove the overhead. [LightGBM] [Info] Total Bins 1727 [LightGBM] [Info] Number of data points in the train set: 24000, number of used features: 26 [LightGBM] [Info] [binary:BoostFromScore]: pavg=0.221208 -> initscore=-1.258639 [LightGBM] [Info] Start training from score -1.258639 [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf Optimizing classification threshold... [LightGBM] [Warning] feature_fraction is set=0.6082480512336117, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.6082480512336117 [LightGBM] [Warning] lambda_l2 is set=0.026538461705427185, reg_lambda=0.0 will be ignored. Current value: lambda_l2=0.026538461705427185 [LightGBM] [Warning] min_gain_to_split is set=0.7288215433160152, min_split_gain=0.0 will be ignored. Current value: min_gain_to_split=0.7288215433160152 [LightGBM] [Warning] lambda_l1 is set=4.791304580577613e-05, reg_alpha=0.0 will be ignored. Current value: lambda_l1=4.791304580577613e-05 [LightGBM] [Warning] bagging_fraction is set=0.6018969670529957, subsample=1.0 will be ignored. Current value: bagging_fraction=0.6018969670529957 [LightGBM] [Warning] bagging_freq is set=1, subsample_freq=0 will be ignored. Current value: bagging_freq=1 Optimal threshold: 0.216 ====================================================================== Training Complete! ====================================================================== === Training Summary === Model: LGBMClassifier Best Score: 0.7854 CV Score: 0.0000 ± 0.0000 Features: 26/26 Optimal Threshold: 0.216 Training Time: 27.7s Optimization Time: 23.4s Best Hyperparameters: num_leaves: 289 max_depth: 5 min_child_samples: 13 lambda_l1: 0.0000 lambda_l2: 0.0265 min_gain_to_split: 0.7288 feature_fraction: 0.6082 bagging_fraction: 0.6019 bagging_freq: 1 learning_rate: 0.0102 max_bin: 127 n_estimators: 536 ====================================================================== Training complete in 27.7s === Training Summary === Model: LGBMClassifier Best Score: 0.7854 CV Score: 0.0000 ± 0.0000 Features: 26/26 Optimal Threshold: 0.216 Training Time: 27.7s Optimization Time: 23.4s Best Hyperparameters: num_leaves: 289 max_depth: 5 min_child_samples: 13 lambda_l1: 0.0000 lambda_l2: 0.0265 min_gain_to_split: 0.7288 feature_fraction: 0.6082 bagging_fraction: 0.6019 bagging_freq: 1 learning_rate: 0.0102 max_bin: 127 n_estimators: 536
In [9]:
Copied!
y_proba_v1 = state_v1.model.predict_proba(X_test_t)[:, 1]
threshold_v1 = state_v1.optimal_threshold or 0.5
y_pred_v1 = (y_proba_v1 >= threshold_v1).astype(int)
test_report_v1 = evaluate_classification(
y_true=y_test,
y_pred_proba=y_proba_v1,
threshold=threshold_v1,
labels=[0, 1],
)
test_roc_auc_v1 = test_report_v1.metrics["roc_auc"]
test_f1_v1 = test_report_v1.metrics["f1_score"]
test_accuracy_v1 = test_report_v1.metrics["accuracy"]
print(f"V1 Test Metrics:")
print(f" ROC AUC : {test_roc_auc_v1:.4f}")
print(f" F1 Score : {test_f1_v1:.4f}")
print(f" Accuracy : {test_accuracy_v1:.4f}")
print(f" Threshold: {threshold_v1:.4f}")
y_proba_v1 = state_v1.model.predict_proba(X_test_t)[:, 1]
threshold_v1 = state_v1.optimal_threshold or 0.5
y_pred_v1 = (y_proba_v1 >= threshold_v1).astype(int)
test_report_v1 = evaluate_classification(
y_true=y_test,
y_pred_proba=y_proba_v1,
threshold=threshold_v1,
labels=[0, 1],
)
test_roc_auc_v1 = test_report_v1.metrics["roc_auc"]
test_f1_v1 = test_report_v1.metrics["f1_score"]
test_accuracy_v1 = test_report_v1.metrics["accuracy"]
print(f"V1 Test Metrics:")
print(f" ROC AUC : {test_roc_auc_v1:.4f}")
print(f" F1 Score : {test_f1_v1:.4f}")
print(f" Accuracy : {test_accuracy_v1:.4f}")
print(f" Threshold: {threshold_v1:.4f}")
[LightGBM] [Warning] feature_fraction is set=0.6082480512336117, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.6082480512336117 [LightGBM] [Warning] lambda_l2 is set=0.026538461705427185, reg_lambda=0.0 will be ignored. Current value: lambda_l2=0.026538461705427185 [LightGBM] [Warning] min_gain_to_split is set=0.7288215433160152, min_split_gain=0.0 will be ignored. Current value: min_gain_to_split=0.7288215433160152 [LightGBM] [Warning] lambda_l1 is set=4.791304580577613e-05, reg_alpha=0.0 will be ignored. Current value: lambda_l1=4.791304580577613e-05 [LightGBM] [Warning] bagging_fraction is set=0.6018969670529957, subsample=1.0 will be ignored. Current value: bagging_fraction=0.6018969670529957 [LightGBM] [Warning] bagging_freq is set=1, subsample_freq=0 will be ignored. Current value: bagging_freq=1 V1 Test Metrics: ROC AUC : 0.7801 F1 Score : 0.5350 Accuracy : 0.7552 Threshold: 0.2164
3. ModelMetadata: Your Model's Identity Card¶
ModelMetadata is a structured dataclass that captures everything needed for reproducibility, auditing, lineage tracking, and exact inference replay in data-science environments.
| Field | Purpose |
|---|---|
name + version |
Unique identity — use semantic versioning |
model_type + framework |
What architecture, what library |
task |
binary_classification, multiclass_classification |
feature_names + n_features |
Exact contract with the inference pipeline |
hyperparameters |
Full reproducibility — never lose these |
metrics |
Test-set performance, locked in before shipping |
optimal_threshold |
The decision boundary found during training |
notes |
Free-form context for the human reading this six months later |
feature_schema |
Exact feature names, dtypes, and feature count |
preprocessing |
Pointer to fitted transform pipeline and fit scope |
search_metadata |
Optimizer, sampler, trial budget, CV setup, and best params |
inference_contract |
Minimal replay instructions: feature order, threshold, pipeline dependency |
artifact_metadata |
Structured reports such as evaluate_classification(...).to_dict() |
In [10]:
Copied!
metadata_v1 = ModelMetadata(
name="credit_default_predictor",
version="1.0.0",
model_type=state_v1.model.model_type,
framework=state_v1.model.framework,
task="binary_classification",
trained_at=datetime.now(),
feature_names=list(X_train_t.columns),
n_features=X_train_t.shape[1],
hyperparameters=state_v1.best_params,
classes=[0, 1],
optimal_threshold=threshold_v1,
training_time_seconds=state_v1.training_time_seconds,
tags=["binary_classification", "credit_default", "lightgbm", "uci"],
notes=(
"Credit card default predictor v1.0. "
"Trained on 30k UCI records. "
"Youden-J threshold optimised during training."
)
)
metadata_v1.add_feature_schema(X_train_t)
metadata_v1.preprocessing = {
"pipeline_name": pipeline_v1.name,
"pipeline_path": "pipelines/credit_default_pipeline_v1.pkl",
"fit_scope": "training_data_only",
}
metadata_v1.search_metadata = {
"optimizer": config_v1.optimizer,
"optuna_sampler": config_v1.optuna_sampler,
"n_trials": config_v1.n_trials,
"cv_folds": config_v1.cv_folds,
"optimization_metric": config_v1.optimization_metric,
"best_params": state_v1.best_params,
}
metadata_v1.target_mapping = {"negative_class": 0, "positive_class": 1}
metadata_v1.inference_contract = {
"required_pipeline_artifact": "pipelines/credit_default_pipeline_v1.pkl",
"required_feature_order": list(X_train_t.columns),
"probability_column": 1,
"decision_threshold": threshold_v1,
}
metadata_v1.add_artifact_metadata(classification_metrics=test_report_v1.to_dict())
print("ModelMetadata created with schema, preprocessing, search, and inference metadata.")
print(f" Name : {metadata_v1.name} v{metadata_v1.version}")
print(f" Framework : {metadata_v1.framework} / {metadata_v1.model_type}")
print(f" Features : {metadata_v1.n_features}")
metadata_v1 = ModelMetadata(
name="credit_default_predictor",
version="1.0.0",
model_type=state_v1.model.model_type,
framework=state_v1.model.framework,
task="binary_classification",
trained_at=datetime.now(),
feature_names=list(X_train_t.columns),
n_features=X_train_t.shape[1],
hyperparameters=state_v1.best_params,
classes=[0, 1],
optimal_threshold=threshold_v1,
training_time_seconds=state_v1.training_time_seconds,
tags=["binary_classification", "credit_default", "lightgbm", "uci"],
notes=(
"Credit card default predictor v1.0. "
"Trained on 30k UCI records. "
"Youden-J threshold optimised during training."
)
)
metadata_v1.add_feature_schema(X_train_t)
metadata_v1.preprocessing = {
"pipeline_name": pipeline_v1.name,
"pipeline_path": "pipelines/credit_default_pipeline_v1.pkl",
"fit_scope": "training_data_only",
}
metadata_v1.search_metadata = {
"optimizer": config_v1.optimizer,
"optuna_sampler": config_v1.optuna_sampler,
"n_trials": config_v1.n_trials,
"cv_folds": config_v1.cv_folds,
"optimization_metric": config_v1.optimization_metric,
"best_params": state_v1.best_params,
}
metadata_v1.target_mapping = {"negative_class": 0, "positive_class": 1}
metadata_v1.inference_contract = {
"required_pipeline_artifact": "pipelines/credit_default_pipeline_v1.pkl",
"required_feature_order": list(X_train_t.columns),
"probability_column": 1,
"decision_threshold": threshold_v1,
}
metadata_v1.add_artifact_metadata(classification_metrics=test_report_v1.to_dict())
print("ModelMetadata created with schema, preprocessing, search, and inference metadata.")
print(f" Name : {metadata_v1.name} v{metadata_v1.version}")
print(f" Framework : {metadata_v1.framework} / {metadata_v1.model_type}")
print(f" Features : {metadata_v1.n_features}")
ModelMetadata created with schema, preprocessing, search, and inference metadata. Name : credit_default_predictor v1.0.0 Framework : lightgbm / LGBMClassifier Features : 26
4. add_metric() and add_cv_scores()¶
In [11]:
Copied!
metadata_v1.add_metric("test_roc_auc", test_roc_auc_v1)
metadata_v1.add_metric("test_f1", test_f1_v1)
metadata_v1.add_metric("test_accuracy", test_accuracy_v1)
if state_v1.study:
fold_scores = state_v1.study.best_trial.user_attrs.get('cv_scores', [state_v1.best_score])
else:
fold_scores = state_v1.cv_scores if state_v1.cv_scores else [state_v1.best_score]
metadata_v1.add_cv_scores("roc_auc", fold_scores)
print("Metrics recorded:")
for name, val in metadata_v1.metrics.items():
print(f" {name:<20s}: {val:.4f}")
print(f"\nCV scores ({metadata_v1.cv_folds} folds):")
for metric, scores in metadata_v1.cv_scores.items():
print(f" {metric}: {[round(s, 4) for s in scores]}")
print(f" mean={np.mean(scores):.4f} std={np.std(scores):.4f}")
metadata_v1.add_metric("test_roc_auc", test_roc_auc_v1)
metadata_v1.add_metric("test_f1", test_f1_v1)
metadata_v1.add_metric("test_accuracy", test_accuracy_v1)
if state_v1.study:
fold_scores = state_v1.study.best_trial.user_attrs.get('cv_scores', [state_v1.best_score])
else:
fold_scores = state_v1.cv_scores if state_v1.cv_scores else [state_v1.best_score]
metadata_v1.add_cv_scores("roc_auc", fold_scores)
print("Metrics recorded:")
for name, val in metadata_v1.metrics.items():
print(f" {name:<20s}: {val:.4f}")
print(f"\nCV scores ({metadata_v1.cv_folds} folds):")
for metric, scores in metadata_v1.cv_scores.items():
print(f" {metric}: {[round(s, 4) for s in scores]}")
print(f" mean={np.mean(scores):.4f} std={np.std(scores):.4f}")
Metrics recorded:
test_roc_auc : 0.7801
test_f1 : 0.5350
test_accuracy : 0.7552
CV scores (3 folds):
roc_auc: [0.7865, 0.7797, 0.7899]
mean=0.7854 std=0.0042
5. DatasetMetadata: Knowing Exactly What You Trained On¶
In [12]:
Copied!
train_meta = DatasetMetadata.from_dataframe(
X_train_t, y_train,
name="credit_default_train_v1",
compute_hash=True
)
test_meta = DatasetMetadata.from_dataframe(
X_test_t, y_test,
name="credit_default_test_v1",
compute_hash=True
)
print("=" * 55)
print(f" TRAIN DATASET: {train_meta.name}")
print("=" * 55)
print(f" Samples : {train_meta.n_samples:,}")
print(f" Features : {train_meta.n_features}")
print(f" Default rate : {train_meta.class_distribution}")
print(f" Data hash : {train_meta.data_hash}")
missing = {k: v for k, v in train_meta.statistics['missing_values'].items() if v > 0}
print(f" Missing values: {'none' if not missing else missing}")
metadata_v1.add_dataset(X_train_t, y_train, "train")
metadata_v1.add_dataset(X_test_t, y_test, "test")
print("\nDataset provenance attached to model metadata.")
train_meta = DatasetMetadata.from_dataframe(
X_train_t, y_train,
name="credit_default_train_v1",
compute_hash=True
)
test_meta = DatasetMetadata.from_dataframe(
X_test_t, y_test,
name="credit_default_test_v1",
compute_hash=True
)
print("=" * 55)
print(f" TRAIN DATASET: {train_meta.name}")
print("=" * 55)
print(f" Samples : {train_meta.n_samples:,}")
print(f" Features : {train_meta.n_features}")
print(f" Default rate : {train_meta.class_distribution}")
print(f" Data hash : {train_meta.data_hash}")
missing = {k: v for k, v in train_meta.statistics['missing_values'].items() if v > 0}
print(f" Missing values: {'none' if not missing else missing}")
metadata_v1.add_dataset(X_train_t, y_train, "train")
metadata_v1.add_dataset(X_test_t, y_test, "test")
print("\nDataset provenance attached to model metadata.")
=======================================================
TRAIN DATASET: credit_default_train_v1
=======================================================
Samples : 24,000
Features : 26
Default rate : {0: 18691, 1: 5309}
Data hash : 5d3b35fff8b0da10a6ed3a35b064f769
Missing values: none
Dataset provenance attached to model metadata.
6. metadata.summary() — Human-Readable Model Cards¶
In [13]:
Copied!
print(metadata_v1.summary())
print(metadata_v1.summary())
Model: credit_default_predictor (v1.0.0) Type: LGBMClassifier (lightgbm) Task: binary_classification Features: 26 Trained: 2026-05-06 14:29 Performance Metrics: test_roc_auc: 0.7801 test_f1: 0.5350 test_accuracy: 0.7552 Cross-Validation: roc_auc: 0.7854 ± 0.0042 Training Data: 24000 samples Test Data: 6000 samples
7. metadata.to_dict() — JSON-Serialisable Output¶
In [14]:
Copied!
meta_dict = metadata_v1.to_dict()
print("Top-level keys:")
for k, v in meta_dict.items():
if isinstance(v, dict):
print(f" {k:<28s}: {{...}} ({len(v)} keys)")
elif isinstance(v, list) and len(v) > 3:
print(f" {k:<28s}: [...] ({len(v)} items)")
else:
print(f" {k:<28s}: {v}")
try:
json_str = json.dumps(meta_dict, default=str, indent=2)
print(f"\nJSON serialisation: OK ({len(json_str):,} characters)")
except (TypeError, ValueError) as e:
print(f"Serialisation failed: {e}")
meta_dict = metadata_v1.to_dict()
print("Top-level keys:")
for k, v in meta_dict.items():
if isinstance(v, dict):
print(f" {k:<28s}: {{...}} ({len(v)} keys)")
elif isinstance(v, list) and len(v) > 3:
print(f" {k:<28s}: [...] ({len(v)} items)")
else:
print(f" {k:<28s}: {v}")
try:
json_str = json.dumps(meta_dict, default=str, indent=2)
print(f"\nJSON serialisation: OK ({len(json_str):,} characters)")
except (TypeError, ValueError) as e:
print(f"Serialisation failed: {e}")
Top-level keys:
name : credit_default_predictor
version : 1.0.0
model_type : LGBMClassifier
framework : lightgbm
task : binary_classification
created_at : 2026-05-06T14:29:13.165864
trained_at : 2026-05-06T14:29:13.165791
hyperparameters : {...} (12 keys)
feature_names : [...] (26 items)
n_features : 26
classes : [0, 1]
metrics : {...} (3 keys)
training_time_seconds : 27.72658348083496
cv_folds : 3
cv_scores : {...} (1 keys)
optimal_threshold : 0.21636717755711785
training_dataset : {...} (12 keys)
validation_dataset : None
test_dataset : {...} (12 keys)
environment : {...} (0 keys)
tags : [...] (4 items)
notes : Credit card default predictor v1.0. Trained on 30k UCI records. Youden-J threshold optimised during training.
JSON serialisation: OK (7,742 characters)
8. ModelSerializer: Save Once, Reproduce Anywhere¶
In [15]:
Copied!
models_dir = Path("models")
models_dir.mkdir(exist_ok=True)
v1_dir = models_dir / "credit_default_predictor" / "1.0.0"
v1_dir.mkdir(parents=True, exist_ok=True)
v1_path = str(v1_dir / "model.pkl")
ModelSerializer.save(
model=state_v1.model,
path=v1_path,
metadata=metadata_v1,
train_data=(X_train_t, y_train),
test_data=(X_test_t, y_test),
include_datasets=True
)
print("Files written:")
for f in sorted(v1_dir.glob("*")):
size_kb = f.stat().st_size / 1024
print(f" {f.name:<35s} {size_kb:>8.1f} KB")
models_dir = Path("models")
models_dir.mkdir(exist_ok=True)
v1_dir = models_dir / "credit_default_predictor" / "1.0.0"
v1_dir.mkdir(parents=True, exist_ok=True)
v1_path = str(v1_dir / "model.pkl")
ModelSerializer.save(
model=state_v1.model,
path=v1_path,
metadata=metadata_v1,
train_data=(X_train_t, y_train),
test_data=(X_test_t, y_test),
include_datasets=True
)
print("Files written:")
for f in sorted(v1_dir.glob("*")):
size_kb = f.stat().st_size / 1024
print(f" {f.name:<35s} {size_kb:>8.1f} KB")
Files written: model.pkl 8583.9 KB model_metadata.json 7.6 KB
9. load_metadata() — Fast Inspection Without Loading the Model¶
In [16]:
Copied!
fast_meta = ModelSerializer.load_metadata(v1_path)
print("Metadata loaded (JSON sidecar only — no model deserialisation):")
print(f" Model name : {fast_meta.get('name')} v{fast_meta.get('version')}")
print(f" Task : {fast_meta.get('task')}")
print(f" Framework : {fast_meta.get('framework')}")
print(f" Features : {fast_meta.get('n_features')}")
print(f" Trained at : {fast_meta.get('trained_at')}")
print()
print(" Recorded metrics:")
for metric, val in (fast_meta.get('metrics') or {}).items():
print(f" {metric:<22s}: {val:.4f}")
fast_meta = ModelSerializer.load_metadata(v1_path)
print("Metadata loaded (JSON sidecar only — no model deserialisation):")
print(f" Model name : {fast_meta.get('name')} v{fast_meta.get('version')}")
print(f" Task : {fast_meta.get('task')}")
print(f" Framework : {fast_meta.get('framework')}")
print(f" Features : {fast_meta.get('n_features')}")
print(f" Trained at : {fast_meta.get('trained_at')}")
print()
print(" Recorded metrics:")
for metric, val in (fast_meta.get('metrics') or {}).items():
print(f" {metric:<22s}: {val:.4f}")
Metadata loaded (JSON sidecar only — no model deserialisation):
Model name : credit_default_predictor v1.0.0
Task : binary_classification
Framework : lightgbm
Features : 26
Trained at : 2026-05-06T14:29:13.165791
Recorded metrics:
test_roc_auc : 0.7801
test_f1 : 0.5350
test_accuracy : 0.7552
10. ModelSerializer.load() — Full Package Reconstruction¶
In [17]:
Copied!
package_v1 = ModelSerializer.load(v1_path)
print(package_v1.summary())
print("\nPackage components:")
print(f" package.model : {package_v1.model}")
print(f" package.metadata : ModelMetadata(name={package_v1.metadata.name!r})")
print(f" package.train_data : X={package_v1.train_data['X'].shape} y={package_v1.train_data['y'].shape}")
print(f" package.test_data : X={package_v1.test_data['X'].shape} y={package_v1.test_data['y'].shape}")
package_v1 = ModelSerializer.load(v1_path)
print(package_v1.summary())
print("\nPackage components:")
print(f" package.model : {package_v1.model}")
print(f" package.metadata : ModelMetadata(name={package_v1.metadata.name!r})")
print(f" package.train_data : X={package_v1.train_data['X'].shape} y={package_v1.train_data['y'].shape}")
print(f" package.test_data : X={package_v1.test_data['X'].shape} y={package_v1.test_data['y'].shape}")
Model: credit_default_predictor (v1.0.0) Type: LGBMClassifier (lightgbm) Task: binary_classification Features: 26 Trained: 2026-05-06 14:29 Performance Metrics: test_roc_auc: 0.7801 test_f1: 0.5350 test_accuracy: 0.7552 Cross-Validation: roc_auc: 0.7854 ± 0.0042 Training Data: 24000 samples Test Data: 6000 samples Training data included: 24000 samples Test data included: 6000 samples Package components: package.model : LGBMClassifierWrapper(fitted, n_features=26) package.metadata : ModelMetadata(name='credit_default_predictor') package.train_data : X=(24000, 26) y=(24000,) package.test_data : X=(6000, 26) y=(6000,)
In [18]:
Copied!
proba_original = state_v1.model.predict_proba(X_test_t)[:, 1]
proba_from_disk = package_v1.model.predict_proba(package_v1.test_data['X'])[:, 1]
max_delta = np.abs(proba_original - proba_from_disk).max()
print(f"Round-trip prediction delta (max): {max_delta:.2e}")
print(f"Prediction integrity: {'PASS' if max_delta < 1e-10 else 'FAIL'}")
proba_original = state_v1.model.predict_proba(X_test_t)[:, 1]
proba_from_disk = package_v1.model.predict_proba(package_v1.test_data['X'])[:, 1]
max_delta = np.abs(proba_original - proba_from_disk).max()
print(f"Round-trip prediction delta (max): {max_delta:.2e}")
print(f"Prediction integrity: {'PASS' if max_delta < 1e-10 else 'FAIL'}")
[LightGBM] [Warning] feature_fraction is set=0.6082480512336117, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.6082480512336117 [LightGBM] [Warning] lambda_l2 is set=0.026538461705427185, reg_lambda=0.0 will be ignored. Current value: lambda_l2=0.026538461705427185 [LightGBM] [Warning] min_gain_to_split is set=0.7288215433160152, min_split_gain=0.0 will be ignored. Current value: min_gain_to_split=0.7288215433160152 [LightGBM] [Warning] lambda_l1 is set=4.791304580577613e-05, reg_alpha=0.0 will be ignored. Current value: lambda_l1=4.791304580577613e-05 [LightGBM] [Warning] bagging_fraction is set=0.6018969670529957, subsample=1.0 will be ignored. Current value: bagging_fraction=0.6018969670529957 [LightGBM] [Warning] bagging_freq is set=1, subsample_freq=0 will be ignored. Current value: bagging_freq=1 [LightGBM] [Warning] feature_fraction is set=0.6082480512336117, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.6082480512336117 [LightGBM] [Warning] lambda_l2 is set=0.026538461705427185, reg_lambda=0.0 will be ignored. Current value: lambda_l2=0.026538461705427185 [LightGBM] [Warning] min_gain_to_split is set=0.7288215433160152, min_split_gain=0.0 will be ignored. Current value: min_gain_to_split=0.7288215433160152 [LightGBM] [Warning] lambda_l1 is set=4.791304580577613e-05, reg_alpha=0.0 will be ignored. Current value: lambda_l1=4.791304580577613e-05 [LightGBM] [Warning] bagging_fraction is set=0.6018969670529957, subsample=1.0 will be ignored. Current value: bagging_fraction=0.6018969670529957 [LightGBM] [Warning] bagging_freq is set=1, subsample_freq=0 will be ignored. Current value: bagging_freq=1 Round-trip prediction delta (max): 0.00e+00 Prediction integrity: PASS
11. save_model_only() + load_model_only() — Lean Model-Only Artefacts¶
In [19]:
Copied!
deploy_path = str(v1_dir / "model_deploy.pkl")
ModelSerializer.save_model_only(model=state_v1.model, path=deploy_path)
full_size = Path(v1_path).stat().st_size / 1024
deploy_size = Path(deploy_path).stat().st_size / 1024
print("File size comparison:")
print(f" Full package (model + data) : {full_size:>8.1f} KB")
print(f" Deploy artefact (model only): {deploy_size:>8.1f} KB")
print(f" Size reduction : {(1 - deploy_size/full_size)*100:.1f}%")
inference_model = ModelSerializer.load_model_only(deploy_path)
sample_scores = inference_model.predict_proba(X_test_t.iloc[:5])[:, 1]
print(f"\nSample default probabilities: {sample_scores.round(4)}")
deploy_path = str(v1_dir / "model_deploy.pkl")
ModelSerializer.save_model_only(model=state_v1.model, path=deploy_path)
full_size = Path(v1_path).stat().st_size / 1024
deploy_size = Path(deploy_path).stat().st_size / 1024
print("File size comparison:")
print(f" Full package (model + data) : {full_size:>8.1f} KB")
print(f" Deploy artefact (model only): {deploy_size:>8.1f} KB")
print(f" Size reduction : {(1 - deploy_size/full_size)*100:.1f}%")
inference_model = ModelSerializer.load_model_only(deploy_path)
sample_scores = inference_model.predict_proba(X_test_t.iloc[:5])[:, 1]
print(f"\nSample default probabilities: {sample_scores.round(4)}")
File size comparison: Full package (model + data) : 8583.9 KB Deploy artefact (model only): 1547.9 KB Size reduction : 82.0% [LightGBM] [Warning] feature_fraction is set=0.6082480512336117, colsample_bytree=1.0 will be ignored. Current value: feature_fraction=0.6082480512336117 [LightGBM] [Warning] lambda_l2 is set=0.026538461705427185, reg_lambda=0.0 will be ignored. Current value: lambda_l2=0.026538461705427185 [LightGBM] [Warning] min_gain_to_split is set=0.7288215433160152, min_split_gain=0.0 will be ignored. Current value: min_gain_to_split=0.7288215433160152 [LightGBM] [Warning] lambda_l1 is set=4.791304580577613e-05, reg_alpha=0.0 will be ignored. Current value: lambda_l1=4.791304580577613e-05 [LightGBM] [Warning] bagging_fraction is set=0.6018969670529957, subsample=1.0 will be ignored. Current value: bagging_fraction=0.6018969670529957 [LightGBM] [Warning] bagging_freq is set=1, subsample_freq=0 will be ignored. Current value: bagging_freq=1 Sample default probabilities: [0.1483 0.1437 0.1475 0.1676 0.0394]
12. ModelRegistry: A Catalogue of Your ML Arsenal¶
In [20]:
Copied!
from bitbullet.model.wrappers import lgbm_wrapper, xgb_wrapper
all_models = ModelRegistry.list_models()
print(f"Registered model wrappers ({len(all_models)} total):")
for name, cls in sorted(all_models.items()):
print(f" {name:<30s} -> {cls.__module__}.{cls.__name__}")
from bitbullet.model.wrappers import lgbm_wrapper, xgb_wrapper
all_models = ModelRegistry.list_models()
print(f"Registered model wrappers ({len(all_models)} total):")
for name, cls in sorted(all_models.items()):
print(f" {name:<30s} -> {cls.__module__}.{cls.__name__}")
Registered model wrappers (4 total): lgbm_classifier -> bitbullet.model.wrappers.lgbm_wrapper.LGBMClassifierWrapper lgbm_regressor -> bitbullet.model.wrappers.lgbm_wrapper.LGBMRegressorWrapper xgb_classifier -> bitbullet.model.wrappers.xgb_wrapper.XGBClassifierWrapper xgb_regressor -> bitbullet.model.wrappers.xgb_wrapper.XGBRegressorWrapper
In [21]:
Copied!
for name in ["lgbm_classifier", "lgbm_regressor", "xgb_classifier", "nonexistent_model"]:
status = "registered" if ModelRegistry.is_registered(name) else "not found"
print(f" {name:<30s}: {status}")
for name in ["lgbm_classifier", "lgbm_regressor", "xgb_classifier", "nonexistent_model"]:
status = "registered" if ModelRegistry.is_registered(name) else "not found"
print(f" {name:<30s}: {status}")
lgbm_classifier : registered lgbm_regressor : registered xgb_classifier : registered nonexistent_model : not found
In [22]:
Copied!
factory_model = ModelRegistry.create("lgbm_classifier", n_estimators=100, max_depth=5, learning_rate=0.05)
print(f"Created via factory:")
print(f" Type : {type(factory_model).__name__}")
print(f" Is fitted : {factory_model.is_fitted}")
factory_model = ModelRegistry.create("lgbm_classifier", n_estimators=100, max_depth=5, learning_rate=0.05)
print(f"Created via factory:")
print(f" Type : {type(factory_model).__name__}")
print(f" Is fitted : {factory_model.is_fitted}")
Created via factory: Type : LGBMClassifierWrapper Is fitted : False
13. @ModelRegistry.register — Custom Model Wrappers¶
In [23]:
Copied!
from sklearn.linear_model import LogisticRegression
from datetime import datetime as dt
@ModelRegistry.register("logistic_regression_custom")
class LogisticRegressionWrapper(BaseModel):
def __init__(self, **kwargs):
super().__init__(**kwargs)
self._state.model_type = "LogisticRegression"
self._state.framework = "sklearn"
self._lr_model = None
def fit(self, X, y, **kwargs):
self._state.feature_names = self._extract_feature_names(X)
self._state.n_features = len(self._state.feature_names)
self._lr_model = LogisticRegression(**self._hyperparameters)
self._lr_model.fit(X, y)
self._state.model = self._lr_model
self._state.classes = self._lr_model.classes_
self._state.is_fitted = True
self._state.fitted_at = dt.now()
return self
def predict(self, X):
return self._lr_model.predict(X)
def predict_proba(self, X):
return self._lr_model.predict_proba(X)
print(f"Registered: {ModelRegistry.is_registered('logistic_regression_custom')}")
print(f"Registry now contains {len(ModelRegistry.list_models())} model types")
lr_model = ModelRegistry.create("logistic_regression_custom", max_iter=1000, C=1.0)
lr_model.fit(X_train_t, y_train)
lr_auc = roc_auc_score(y_test, lr_model.predict_proba(X_test_t)[:, 1])
print(f"\nLogistic Regression Test AUC: {lr_auc:.4f}")
from sklearn.linear_model import LogisticRegression
from datetime import datetime as dt
@ModelRegistry.register("logistic_regression_custom")
class LogisticRegressionWrapper(BaseModel):
def __init__(self, **kwargs):
super().__init__(**kwargs)
self._state.model_type = "LogisticRegression"
self._state.framework = "sklearn"
self._lr_model = None
def fit(self, X, y, **kwargs):
self._state.feature_names = self._extract_feature_names(X)
self._state.n_features = len(self._state.feature_names)
self._lr_model = LogisticRegression(**self._hyperparameters)
self._lr_model.fit(X, y)
self._state.model = self._lr_model
self._state.classes = self._lr_model.classes_
self._state.is_fitted = True
self._state.fitted_at = dt.now()
return self
def predict(self, X):
return self._lr_model.predict(X)
def predict_proba(self, X):
return self._lr_model.predict_proba(X)
print(f"Registered: {ModelRegistry.is_registered('logistic_regression_custom')}")
print(f"Registry now contains {len(ModelRegistry.list_models())} model types")
lr_model = ModelRegistry.create("logistic_regression_custom", max_iter=1000, C=1.0)
lr_model.fit(X_train_t, y_train)
lr_auc = roc_auc_score(y_test, lr_model.predict_proba(X_test_t)[:, 1])
print(f"\nLogistic Regression Test AUC: {lr_auc:.4f}")
Registered: True Registry now contains 5 model types Logistic Regression Test AUC: 0.7316
14. Version Management — Semantic Versioning Pattern¶
models/
credit_default_predictor/
1.0.0/
model.pkl
model_metadata.json
model_deploy.pkl
2.0.0/
model.pkl
...
Each version directory is immutable once promoted. Rolling back means pointing your serving layer at the previous directory.
15. Training V2 — Simulating a Model Improvement¶
In [ ]:
Copied!
config_v2 = TrainConfig(
name="credit_default_lgbm_v2",
model_type="lgbm",
task="binary_classification",
optimization_metric="roc_auc",
optuna_sampler="tpe",
n_trials=10,
cv_folds=3,
optimize_threshold=True,
threshold_optimization_method="youden",
save_feature_importance=True,
generate_shap=False,
verbose=True,
optuna_show_progress=False,
random_state=99 # Different seed — different Optuna search trajectory
)
t0 = time.time()
trainer_v2 = OptunaTrainer(config=config_v2)
trainer_v2.fit(X_train_t, y_train)
state_v2 = trainer_v2.state
print(f"V2 training complete in {time.time() - t0:.1f}s")
config_v2 = TrainConfig(
name="credit_default_lgbm_v2",
model_type="lgbm",
task="binary_classification",
optimization_metric="roc_auc",
optuna_sampler="tpe",
n_trials=10,
cv_folds=3,
optimize_threshold=True,
threshold_optimization_method="youden",
save_feature_importance=True,
generate_shap=False,
verbose=True,
optuna_show_progress=False,
random_state=99 # Different seed — different Optuna search trajectory
)
t0 = time.time()
trainer_v2 = OptunaTrainer(config=config_v2)
trainer_v2.fit(X_train_t, y_train)
state_v2 = trainer_v2.state
print(f"V2 training complete in {time.time() - t0:.1f}s")
[I 2026-05-06 14:29:35,083] A new study created in memory with name: credit_default_lgbm_v2
======================================================================
BitBullet Train - credit_default_lgbm_v2
======================================================================
Training samples: 24000
Features: 26
Class distribution: {0: 18691, 1: 5309}
Starting hyperparameter optimization (optuna)...
Trials: 10, CV Folds: 3
[I 2026-05-06 14:29:39,347] Trial 5 finished with value: 0.782912773978182 and parameters: {'num_leaves': 21, 'max_depth': 12, 'min_child_samples': 51, 'lambda_l1': 3.198129938918566e-05, 'lambda_l2': 1.2146761448930005e-05, 'min_gain_to_split': 0.8902704104749739, 'feature_fraction': 0.564218275579441, 'bagging_fraction': 0.8930131995427238, 'bagging_freq': 5, 'learning_rate': 0.1838055561830655, 'max_bin': 223}. Best is trial 5 with value: 0.782912773978182.
[I 2026-05-06 14:29:43,744] Trial 3 finished with value: 0.7840345942057775 and parameters: {'num_leaves': 35, 'max_depth': 7, 'min_child_samples': 5, 'lambda_l1': 1.2668443686277898e-05, 'lambda_l2': 6.667569170828315e-05, 'min_gain_to_split': 0.46512943642656523, 'feature_fraction': 0.6950636679489371, 'bagging_fraction': 0.6368559328840082, 'bagging_freq': 3, 'learning_rate': 0.05814625308392654, 'max_bin': 95}. Best is trial 3 with value: 0.7840345942057775.
[I 2026-05-06 14:29:45,129] Trial 6 finished with value: 0.7835014787792075 and parameters: {'num_leaves': 185, 'max_depth': 6, 'min_child_samples': 7, 'lambda_l1': 1.7317952674829006e-06, 'lambda_l2': 0.00015569105138337706, 'min_gain_to_split': 0.5991286983102695, 'feature_fraction': 0.6197110841995219, 'bagging_fraction': 0.9557092041573234, 'bagging_freq': 1, 'learning_rate': 0.05681276382860915, 'max_bin': 255}. Best is trial 3 with value: 0.7840345942057775.
[I 2026-05-06 14:29:45,368] Trial 7 finished with value: 0.7830796797480789 and parameters: {'num_leaves': 110, 'max_depth': 6, 'min_child_samples': 6, 'lambda_l1': 8.42874454972082e-05, 'lambda_l2': 0.6566089948571227, 'min_gain_to_split': 0.2818247902754588, 'feature_fraction': 0.6432105790897444, 'bagging_fraction': 0.898022314047384, 'bagging_freq': 3, 'learning_rate': 0.05522080967791649, 'max_bin': 127}. Best is trial 3 with value: 0.7840345942057775.
[I 2026-05-06 14:29:57,950] Trial 4 finished with value: 0.7846914017570068 and parameters: {'num_leaves': 236, 'max_depth': 11, 'min_child_samples': 34, 'lambda_l1': 1.668755606473228, 'lambda_l2': 3.931392110336462e-06, 'min_gain_to_split': 0.7051414683324333, 'feature_fraction': 0.5932723358063668, 'bagging_fraction': 0.5993729128275531, 'bagging_freq': 5, 'learning_rate': 0.030082493445507202, 'max_bin': 159}. Best is trial 4 with value: 0.7846914017570068.
[I 2026-05-06 14:30:06,118] Trial 9 finished with value: 0.7839114270594033 and parameters: {'num_leaves': 86, 'max_depth': 6, 'min_child_samples': 18, 'lambda_l1': 1.5940386030280055e-05, 'lambda_l2': 1.2289642680019808e-06, 'min_gain_to_split': 0.08463529119537327, 'feature_fraction': 0.9135673145468102, 'bagging_fraction': 0.5924639929700057, 'bagging_freq': 7, 'learning_rate': 0.007870767412293792, 'max_bin': 223}. Best is trial 4 with value: 0.7846914017570068.
[I 2026-05-06 14:30:07,638] Trial 0 finished with value: 0.7829746167981749 and parameters: {'num_leaves': 112, 'max_depth': 5, 'min_child_samples': 9, 'lambda_l1': 5.850246529282143e-08, 'lambda_l2': 1.3632684578775255e-05, 'min_gain_to_split': 0.1971884367643605, 'feature_fraction': 0.954505357601366, 'bagging_fraction': 0.9024055258424614, 'bagging_freq': 4, 'learning_rate': 0.005873253028943797, 'max_bin': 95}. Best is trial 4 with value: 0.7846914017570068.
[I 2026-05-06 14:30:09,042] Trial 8 finished with value: 0.7789455830066881 and parameters: {'num_leaves': 140, 'max_depth': 11, 'min_child_samples': 9, 'lambda_l1': 2.8084078692392386e-07, 'lambda_l2': 0.0019625746887455314, 'min_gain_to_split': 0.25085040380737944, 'feature_fraction': 0.9920936949061228, 'bagging_fraction': 0.8439101897417567, 'bagging_freq': 1, 'learning_rate': 0.022712821246251496, 'max_bin': 223}. Best is trial 4 with value: 0.7846914017570068.
[I 2026-05-06 14:30:10,284] Trial 2 finished with value: 0.7845402427830183 and parameters: {'num_leaves': 133, 'max_depth': 9, 'min_child_samples': 35, 'lambda_l1': 2.3501673550543083e-05, 'lambda_l2': 0.0012381674477924278, 'min_gain_to_split': 0.5803657748940718, 'feature_fraction': 0.5393470052772198, 'bagging_fraction': 0.7630633112773256, 'bagging_freq': 4, 'learning_rate': 0.014261890626007459, 'max_bin': 191}. Best is trial 4 with value: 0.7846914017570068.
In [ ]:
Copied!
y_proba_v2 = state_v2.model.predict_proba(X_test_t)[:, 1]
threshold_v2 = state_v2.optimal_threshold or 0.5
y_pred_v2 = (y_proba_v2 >= threshold_v2).astype(int)
test_roc_auc_v2 = roc_auc_score(y_test, y_proba_v2)
test_f1_v2 = f1_score(y_test, y_pred_v2)
test_accuracy_v2 = accuracy_score(y_test, y_pred_v2)
print(f"V2 Test Metrics:")
print(f" ROC AUC : {test_roc_auc_v2:.4f}")
print(f" F1 Score : {test_f1_v2:.4f}")
print(f" Accuracy : {test_accuracy_v2:.4f}")
y_proba_v2 = state_v2.model.predict_proba(X_test_t)[:, 1]
threshold_v2 = state_v2.optimal_threshold or 0.5
y_pred_v2 = (y_proba_v2 >= threshold_v2).astype(int)
test_roc_auc_v2 = roc_auc_score(y_test, y_proba_v2)
test_f1_v2 = f1_score(y_test, y_pred_v2)
test_accuracy_v2 = accuracy_score(y_test, y_pred_v2)
print(f"V2 Test Metrics:")
print(f" ROC AUC : {test_roc_auc_v2:.4f}")
print(f" F1 Score : {test_f1_v2:.4f}")
print(f" Accuracy : {test_accuracy_v2:.4f}")
In [ ]:
Copied!
metadata_v2 = ModelMetadata(
name="credit_default_predictor",
version="2.0.0",
model_type=state_v2.model.model_type,
framework=state_v2.model.framework,
task="binary_classification",
trained_at=datetime.now(),
feature_names=list(X_train_t.columns),
n_features=X_train_t.shape[1],
hyperparameters=state_v2.best_params,
classes=[0, 1],
optimal_threshold=threshold_v2,
training_time_seconds=state_v2.training_time_seconds,
tags=["binary_classification", "credit_default", "lightgbm", "uci"],
notes="Credit default predictor v2.0. Retrained with different random seed."
)
test_report_v2 = evaluate_classification(
y_true=y_test,
y_pred_proba=y_proba_v2,
threshold=threshold_v2,
labels=[0, 1],
)
metadata_v2.add_metric("test_roc_auc", test_report_v2.metrics["roc_auc"])
metadata_v2.add_metric("test_f1", test_report_v2.metrics["f1_score"])
metadata_v2.add_metric("test_accuracy", test_report_v2.metrics["accuracy"])
metadata_v2.add_feature_schema(X_train_t)
metadata_v2.preprocessing = metadata_v1.preprocessing.copy()
metadata_v2.search_metadata = {
"optimizer": config_v2.optimizer,
"optuna_sampler": config_v2.optuna_sampler,
"n_trials": config_v2.n_trials,
"cv_folds": config_v2.cv_folds,
"optimization_metric": config_v2.optimization_metric,
"best_params": state_v2.best_params,
}
metadata_v2.target_mapping = metadata_v1.target_mapping.copy()
metadata_v2.inference_contract = {
**metadata_v1.inference_contract,
"decision_threshold": threshold_v2,
}
metadata_v2.add_artifact_metadata(classification_metrics=test_report_v2.to_dict())
if state_v2.study:
fold_scores_v2 = state_v2.study.best_trial.user_attrs.get('cv_scores', [state_v2.best_score])
else:
fold_scores_v2 = state_v2.cv_scores if state_v2.cv_scores else [state_v2.best_score]
metadata_v2.add_cv_scores("roc_auc", fold_scores_v2)
metadata_v2.add_dataset(X_train_t, y_train, "train")
metadata_v2.add_dataset(X_test_t, y_test, "test")
v2_dir = models_dir / "credit_default_predictor" / "2.0.0"
v2_dir.mkdir(parents=True, exist_ok=True)
v2_path = str(v2_dir / "model.pkl")
ModelSerializer.save(
model=state_v2.model,
path=v2_path,
metadata=metadata_v2,
train_data=(X_train_t, y_train),
test_data=(X_test_t, y_test),
include_datasets=True
)
print("V2 saved.")
for f in sorted((models_dir / "credit_default_predictor").rglob("*")):
if f.is_file():
indent = " " * (len(f.relative_to(models_dir).parts) - 1)
print(f" {indent}{f.name} ({f.stat().st_size / 1024:.1f} KB)")
metadata_v2 = ModelMetadata(
name="credit_default_predictor",
version="2.0.0",
model_type=state_v2.model.model_type,
framework=state_v2.model.framework,
task="binary_classification",
trained_at=datetime.now(),
feature_names=list(X_train_t.columns),
n_features=X_train_t.shape[1],
hyperparameters=state_v2.best_params,
classes=[0, 1],
optimal_threshold=threshold_v2,
training_time_seconds=state_v2.training_time_seconds,
tags=["binary_classification", "credit_default", "lightgbm", "uci"],
notes="Credit default predictor v2.0. Retrained with different random seed."
)
test_report_v2 = evaluate_classification(
y_true=y_test,
y_pred_proba=y_proba_v2,
threshold=threshold_v2,
labels=[0, 1],
)
metadata_v2.add_metric("test_roc_auc", test_report_v2.metrics["roc_auc"])
metadata_v2.add_metric("test_f1", test_report_v2.metrics["f1_score"])
metadata_v2.add_metric("test_accuracy", test_report_v2.metrics["accuracy"])
metadata_v2.add_feature_schema(X_train_t)
metadata_v2.preprocessing = metadata_v1.preprocessing.copy()
metadata_v2.search_metadata = {
"optimizer": config_v2.optimizer,
"optuna_sampler": config_v2.optuna_sampler,
"n_trials": config_v2.n_trials,
"cv_folds": config_v2.cv_folds,
"optimization_metric": config_v2.optimization_metric,
"best_params": state_v2.best_params,
}
metadata_v2.target_mapping = metadata_v1.target_mapping.copy()
metadata_v2.inference_contract = {
**metadata_v1.inference_contract,
"decision_threshold": threshold_v2,
}
metadata_v2.add_artifact_metadata(classification_metrics=test_report_v2.to_dict())
if state_v2.study:
fold_scores_v2 = state_v2.study.best_trial.user_attrs.get('cv_scores', [state_v2.best_score])
else:
fold_scores_v2 = state_v2.cv_scores if state_v2.cv_scores else [state_v2.best_score]
metadata_v2.add_cv_scores("roc_auc", fold_scores_v2)
metadata_v2.add_dataset(X_train_t, y_train, "train")
metadata_v2.add_dataset(X_test_t, y_test, "test")
v2_dir = models_dir / "credit_default_predictor" / "2.0.0"
v2_dir.mkdir(parents=True, exist_ok=True)
v2_path = str(v2_dir / "model.pkl")
ModelSerializer.save(
model=state_v2.model,
path=v2_path,
metadata=metadata_v2,
train_data=(X_train_t, y_train),
test_data=(X_test_t, y_test),
include_datasets=True
)
print("V2 saved.")
for f in sorted((models_dir / "credit_default_predictor").rglob("*")):
if f.is_file():
indent = " " * (len(f.relative_to(models_dir).parts) - 1)
print(f" {indent}{f.name} ({f.stat().st_size / 1024:.1f} KB)")
16. Comparing V1 vs V2 — Metadata-Driven Version Diffing¶
In [ ]:
Copied!
v1_meta_dict = ModelSerializer.load_metadata(v1_path)
v2_meta_dict = ModelSerializer.load_metadata(v2_path)
v1_metrics = v1_meta_dict.get('metrics', {})
v2_metrics = v2_meta_dict.get('metrics', {})
all_metric_names = sorted(set(v1_metrics) | set(v2_metrics))
print(f"{'Metric':<22s} {'V1':>8s} {'V2':>8s} {'Delta':>10s} Direction")
print("-" * 62)
for metric in all_metric_names:
v1_val = v1_metrics.get(metric)
v2_val = v2_metrics.get(metric)
if v1_val is None or v2_val is None:
continue
delta = v2_val - v1_val
direction = "IMPROVED" if delta > 0 else ("REGRESSED" if delta < 0 else "UNCHANGED")
arrow = "^" if delta > 0 else ("v" if delta < 0 else "=")
print(f"{metric:<22s} {v1_val:>8.4f} {v2_val:>8.4f} {delta:>+10.4f} {arrow} {direction}")
v1_meta_dict = ModelSerializer.load_metadata(v1_path)
v2_meta_dict = ModelSerializer.load_metadata(v2_path)
v1_metrics = v1_meta_dict.get('metrics', {})
v2_metrics = v2_meta_dict.get('metrics', {})
all_metric_names = sorted(set(v1_metrics) | set(v2_metrics))
print(f"{'Metric':<22s} {'V1':>8s} {'V2':>8s} {'Delta':>10s} Direction")
print("-" * 62)
for metric in all_metric_names:
v1_val = v1_metrics.get(metric)
v2_val = v2_metrics.get(metric)
if v1_val is None or v2_val is None:
continue
delta = v2_val - v1_val
direction = "IMPROVED" if delta > 0 else ("REGRESSED" if delta < 0 else "UNCHANGED")
arrow = "^" if delta > 0 else ("v" if delta < 0 else "=")
print(f"{metric:<22s} {v1_val:>8.4f} {v2_val:>8.4f} {delta:>+10.4f} {arrow} {direction}")
In [ ]:
Copied!
v1_cv = v1_meta_dict.get('cv_scores', {}).get('roc_auc', [])
v2_cv = v2_meta_dict.get('cv_scores', {}).get('roc_auc', [])
if v1_cv and v2_cv:
print(f"V1 CV ({len(v1_cv)} folds): {[round(s, 4) for s in v1_cv]} mean={np.mean(v1_cv):.4f}")
print(f"V2 CV ({len(v2_cv)} folds): {[round(s, 4) for s in v2_cv]} mean={np.mean(v2_cv):.4f}")
v1_cv = v1_meta_dict.get('cv_scores', {}).get('roc_auc', [])
v2_cv = v2_meta_dict.get('cv_scores', {}).get('roc_auc', [])
if v1_cv and v2_cv:
print(f"V1 CV ({len(v1_cv)} folds): {[round(s, 4) for s in v1_cv]} mean={np.mean(v1_cv):.4f}")
print(f"V2 CV ({len(v2_cv)} folds): {[round(s, 4) for s in v2_cv]} mean={np.mean(v2_cv):.4f}")
17. Visualising the V1 vs V2 Metric Comparison¶
In [ ]:
Copied!
BLUE = "#2563EB"
GREEN = "#16A34A"
RED = "#DC2626"
metric_labels = [m.replace("test_", "").upper() for m in all_metric_names]
v1_vals = [v1_metrics.get(m, 0) for m in all_metric_names]
v2_vals = [v2_metrics.get(m, 0) for m in all_metric_names]
n_metrics = len(all_metric_names)
x = np.arange(n_metrics)
bar_w = 0.35
fig, axes = plt.subplots(1, 2, figsize=(13, 5))
fig.suptitle("credit_default_predictor: V1 vs V2 Performance Comparison",
fontsize=13, fontweight="bold")
ax = axes[0]
bars1 = ax.bar(x - bar_w/2, v1_vals, bar_w, label="v1.0.0", color=BLUE, alpha=0.85)
bars2 = ax.bar(x + bar_w/2, v2_vals, bar_w, label="v2.0.0", color=GREEN, alpha=0.85)
for bar in list(bars1) + list(bars2):
ax.text(bar.get_x() + bar.get_width() / 2, bar.get_height() + 0.002,
f"{bar.get_height():.3f}", ha='center', va='bottom', fontsize=8)
ax.set_xticks(x)
ax.set_xticklabels(metric_labels, fontsize=10)
y_min = max(0, min(v1_vals + v2_vals) - 0.05)
ax.set_ylim(y_min, min(1.05, max(v1_vals + v2_vals) + 0.07))
ax.set_title("Test Metrics by Version", fontweight="bold")
ax.set_ylabel("Score")
ax.legend()
ax.grid(axis='y', alpha=0.3)
ax = axes[1]
deltas = [v2 - v1 for v1, v2 in zip(v1_vals, v2_vals)]
colors = [GREEN if d >= 0 else RED for d in deltas]
bars = ax.bar(x, deltas, bar_w * 2, color=colors, alpha=0.85)
for bar, delta in zip(bars, deltas):
va = 'bottom' if delta >= 0 else 'top'
ypos = bar.get_height() + 0.0005 if delta >= 0 else bar.get_height() - 0.0005
ax.text(bar.get_x() + bar.get_width() / 2, ypos,
f"{delta:+.4f}", ha='center', va=va, fontsize=9, fontweight='bold')
ax.axhline(0, color='black', lw=0.8)
ax.set_xticks(x)
ax.set_xticklabels(metric_labels, fontsize=10)
ax.set_title("V2 vs V1 Delta", fontweight="bold")
ax.grid(axis='y', alpha=0.3)
plt.tight_layout()
plt.show()
auc_delta = v2_metrics.get('test_roc_auc', 0) - v1_metrics.get('test_roc_auc', 0)
if auc_delta > 0.005:
print("Recommendation: PROMOTE v2 — meaningful AUC improvement")
elif auc_delta < -0.005:
print("Recommendation: DO NOT PROMOTE — AUC regression vs v1")
else:
print("Recommendation: NEUTRAL — delta within noise threshold (< 0.005)")
BLUE = "#2563EB"
GREEN = "#16A34A"
RED = "#DC2626"
metric_labels = [m.replace("test_", "").upper() for m in all_metric_names]
v1_vals = [v1_metrics.get(m, 0) for m in all_metric_names]
v2_vals = [v2_metrics.get(m, 0) for m in all_metric_names]
n_metrics = len(all_metric_names)
x = np.arange(n_metrics)
bar_w = 0.35
fig, axes = plt.subplots(1, 2, figsize=(13, 5))
fig.suptitle("credit_default_predictor: V1 vs V2 Performance Comparison",
fontsize=13, fontweight="bold")
ax = axes[0]
bars1 = ax.bar(x - bar_w/2, v1_vals, bar_w, label="v1.0.0", color=BLUE, alpha=0.85)
bars2 = ax.bar(x + bar_w/2, v2_vals, bar_w, label="v2.0.0", color=GREEN, alpha=0.85)
for bar in list(bars1) + list(bars2):
ax.text(bar.get_x() + bar.get_width() / 2, bar.get_height() + 0.002,
f"{bar.get_height():.3f}", ha='center', va='bottom', fontsize=8)
ax.set_xticks(x)
ax.set_xticklabels(metric_labels, fontsize=10)
y_min = max(0, min(v1_vals + v2_vals) - 0.05)
ax.set_ylim(y_min, min(1.05, max(v1_vals + v2_vals) + 0.07))
ax.set_title("Test Metrics by Version", fontweight="bold")
ax.set_ylabel("Score")
ax.legend()
ax.grid(axis='y', alpha=0.3)
ax = axes[1]
deltas = [v2 - v1 for v1, v2 in zip(v1_vals, v2_vals)]
colors = [GREEN if d >= 0 else RED for d in deltas]
bars = ax.bar(x, deltas, bar_w * 2, color=colors, alpha=0.85)
for bar, delta in zip(bars, deltas):
va = 'bottom' if delta >= 0 else 'top'
ypos = bar.get_height() + 0.0005 if delta >= 0 else bar.get_height() - 0.0005
ax.text(bar.get_x() + bar.get_width() / 2, ypos,
f"{delta:+.4f}", ha='center', va=va, fontsize=9, fontweight='bold')
ax.axhline(0, color='black', lw=0.8)
ax.set_xticks(x)
ax.set_xticklabels(metric_labels, fontsize=10)
ax.set_title("V2 vs V1 Delta", fontweight="bold")
ax.grid(axis='y', alpha=0.3)
plt.tight_layout()
plt.show()
auc_delta = v2_metrics.get('test_roc_auc', 0) - v1_metrics.get('test_roc_auc', 0)
if auc_delta > 0.005:
print("Recommendation: PROMOTE v2 — meaningful AUC improvement")
elif auc_delta < -0.005:
print("Recommendation: DO NOT PROMOTE — AUC regression vs v1")
else:
print("Recommendation: NEUTRAL — delta within noise threshold (< 0.005)")
18. Production Checklist¶
In [ ]:
Copied!
def pre_promotion_check(model_path: str, min_roc_auc: float = 0.70) -> bool:
checks = {}
try:
meta = ModelSerializer.load_metadata(model_path)
checks["metadata_readable"] = True
except Exception as e:
print(f" FAIL metadata_readable: {e}")
return False
auc = (meta.get('metrics') or {}).get('test_roc_auc', 0)
checks["auc_floor"] = auc >= min_roc_auc
checks["feature_contract"] = bool(meta.get('feature_names'))
checks["threshold_set"] = meta.get('optimal_threshold') is not None
checks["data_hash_present"] = bool((meta.get('training_dataset') or {}).get('data_hash'))
checks["notes_populated"] = bool((meta.get('notes') or '').strip())
try:
pkg = ModelSerializer.load(model_path)
if pkg.test_data:
p1 = pkg.model.predict_proba(pkg.test_data['X'])[:, 1]
p2 = pkg.model.predict_proba(pkg.test_data['X'])[:, 1]
checks["round_trip"] = np.abs(p1 - p2).max() < 1e-10
else:
checks["round_trip"] = None
except Exception as e:
checks["round_trip"] = False
print(f"Pre-promotion checks for: {model_path}")
print(f" Min AUC floor: {min_roc_auc} | Actual AUC: {auc:.4f}")
print()
all_pass = True
for check, result in checks.items():
symbol = "SKIP" if result is None else ("PASS" if result else "FAIL")
if result is False:
all_pass = False
print(f" [{symbol}] {check}")
print()
print(f" Overall: {'READY TO PROMOTE' if all_pass else 'BLOCKED — fix failures above'}")
return all_pass
print("Running pre-promotion checks on V2 model:\n")
result = pre_promotion_check(v2_path, min_roc_auc=0.70)
def pre_promotion_check(model_path: str, min_roc_auc: float = 0.70) -> bool:
checks = {}
try:
meta = ModelSerializer.load_metadata(model_path)
checks["metadata_readable"] = True
except Exception as e:
print(f" FAIL metadata_readable: {e}")
return False
auc = (meta.get('metrics') or {}).get('test_roc_auc', 0)
checks["auc_floor"] = auc >= min_roc_auc
checks["feature_contract"] = bool(meta.get('feature_names'))
checks["threshold_set"] = meta.get('optimal_threshold') is not None
checks["data_hash_present"] = bool((meta.get('training_dataset') or {}).get('data_hash'))
checks["notes_populated"] = bool((meta.get('notes') or '').strip())
try:
pkg = ModelSerializer.load(model_path)
if pkg.test_data:
p1 = pkg.model.predict_proba(pkg.test_data['X'])[:, 1]
p2 = pkg.model.predict_proba(pkg.test_data['X'])[:, 1]
checks["round_trip"] = np.abs(p1 - p2).max() < 1e-10
else:
checks["round_trip"] = None
except Exception as e:
checks["round_trip"] = False
print(f"Pre-promotion checks for: {model_path}")
print(f" Min AUC floor: {min_roc_auc} | Actual AUC: {auc:.4f}")
print()
all_pass = True
for check, result in checks.items():
symbol = "SKIP" if result is None else ("PASS" if result else "FAIL")
if result is False:
all_pass = False
print(f" [{symbol}] {check}")
print()
print(f" Overall: {'READY TO PROMOTE' if all_pass else 'BLOCKED — fix failures above'}")
return all_pass
print("Running pre-promotion checks on V2 model:\n")
result = pre_promotion_check(v2_path, min_roc_auc=0.70)
19. Cleanup¶
In [ ]:
Copied!
shutil.rmtree("models", ignore_errors=True)
print("Demo artefacts removed.")
shutil.rmtree("models", ignore_errors=True)
print("Demo artefacts removed.")
20. Conclusion¶
| Problem | BitBullet Solution |
|---|---|
| Lost model lineage | ModelMetadata captures identity, features, hyperparameters, metrics, dataset hashes |
| Unreproducible evaluations | Test metrics locked into metadata after evaluation |
| Silent data drift | data_hash — cryptographic fingerprint of every training dataset |
| Expensive metadata scans | _metadata.json sidecar — compare versions without loading pickle |
| Oversized experiment packages | save_model_only() + load_model_only() for lean model-only artefacts |
| Architecture coupling | ModelRegistry — create any model by name |
| No rollback | Versioned directory structure — previous versions always loadable |
Continue the Academy:
01_Binary_Classification.ipynb— end-to-end binary classification with SHAP and formal reports02_Multi_Class_Classification.ipynb— extending the workflow to seven-class targets03_Clustering.ipynb— unsupervised wholesale customer segmentation04_Data_Transformations.ipynb— deep dive into every transform in the library