BitBullet:Lessons — 04: Data Transformations¶

A Deep Dive into bitbullet.transform¶

---

Raw data does not feed models cleanly. A feature with extreme right skew pushes tree splits to the extremes. An unscaled column ranging [0, 1,000,000] next to one ranging [0, 1] biases distance-based algorithms. A categorical column with 50 unique values explodes into 50 one-hot columns unless you have a smarter strategy.

bitbullet.transform is built around one critical constraint: transformers must be fit only on training data and applied identically to test data. Violating this rule is one of the most common sources of optimistic, non-reproducible models in the industry.

Dataset: default_of_credit_card_clients.xls — 30,000 Taiwanese credit card clients, 23 features after cleaning, binary target: default_payment.
Source: UCI ML Repository — Default of Credit Card Clients
Context: Predict whether a client will default on next month's payment based on credit limit, repayment history, bill statements, and demographics.

---

Notebook Roadmap¶

Section	Topic
2	Setup and imports
3	Load and clean data
4	EDA with generate_feature_stats
5	The pipeline mental model
6	Train/test split
7a–7e	Numerical transforms: log/power, scaling, quantile, winsorize, binning
8a–8f	Categorical transforms: label, one-hot, frequency, target, rare-bin, hash
9	Composing a production pipeline
10	fit_transform on train
11	Save, reload, transform test set
12	Pipeline management: list, disable, enable
13	Visualising transformations
14	Pipeline metadata inspection
15	Conclusion

Let a guided workflow carry reviewed transformations through to training. BitBullet Platform centralises datasets, profiles, managed compute, storage, and modelling work in one place. Configure and review transformations with guided controls or AI-assisted drafting, compare completed results, and export the fitted preprocessing with the trained result, metadata, and generated inference code. This lesson provides the same transformation discipline directly in the BitBullet SDK.

2. Setup and Imports¶

In [1]:

import sys
import os
import warnings
warnings.filterwarnings('ignore')

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split

# When running this notebook from bitbullet/lessons in a local clone, prefer the local SDK source.
sdk_root = os.path.abspath("..")
if sdk_root not in sys.path:
    sys.path.insert(0, sdk_root)

from bitbullet.transform import (
    TransformPipeline,
    BaseTransformer,
    TransformConfig,
    NumericalTransformer,
    CategoricalTransformer,
    DateTimeTransformer,
    generate_feature_stats,
)
from bitbullet.transform.eda import plot_pipeline_transformations

import bitbullet
print(f"bitbullet version : {getattr(bitbullet, '__version__', 'dev')}")
print("All imports successful.")

plt.rcParams.update({
    'figure.facecolor': 'white',
    'axes.spines.top': False,
    'axes.spines.right': False,
    'axes.grid': True,
    'grid.alpha': 0.3,
    'font.size': 11,
})
BLUE   = '#2563EB'
ORANGE = '#F59E0B'
GREEN  = '#10B981'
RED    = '#EF4444'
GREY   = '#9CA3AF'

import sys import os import warnings warnings.filterwarnings('ignore') import pandas as pd import numpy as np import matplotlib.pyplot as plt import seaborn as sns from sklearn.model_selection import train_test_split # When running this notebook from bitbullet/lessons in a local clone, prefer the local SDK source. sdk_root = os.path.abspath("..") if sdk_root not in sys.path: sys.path.insert(0, sdk_root) from bitbullet.transform import ( TransformPipeline, BaseTransformer, TransformConfig, NumericalTransformer, CategoricalTransformer, DateTimeTransformer, generate_feature_stats, ) from bitbullet.transform.eda import plot_pipeline_transformations import bitbullet print(f"bitbullet version : {getattr(bitbullet, '__version__', 'dev')}") print("All imports successful.") plt.rcParams.update({ 'figure.facecolor': 'white', 'axes.spines.top': False, 'axes.spines.right': False, 'axes.grid': True, 'grid.alpha': 0.3, 'font.size': 11, }) BLUE = '#2563EB' ORANGE = '#F59E0B' GREEN = '#10B981' RED = '#EF4444' GREY = '#9CA3AF'

bitbullet version : dev
All imports successful.

3. Load and Clean the Dataset¶

Dataset: Default of Credit Card Clients Dataset
Source: UCI ML Repository — Default of Credit Card Clients
File: default_of_credit_card_clients.xls

Before running this cell: download default_of_credit_card_clients.xls from the link above and place it in the same directory as this notebook. If you store it elsewhere, update data_path in the code cell below to match your chosen location.

In [ ]:

# Update this path if you stored the file in a different location.
data_path = "default_of_credit_card_clients.xls"
# The XLS file has a descriptive first row — header is on row 1 (0-indexed)
df_raw = pd.read_excel(data_path, header=1)

print(f"Raw shape: {df_raw.shape}")
print(f"Columns: {list(df_raw.columns)}")

# Update this path if you stored the file in a different location. data_path = "default_of_credit_card_clients.xls" # The XLS file has a descriptive first row — header is on row 1 (0-indexed) df_raw = pd.read_excel(data_path, header=1) print(f"Raw shape: {df_raw.shape}") print(f"Columns: {list(df_raw.columns)}")

Raw shape: (30000, 25)
Columns: ['ID', 'LIMIT_BAL', 'SEX', 'EDUCATION', 'MARRIAGE', 'AGE', 'PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6', 'BILL_AMT1', 'BILL_AMT2', 'BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6', 'PAY_AMT1', 'PAY_AMT2', 'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6', 'default payment next month']

In [5]:

# Drop the ID column — it is a row identifier with no predictive value
df = df_raw.drop(columns=['ID'])

# Rename target to a clean identifier
df = df.rename(columns={'default payment next month': 'default_payment'})

# Map integer-coded categorical columns to descriptive strings
df['SEX'] = df['SEX'].map({1: 'Male', 2: 'Female'})
df['EDUCATION'] = df['EDUCATION'].map({
    1: 'Graduate', 2: 'University', 3: 'High_School', 4: 'Other',
    5: 'Other', 6: 'Other', 0: 'Other'   # values 0, 5, 6 are undocumented — treat as Other
})
df['MARRIAGE'] = df['MARRIAGE'].map({
    1: 'Married', 2: 'Single', 3: 'Other', 0: 'Other'
})

TARGET = 'default_payment'

print(f"Dataset shape (after cleaning): {df.shape}")
print(f"\nTarget distribution:")
counts = df[TARGET].value_counts()
print(f"  No default (0) : {counts[0]:,}  ({counts[0]/len(df)*100:.1f}%)")
print(f"  Default    (1) : {counts[1]:,}  ({counts[1]/len(df)*100:.1f}%)")
print()
display(df.head())

# Drop the ID column — it is a row identifier with no predictive value df = df_raw.drop(columns=['ID']) # Rename target to a clean identifier df = df.rename(columns={'default payment next month': 'default_payment'}) # Map integer-coded categorical columns to descriptive strings df['SEX'] = df['SEX'].map({1: 'Male', 2: 'Female'}) df['EDUCATION'] = df['EDUCATION'].map({ 1: 'Graduate', 2: 'University', 3: 'High_School', 4: 'Other', 5: 'Other', 6: 'Other', 0: 'Other' # values 0, 5, 6 are undocumented — treat as Other }) df['MARRIAGE'] = df['MARRIAGE'].map({ 1: 'Married', 2: 'Single', 3: 'Other', 0: 'Other' }) TARGET = 'default_payment' print(f"Dataset shape (after cleaning): {df.shape}") print(f"\nTarget distribution:") counts = df[TARGET].value_counts() print(f" No default (0) : {counts[0]:,} ({counts[0]/len(df)*100:.1f}%)") print(f" Default (1) : {counts[1]:,} ({counts[1]/len(df)*100:.1f}%)") print() display(df.head())

Dataset shape (after cleaning): (30000, 24)

Target distribution:
  No default (0) : 23,364  (77.9%)
  Default    (1) : 6,636  (22.1%)

	LIMIT_BAL	SEX	EDUCATION	MARRIAGE	AGE	PAY_0	PAY_2	PAY_3	PAY_4	PAY_5	...	BILL_AMT4	BILL_AMT5	BILL_AMT6	PAY_AMT1	PAY_AMT2	PAY_AMT3	PAY_AMT4	PAY_AMT5	PAY_AMT6	default_payment
0	20000	Female	University	Married	24	2	2	-1	-1	-2	...	0	0	0	0	689	0	0	0	0	1
1	120000	Female	University	Single	26	-1	2	0	0	0	...	3272	3455	3261	0	1000	1000	1000	0	2000	1
2	90000	Female	University	Single	34	0	0	0	0	0	...	14331	14948	15549	1518	1500	1000	1000	1000	5000	0
3	50000	Female	University	Married	37	0	0	0	0	0	...	28314	28959	29547	2000	2019	1200	1100	1069	1000	0
4	50000	Male	University	Married	57	-1	0	-1	0	0	...	20940	19146	19131	2000	36681	10000	9000	689	679	0

5 rows × 24 columns

Column Glossary¶

Column	Type	Description
LIMIT_BAL	numerical	Credit limit (NT dollar)
SEX	categorical	Male / Female
EDUCATION	categorical	Graduate / University / High_School / Other
MARRIAGE	categorical	Married / Single / Other
AGE	numerical	Age in years
PAY_0–PAY_6	numerical	Repayment status for Sept–April (-2=no consumption, -1=paid duly, 0=revolving credit, 1–9=months delayed)
BILL_AMT1–BILL_AMT6	numerical	Bill statement amount Sept–April (NT dollar)
PAY_AMT1–PAY_AMT6	numerical	Previous payment amount Sept–April (NT dollar)
default_payment	target	1 = defaulted next month, 0 = did not

4. EDA with generate_feature_stats¶

In [6]:

X_full = df.drop(columns=[TARGET])
y_full = df[TARGET]

stats_df, numerical_cols, categorical_cols = generate_feature_stats(X_full)

print(f"Numerical   ({len(numerical_cols)}): {numerical_cols}")
print(f"Categorical ({len(categorical_cols)}): {categorical_cols}")
print()
display(stats_df)

X_full = df.drop(columns=[TARGET]) y_full = df[TARGET] stats_df, numerical_cols, categorical_cols = generate_feature_stats(X_full) print(f"Numerical ({len(numerical_cols)}): {numerical_cols}") print(f"Categorical ({len(categorical_cols)}): {categorical_cols}") print() display(stats_df)

Numerical   (20): ['LIMIT_BAL', 'AGE', 'PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6', 'BILL_AMT1', 'BILL_AMT2', 'BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6', 'PAY_AMT1', 'PAY_AMT2', 'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6']
Categorical (3): ['SEX', 'EDUCATION', 'MARRIAGE']

	dtype	missing_count	missing_percent	unique_values	zero_count	negative_count	mean	std	skewness	kurtosis	mean_percentile	min	25%	50%	75%	max	top	freq
LIMIT_BAL	int64	0	0.0	81	0	0	167484.322667	129747.661567	0.992867	0.536263	56.98	10000.0	50000.0	140000.0	240000.0	1000000.0	-	-
SEX	object	0	0.0	2	0	0	-	-	-	-	-	-	-	-	-	-	Female	18112
EDUCATION	object	0	0.0	4	0	0	-	-	-	-	-	-	-	-	-	-	University	14030
MARRIAGE	object	0	0.0	3	0	0	-	-	-	-	-	-	-	-	-	-	Single	15964
AGE	int64	0	0.0	56	0	0	35.4855	9.217904	0.732246	0.044303	56.03	21.0	28.0	34.0	41.0	79.0	-	-
PAY_0	int64	0	0.0	11	14737	8445	-0.0167	1.123802	0.731975	2.720715	28.15	-2.0	-1.0	0.0	0.0	8.0	-	-
PAY_2	int64	0	0.0	11	15730	9832	-0.133767	1.197186	0.790565	1.570418	32.773333	-2.0	-1.0	0.0	0.0	8.0	-	-
PAY_3	int64	0	0.0	11	15764	10023	-0.1662	1.196868	0.840682	2.084436	33.41	-2.0	-1.0	0.0	0.0	8.0	-	-
PAY_4	int64	0	0.0	11	16455	10035	-0.220667	1.169139	0.999629	3.496983	33.45	-2.0	-1.0	0.0	0.0	8.0	-	-
PAY_5	int64	0	0.0	10	16947	10085	-0.2662	1.133187	1.008197	3.989748	33.616667	-2.0	-1.0	0.0	0.0	8.0	-	-
PAY_6	int64	0	0.0	10	16286	10635	-0.2911	1.149988	0.948029	3.426534	35.45	-2.0	-1.0	0.0	0.0	8.0	-	-
BILL_AMT1	int64	0	0.0	22723	2008	590	51223.3309	73635.860576	2.663861	9.806289	69.516667	-165580.0	3558.75	22381.5	67091.0	964511.0	-	-
BILL_AMT2	int64	0	0.0	22346	2506	669	49179.075167	71173.768783	2.705221	10.302946	68.426667	-69777.0	2984.75	21200.0	64006.25	983931.0	-	-
BILL_AMT3	int64	0	0.0	22026	2870	655	47013.1548	69349.387427	3.08783	19.783255	68.073333	-157264.0	2666.25	20088.5	60164.75	1664089.0	-	-
BILL_AMT4	int64	0	0.0	21548	3195	675	43262.948967	64332.856134	2.821965	11.309325	68.856667	-170000.0	2326.75	19052.0	54506.0	891586.0	-	-
BILL_AMT5	int64	0	0.0	21010	3506	655	40311.400967	60797.15577	2.87638	12.305881	69.696667	-81334.0	1763.0	18104.5	50190.5	927171.0	-	-
BILL_AMT6	int64	0	0.0	20604	4020	688	38871.7604	59554.107537	2.846645	12.270705	69.846667	-339603.0	1256.0	17071.0	49198.25	961664.0	-	-
PAY_AMT1	int64	0	0.0	7943	5249	0	5663.5805	16563.280354	14.668364	415.254743	77.59	0.0	1000.0	2100.0	5006.0	873552.0	-	-
PAY_AMT2	int64	0	0.0	7899	5396	0	5921.1635	23040.870402	30.453817	1641.631911	79.133333	0.0	833.0	2009.0	5000.0	1684259.0	-	-
PAY_AMT3	int64	0	0.0	7518	5968	0	5225.6815	17606.96147	17.216635	564.311229	79.873333	0.0	390.0	1800.0	4505.0	896040.0	-	-
PAY_AMT4	int64	0	0.0	6937	6408	0	4826.076867	15666.159744	12.904985	277.333768	77.683333	0.0	296.0	1500.0	4013.25	621000.0	-	-
PAY_AMT5	int64	0	0.0	6897	6703	0	4799.387633	15278.305679	11.127417	180.06394	77.5	0.0	252.5	1500.0	4031.5	426529.0	-	-
PAY_AMT6	int64	0	0.0	6939	7173	0	5215.502567	17777.465775	10.640727	167.16143	81.986667	0.0	117.75	1500.0	4000.0	528666.0	-	-

In [7]:

num_stats = stats_df.loc[numerical_cols, ['dtype', 'missing_percent', 'skewness', 'kurtosis',
                                                   'zero_count', 'negative_count', 'min', 'max']]
print("Numerical feature audit:")
display(num_stats)

print()
print("Categorical feature audit:")
cat_stats = stats_df.loc[categorical_cols, ['dtype', 'missing_percent', 'unique_values', 'top', 'freq']]
display(cat_stats)

num_stats = stats_df.loc[numerical_cols, ['dtype', 'missing_percent', 'skewness', 'kurtosis', 'zero_count', 'negative_count', 'min', 'max']] print("Numerical feature audit:") display(num_stats) print() print("Categorical feature audit:") cat_stats = stats_df.loc[categorical_cols, ['dtype', 'missing_percent', 'unique_values', 'top', 'freq']] display(cat_stats)

Numerical feature audit:

	dtype	missing_percent	skewness	kurtosis	zero_count	negative_count	min	max
LIMIT_BAL	int64	0.0	0.992867	0.536263	0	0	10000.0	1000000.0
AGE	int64	0.0	0.732246	0.044303	0	0	21.0	79.0
PAY_0	int64	0.0	0.731975	2.720715	14737	8445	-2.0	8.0
PAY_2	int64	0.0	0.790565	1.570418	15730	9832	-2.0	8.0
PAY_3	int64	0.0	0.840682	2.084436	15764	10023	-2.0	8.0
PAY_4	int64	0.0	0.999629	3.496983	16455	10035	-2.0	8.0
PAY_5	int64	0.0	1.008197	3.989748	16947	10085	-2.0	8.0
PAY_6	int64	0.0	0.948029	3.426534	16286	10635	-2.0	8.0
BILL_AMT1	int64	0.0	2.663861	9.806289	2008	590	-165580.0	964511.0
BILL_AMT2	int64	0.0	2.705221	10.302946	2506	669	-69777.0	983931.0
BILL_AMT3	int64	0.0	3.08783	19.783255	2870	655	-157264.0	1664089.0
BILL_AMT4	int64	0.0	2.821965	11.309325	3195	675	-170000.0	891586.0
BILL_AMT5	int64	0.0	2.87638	12.305881	3506	655	-81334.0	927171.0
BILL_AMT6	int64	0.0	2.846645	12.270705	4020	688	-339603.0	961664.0
PAY_AMT1	int64	0.0	14.668364	415.254743	5249	0	0.0	873552.0
PAY_AMT2	int64	0.0	30.453817	1641.631911	5396	0	0.0	1684259.0
PAY_AMT3	int64	0.0	17.216635	564.311229	5968	0	0.0	896040.0
PAY_AMT4	int64	0.0	12.904985	277.333768	6408	0	0.0	621000.0
PAY_AMT5	int64	0.0	11.127417	180.06394	6703	0	0.0	426529.0
PAY_AMT6	int64	0.0	10.640727	167.16143	7173	0	0.0	528666.0

Categorical feature audit:

	dtype	missing_percent	unique_values	top	freq
SEX	object	0.0	2	Female	18112
EDUCATION	object	0.0	4	University	14030
MARRIAGE	object	0.0	3	Single	15964

Reading the Stats Table — Key Observations for This Dataset¶

• PAY_AMT1–PAY_AMT6: extreme right skew (> 10). Clients who pay large lump sums dominate the tail. log1p is the right choice — safe for zeros, compresses the right tail.
• BILL_AMT1–BILL_AMT6: moderate-to-high right skew, and some negative values (credit balance). yeo_johnson handles negatives that log and log1p cannot.
• LIMIT_BAL: right-skewed credit limits. log1p reduces the long tail.
• AGE: roughly normal — standard_scale is appropriate.
• PAY_0–PAY_6: ordinal integer repayment status (-2 to 9). Treat as numerical; robust_scale is appropriate for this bounded ordinal range.
• SEX, EDUCATION, MARRIAGE: three low-cardinality categoricals, 2–4 unique values each. label_encode works for tree models; onehot_encode for linear models.

5. The Pipeline Mental Model¶

Before writing a single transform, we internalise one rule:

Transformers must be fit exclusively on training data. They must then be applied — without refitting — to validation and test data.

What TransformPipeline Remembers After Fitting¶

Transform	Fitted Parameters Stored
standard_scale	mean, std per column
robust_scale	median, IQR per column
yeo_johnson	PowerTransformer object per column
log1p	Stateless — no fitting required
winsorize	Lower and upper clip thresholds per column
label_encode	LabelEncoder object per column
onehot_encode	OneHotEncoder object per column
target_encode	Leakage-aware target statistics: OOF values from fit_transform(..., y=target) for training, full-data smoothed map for transform(...) at inference

6. Train/Test Split¶

In [8]:

X_train, X_test, y_train, y_test = train_test_split(
    X_full, y_full, test_size=0.20, random_state=42, stratify=y_full
)

print(f"Training set : {X_train.shape[0]:,} samples  (default rate: {y_train.mean():.3f})")
print(f"Test set     : {X_test.shape[0]:,} samples  (default rate: {y_test.mean():.3f})")
print()
print("Test set is sealed. No fitting step will touch it until final evaluation.")

X_train, X_test, y_train, y_test = train_test_split( X_full, y_full, test_size=0.20, random_state=42, stratify=y_full ) print(f"Training set : {X_train.shape[0]:,} samples (default rate: {y_train.mean():.3f})") print(f"Test set : {X_test.shape[0]:,} samples (default rate: {y_test.mean():.3f})") print() print("Test set is sealed. No fitting step will touch it until final evaluation.")

Training set : 24,000 samples  (default rate: 0.221)
Test set     : 6,000 samples  (default rate: 0.221)

Test set is sealed. No fitting step will touch it until final evaluation.

---

7. Numerical Transformations¶

7a. Logarithmic and Power Transforms: log1p, yeo_johnson, sqrt, boxcox¶

Decision rule for this dataset:

• PAY_AMT columns: zeros present, right-skewed → log1p
• BILL_AMT columns: possible negatives, right-skewed → yeo_johnson
• LIMIT_BAL: strictly positive, right-skewed → log1p

In [9]:

fig, axes = plt.subplots(2, 4, figsize=(18, 9))
fig.suptitle("Log / Power Transforms — Before vs After", fontsize=14, fontweight='bold')

transforms_demo = [
    ('PAY_AMT1', 'log1p',      BLUE,   GREEN),
    ('PAY_AMT2', 'log1p',      BLUE,   GREEN),
    ('BILL_AMT1','yeo_johnson', BLUE,   ORANGE),
    ('LIMIT_BAL','log1p',      BLUE,   ORANGE),
]

results_power = {}
for col, method, c1, c2 in transforms_demo:
    cfg = TransformConfig(name=f"{method}_{col}", transformer_type="numerical",
                          method=method, columns=[col], params={})
    t = NumericalTransformer(cfg)
    results_power[col] = t.fit_transform(X_train)

for i, (col, method, c1, c2) in enumerate(transforms_demo):
    before = X_train[col]
    after  = results_power[col][col]

    axes[0, i].hist(before.dropna(), bins=40, color=c1, alpha=0.8)
    axes[0, i].set_title(f"`{col}`\n(before)", fontsize=10, fontweight='bold')
    axes[0, i].text(0.97, 0.95, f'skew={before.skew():.2f}', transform=axes[0, i].transAxes,
                    ha='right', va='top', fontsize=9,
                    bbox=dict(boxstyle='round,pad=0.3', facecolor='white', edgecolor='grey', alpha=0.8))

    axes[1, i].hist(after.dropna(), bins=40, color=c2, alpha=0.8)
    axes[1, i].set_title(f"`{method}`\n(after)", fontsize=10, fontweight='bold')
    axes[1, i].text(0.97, 0.95, f'skew={after.skew():.2f}', transform=axes[1, i].transAxes,
                    ha='right', va='top', fontsize=9,
                    bbox=dict(boxstyle='round,pad=0.3', facecolor='white', edgecolor='grey', alpha=0.8))

plt.tight_layout()
plt.show()

print("Skewness reduction summary:")
for col, method, _, _ in transforms_demo:
    before_skew = X_train[col].skew()
    after_skew  = results_power[col][col].skew()
    print(f"  {col:15s} [{method:12s}]  {before_skew:+.3f} \u2192 {after_skew:+.3f}")

fig, axes = plt.subplots(2, 4, figsize=(18, 9)) fig.suptitle("Log / Power Transforms — Before vs After", fontsize=14, fontweight='bold') transforms_demo = [ ('PAY_AMT1', 'log1p', BLUE, GREEN), ('PAY_AMT2', 'log1p', BLUE, GREEN), ('BILL_AMT1','yeo_johnson', BLUE, ORANGE), ('LIMIT_BAL','log1p', BLUE, ORANGE), ] results_power = {} for col, method, c1, c2 in transforms_demo: cfg = TransformConfig(name=f"{method}_{col}", transformer_type="numerical", method=method, columns=[col], params={}) t = NumericalTransformer(cfg) results_power[col] = t.fit_transform(X_train) for i, (col, method, c1, c2) in enumerate(transforms_demo): before = X_train[col] after = results_power[col][col] axes[0, i].hist(before.dropna(), bins=40, color=c1, alpha=0.8) axes[0, i].set_title(f"`{col}`\n(before)", fontsize=10, fontweight='bold') axes[0, i].text(0.97, 0.95, f'skew={before.skew():.2f}', transform=axes[0, i].transAxes, ha='right', va='top', fontsize=9, bbox=dict(boxstyle='round,pad=0.3', facecolor='white', edgecolor='grey', alpha=0.8)) axes[1, i].hist(after.dropna(), bins=40, color=c2, alpha=0.8) axes[1, i].set_title(f"`{method}`\n(after)", fontsize=10, fontweight='bold') axes[1, i].text(0.97, 0.95, f'skew={after.skew():.2f}', transform=axes[1, i].transAxes, ha='right', va='top', fontsize=9, bbox=dict(boxstyle='round,pad=0.3', facecolor='white', edgecolor='grey', alpha=0.8)) plt.tight_layout() plt.show() print("Skewness reduction summary:") for col, method, _, _ in transforms_demo: before_skew = X_train[col].skew() after_skew = results_power[col][col].skew() print(f" {col:15s} [{method:12s}] {before_skew:+.3f} \u2192 {after_skew:+.3f}")

Skewness reduction summary:
  PAY_AMT1        [log1p       ]  +15.312 → -1.299
  PAY_AMT2        [log1p       ]  +21.124 → -1.241
  BILL_AMT1       [yeo_johnson ]  +2.691 → -2.029
  LIMIT_BAL       [log1p       ]  +1.000 → -0.517

7b. Scaling Transforms: standard_scale, robust_scale¶

In [10]:

fig, axes = plt.subplots(1, 3, figsize=(16, 5))
fig.suptitle("Scaling Transforms on `AGE`", fontsize=13, fontweight='bold')

axes[0].hist(X_train['AGE'], bins=30, color=GREY, alpha=0.8)
axes[0].set_title(f"Original\nrange [{X_train['AGE'].min():.0f}, {X_train['AGE'].max():.0f}]", fontweight='bold')

for i, (method, params, color) in enumerate([
    ('standard_scale', {}, BLUE),
    ('robust_scale',   {}, ORANGE),
], start=1):
    cfg = TransformConfig(name=f"{method}_age", transformer_type="numerical",
                          method=method, columns=["AGE"], params=params)
    t = NumericalTransformer(cfg)
    result = t.fit_transform(X_train)['AGE']

    axes[i].hist(result, bins=30, color=color, alpha=0.8)
    axes[i].set_title(f"`{method}`\nrange [{result.min():.2f}, {result.max():.2f}]", fontweight='bold')
    axes[i].set_xlabel('Scaled value')

plt.tight_layout()
plt.show()

# Show outlier effect on PAY_0 (can range from -2 to 9)
print("Repayment status (PAY_0) range in training data:")
print(f"  min={X_train['PAY_0'].min()}, max={X_train['PAY_0'].max()}, unique values: {sorted(X_train['PAY_0'].unique())}")
print("\nUsing robust_scale for PAY columns — resistant to extreme delay values (8, 9 months).")

fig, axes = plt.subplots(1, 3, figsize=(16, 5)) fig.suptitle("Scaling Transforms on `AGE`", fontsize=13, fontweight='bold') axes[0].hist(X_train['AGE'], bins=30, color=GREY, alpha=0.8) axes[0].set_title(f"Original\nrange [{X_train['AGE'].min():.0f}, {X_train['AGE'].max():.0f}]", fontweight='bold') for i, (method, params, color) in enumerate([ ('standard_scale', {}, BLUE), ('robust_scale', {}, ORANGE), ], start=1): cfg = TransformConfig(name=f"{method}_age", transformer_type="numerical", method=method, columns=["AGE"], params=params) t = NumericalTransformer(cfg) result = t.fit_transform(X_train)['AGE'] axes[i].hist(result, bins=30, color=color, alpha=0.8) axes[i].set_title(f"`{method}`\nrange [{result.min():.2f}, {result.max():.2f}]", fontweight='bold') axes[i].set_xlabel('Scaled value') plt.tight_layout() plt.show() # Show outlier effect on PAY_0 (can range from -2 to 9) print("Repayment status (PAY_0) range in training data:") print(f" min={X_train['PAY_0'].min()}, max={X_train['PAY_0'].max()}, unique values: {sorted(X_train['PAY_0'].unique())}") print("\nUsing robust_scale for PAY columns — resistant to extreme delay values (8, 9 months).")

Repayment status (PAY_0) range in training data:
  min=-2, max=8, unique values: [np.int64(-2), np.int64(-1), np.int64(0), np.int64(1), np.int64(2), np.int64(3), np.int64(4), np.int64(5), np.int64(6), np.int64(7), np.int64(8)]

Using robust_scale for PAY columns — resistant to extreme delay values (8, 9 months).

7c. Quantile Transforms: quantile_normal, quantile_uniform¶

In [11]:

fig, axes = plt.subplots(1, 3, figsize=(16, 5))
fig.suptitle("Quantile Transforms on `PAY_AMT1`", fontsize=13, fontweight='bold')

axes[0].hist(X_train['PAY_AMT1'], bins=50, color=GREY, alpha=0.8)
axes[0].set_title(f"Original\nskew = {X_train['PAY_AMT1'].skew():.2f}", fontweight='bold')

for i, (method, color, label) in enumerate([
    ('quantile_normal',  BLUE,   'quantile_normal\n\u2192 Gaussian output'),
    ('quantile_uniform', ORANGE, 'quantile_uniform\n\u2192 Uniform [0,1] output'),
], start=1):
    cfg = TransformConfig(name=f"{method}_pamt1", transformer_type="numerical",
                          method=method, columns=["PAY_AMT1"], params={"n_quantiles": 1000})
    t = NumericalTransformer(cfg)
    result = t.fit_transform(X_train)['PAY_AMT1']

    axes[i].hist(result, bins=50, color=color, alpha=0.8)
    axes[i].set_title(f"{label}\nskew = {result.skew():.4f}", fontweight='bold')
    axes[i].set_xlabel('Transformed value')

plt.tight_layout()
plt.show()
print("quantile_normal achieves near-zero skewness by construction — the most powerful normalisation available.")

fig, axes = plt.subplots(1, 3, figsize=(16, 5)) fig.suptitle("Quantile Transforms on `PAY_AMT1`", fontsize=13, fontweight='bold') axes[0].hist(X_train['PAY_AMT1'], bins=50, color=GREY, alpha=0.8) axes[0].set_title(f"Original\nskew = {X_train['PAY_AMT1'].skew():.2f}", fontweight='bold') for i, (method, color, label) in enumerate([ ('quantile_normal', BLUE, 'quantile_normal\n\u2192 Gaussian output'), ('quantile_uniform', ORANGE, 'quantile_uniform\n\u2192 Uniform [0,1] output'), ], start=1): cfg = TransformConfig(name=f"{method}_pamt1", transformer_type="numerical", method=method, columns=["PAY_AMT1"], params={"n_quantiles": 1000}) t = NumericalTransformer(cfg) result = t.fit_transform(X_train)['PAY_AMT1'] axes[i].hist(result, bins=50, color=color, alpha=0.8) axes[i].set_title(f"{label}\nskew = {result.skew():.4f}", fontweight='bold') axes[i].set_xlabel('Transformed value') plt.tight_layout() plt.show() print("quantile_normal achieves near-zero skewness by construction — the most powerful normalisation available.")

quantile_normal achieves near-zero skewness by construction — the most powerful normalisation available.

7d. Winsorization: winsorize¶

In [12]:

fig, axes = plt.subplots(1, 3, figsize=(16, 5))
fig.suptitle("Winsorization on `LIMIT_BAL` — Clipping Extreme Values", fontsize=13, fontweight='bold')

axes[0].boxplot(X_train['LIMIT_BAL'].dropna(), vert=True, patch_artist=True,
                boxprops=dict(facecolor=GREY, alpha=0.7))
axes[0].set_title(f"Original\nmax={X_train['LIMIT_BAL'].max():,}", fontweight='bold')
axes[0].set_xticks([])

for i, (lower, upper, color) in enumerate([(0.01, 0.99, BLUE), (0.05, 0.95, ORANGE)], start=1):
    cfg = TransformConfig(name=f"winsorize_{int(lower*100)}_{int(upper*100)}_bal",
                          transformer_type="numerical", method="winsorize",
                          columns=["LIMIT_BAL"], params={"lower": lower, "upper": upper})
    t = NumericalTransformer(cfg)
    t.fit(X_train)
    result = t.transform(X_train)['LIMIT_BAL']
    fitted = t.get_params()
    clip_lo, clip_hi = fitted['LIMIT_BAL_lower'], fitted['LIMIT_BAL_upper']

    axes[i].boxplot(result.dropna(), vert=True, patch_artist=True,
                    boxprops=dict(facecolor=color, alpha=0.7))
    axes[i].set_title(f"Winsorize ({int(lower*100)}%–{int(upper*100)}%)\nclipped to [{clip_lo:,.0f}, {clip_hi:,.0f}]",
                      fontweight='bold')
    axes[i].set_xticks([])

plt.tight_layout()
plt.show()

fig, axes = plt.subplots(1, 3, figsize=(16, 5)) fig.suptitle("Winsorization on `LIMIT_BAL` — Clipping Extreme Values", fontsize=13, fontweight='bold') axes[0].boxplot(X_train['LIMIT_BAL'].dropna(), vert=True, patch_artist=True, boxprops=dict(facecolor=GREY, alpha=0.7)) axes[0].set_title(f"Original\nmax={X_train['LIMIT_BAL'].max():,}", fontweight='bold') axes[0].set_xticks([]) for i, (lower, upper, color) in enumerate([(0.01, 0.99, BLUE), (0.05, 0.95, ORANGE)], start=1): cfg = TransformConfig(name=f"winsorize_{int(lower*100)}_{int(upper*100)}_bal", transformer_type="numerical", method="winsorize", columns=["LIMIT_BAL"], params={"lower": lower, "upper": upper}) t = NumericalTransformer(cfg) t.fit(X_train) result = t.transform(X_train)['LIMIT_BAL'] fitted = t.get_params() clip_lo, clip_hi = fitted['LIMIT_BAL_lower'], fitted['LIMIT_BAL_upper'] axes[i].boxplot(result.dropna(), vert=True, patch_artist=True, boxprops=dict(facecolor=color, alpha=0.7)) axes[i].set_title(f"Winsorize ({int(lower*100)}%–{int(upper*100)}%)\nclipped to [{clip_lo:,.0f}, {clip_hi:,.0f}]", fontweight='bold') axes[i].set_xticks([]) plt.tight_layout() plt.show()

7e. Percentile Binning: percentile_binning¶

In [13]:

cfg_bin = TransformConfig(name="bin_age", transformer_type="numerical",
                         method="percentile_binning", columns=["AGE"], params={"n_bins": 4})
t_bin = NumericalTransformer(cfg_bin)
X_binned = t_bin.fit_transform(X_train)
bin_edges = t_bin.get_params()['AGE_bin_edges']

fig, axes = plt.subplots(1, 2, figsize=(14, 5))
fig.suptitle("Percentile Binning on `AGE`", fontsize=13, fontweight='bold')

axes[0].hist(X_train['AGE'], bins=30, color=GREY, alpha=0.8)
for edge in bin_edges[1:-1]:
    axes[0].axvline(edge, color=RED, linestyle='--', lw=1.5, alpha=0.7)
axes[0].set_title('Original — dashed lines show bin edges', fontweight='bold')
axes[0].set_xlabel('AGE')

bin_counts = X_binned['AGE'].value_counts().sort_index()
axes[1].bar([str(b) for b in bin_counts.index], bin_counts.values, color=BLUE, alpha=0.8)
axes[1].set_title('After binning — approximately equal counts per bin', fontweight='bold')
axes[1].set_xlabel('Bin (0 = youngest quartile, 3 = oldest)')
axes[1].set_ylabel('Count')

plt.tight_layout()
plt.show()

print("Learned age quartile edges:")
names = ['Young (Q1)', 'Early-career (Q2)', 'Mid-career (Q3)', 'Senior (Q4)']
for name, (lo, hi) in zip(names, [(bin_edges[i], bin_edges[i+1]) for i in range(len(bin_edges)-1)]):
    print(f"  {name}: age [{lo:.0f}, {hi:.0f}) years")

cfg_bin = TransformConfig(name="bin_age", transformer_type="numerical", method="percentile_binning", columns=["AGE"], params={"n_bins": 4}) t_bin = NumericalTransformer(cfg_bin) X_binned = t_bin.fit_transform(X_train) bin_edges = t_bin.get_params()['AGE_bin_edges'] fig, axes = plt.subplots(1, 2, figsize=(14, 5)) fig.suptitle("Percentile Binning on `AGE`", fontsize=13, fontweight='bold') axes[0].hist(X_train['AGE'], bins=30, color=GREY, alpha=0.8) for edge in bin_edges[1:-1]: axes[0].axvline(edge, color=RED, linestyle='--', lw=1.5, alpha=0.7) axes[0].set_title('Original — dashed lines show bin edges', fontweight='bold') axes[0].set_xlabel('AGE') bin_counts = X_binned['AGE'].value_counts().sort_index() axes[1].bar([str(b) for b in bin_counts.index], bin_counts.values, color=BLUE, alpha=0.8) axes[1].set_title('After binning — approximately equal counts per bin', fontweight='bold') axes[1].set_xlabel('Bin (0 = youngest quartile, 3 = oldest)') axes[1].set_ylabel('Count') plt.tight_layout() plt.show() print("Learned age quartile edges:") names = ['Young (Q1)', 'Early-career (Q2)', 'Mid-career (Q3)', 'Senior (Q4)'] for name, (lo, hi) in zip(names, [(bin_edges[i], bin_edges[i+1]) for i in range(len(bin_edges)-1)]): print(f" {name}: age [{lo:.0f}, {hi:.0f}) years")

Learned age quartile edges:
  Young (Q1): age [21, 28) years
  Early-career (Q2): age [28, 34) years
  Mid-career (Q3): age [34, 41) years
  Senior (Q4): age [41, 75) years

---

8. Categorical Transformations¶

8a. Label Encoding: label_encode¶

In [14]:

cfg_le = TransformConfig(name="label_sex", transformer_type="categorical",
                        method="label_encode", columns=["SEX"],
                        params={"unknown_strategy": "use_encoded_value", "unknown_value": -1})
t_le = CategoricalTransformer(cfg_le)
X_le = t_le.fit_transform(X_train)

learned_classes = t_le.get_params()['SEX_classes']
print("Learned label encoding for SEX:")
for i, cls in enumerate(learned_classes):
    count = (X_train['SEX'] == cls).sum()
    print(f"  {cls!r:15s} \u2192 {i}  (n={count:,})")

cfg_le = TransformConfig(name="label_sex", transformer_type="categorical", method="label_encode", columns=["SEX"], params={"unknown_strategy": "use_encoded_value", "unknown_value": -1}) t_le = CategoricalTransformer(cfg_le) X_le = t_le.fit_transform(X_train) learned_classes = t_le.get_params()['SEX_classes'] print("Learned label encoding for SEX:") for i, cls in enumerate(learned_classes): count = (X_train['SEX'] == cls).sum() print(f" {cls!r:15s} \u2192 {i} (n={count:,})")

Learned label encoding for SEX:
  'Female'        → 0  (n=14,514)
  'Male'          → 1  (n=9,486)

8b. One-Hot Encoding: onehot_encode¶

In [15]:

cfg_ohe = TransformConfig(name="ohe_education", transformer_type="categorical",
                         method="onehot_encode", columns=["EDUCATION"],
                         params={"sparse": False, "drop": None})
t_ohe = CategoricalTransformer(cfg_ohe)
X_ohe = t_ohe.fit_transform(X_train)

print(f"Columns before OHE: {X_train.shape[1]}")
print(f"Columns after  OHE: {X_ohe.shape[1]}")
new_cols = [c for c in X_ohe.columns if c.startswith('EDUCATION')]
print(f"\nNew columns: {new_cols}")
for col in new_cols:
    print(f"  {col:30s}  mean={X_ohe[col].mean():.3f}")

cfg_ohe = TransformConfig(name="ohe_education", transformer_type="categorical", method="onehot_encode", columns=["EDUCATION"], params={"sparse": False, "drop": None}) t_ohe = CategoricalTransformer(cfg_ohe) X_ohe = t_ohe.fit_transform(X_train) print(f"Columns before OHE: {X_train.shape[1]}") print(f"Columns after OHE: {X_ohe.shape[1]}") new_cols = [c for c in X_ohe.columns if c.startswith('EDUCATION')] print(f"\nNew columns: {new_cols}") for col in new_cols: print(f" {col:30s} mean={X_ohe[col].mean():.3f}")

Columns before OHE: 23
Columns after  OHE: 26

New columns: ['EDUCATION_Graduate', 'EDUCATION_High_School', 'EDUCATION_Other', 'EDUCATION_University']
  EDUCATION_Graduate              mean=0.352
  EDUCATION_High_School           mean=0.163
  EDUCATION_Other                 mean=0.016
  EDUCATION_University            mean=0.469

8c. Frequency Encoding: frequency_encode¶

In [16]:

cfg_fe = TransformConfig(name="freq_marriage", transformer_type="categorical",
                        method="frequency_encode", columns=["MARRIAGE"],
                        params={"normalize": True})
t_fe = CategoricalTransformer(cfg_fe)
X_fe = t_fe.fit_transform(X_train)
freq_map = t_fe.get_params()['MARRIAGE_freq_map']

print("Frequency map learned from training data:")
for cat, freq in sorted(freq_map.items(), key=lambda x: -x[1]):
    print(f"  {cat:10s} \u2192 {freq:.4f}  ({freq*100:.1f}% of training rows)")

cfg_fe = TransformConfig(name="freq_marriage", transformer_type="categorical", method="frequency_encode", columns=["MARRIAGE"], params={"normalize": True}) t_fe = CategoricalTransformer(cfg_fe) X_fe = t_fe.fit_transform(X_train) freq_map = t_fe.get_params()['MARRIAGE_freq_map'] print("Frequency map learned from training data:") for cat, freq in sorted(freq_map.items(), key=lambda x: -x[1]): print(f" {cat:10s} \u2192 {freq:.4f} ({freq*100:.1f}% of training rows)")

Frequency map learned from training data:
  Single     → 0.5336  (53.4% of training rows)
  Married    → 0.4538  (45.4% of training rows)
  Other      → 0.0126  (1.3% of training rows)

8d. Target Encoding: target_encode¶

Target encoding is supervised: the encoder learns category-to-target statistics from the training target. In BitBullet, pass the target through fit(..., y=...) or pipeline.fit_transform(..., y=...) rather than storing y inside transformer params. This keeps the config declarative and the fitted state explicit.

The upgraded SDK target encoder is leakage-aware. During fit_transform it returns out-of-fold (OOF) encodings for the training rows, so a row never sees its own target. During later transform calls it uses the full training-data smoothed map, which is the state saved with the pipeline for inference. It supports regression targets, binary classification probabilities, and multiclass one-vs-rest probability columns.

In [ ]:

target_pipeline = TransformPipeline(name="target_encoding_example")
target_pipeline.add(
    "categorical",
    "target_encode",
    columns=["EDUCATION"],
    params={
        "target_type": "classification",   # binary target -> P(positive_label | category)
        "positive_label": 1,
        "cv_folds": 5,
        "cv_strategy": "stratified",
        "smoothing": 10.0,
        "random_state": 42,
    },
)

# Training path: out-of-fold encodings, so each row is encoded by folds that did not contain it.
X_train_te = target_pipeline.fit_transform(X_train[["EDUCATION"]], y=y_train)

# Inference path: the saved full-training-data map is applied without refitting.
X_test_te = target_pipeline.transform(X_test[["EDUCATION"]])

target_step = target_pipeline.transformers[0]
description = target_step.describe()

audit = pd.DataFrame({
    "education": X_train["EDUCATION"].to_numpy(),
    "target": y_train.to_numpy(),
    "oof_encoded": X_train_te["EDUCATION"].to_numpy(),
})

summary = (
    audit.groupby("education")
    .agg(
        n=("target", "size"),
        raw_default_rate=("target", "mean"),
        mean_oof_encoding=("oof_encoded", "mean"),
    )
    .sort_values("mean_oof_encoding", ascending=False)
)

print("Target encoder audit:")
print(f"  mode:       {description['mode']}")
print(f"  fit_scope:  {description['fit_scope']}")
print(f"  output:     {description['output_columns_by_input']}")
print(f"  test dtype:  {X_test_te['EDUCATION'].dtype}")
display(summary)

print("\nMulticlass target encoding expands one source column into one probability channel per class:")
toy_X = pd.DataFrame({
    "merchant_segment": ["grocery", "grocery", "fuel", "fuel", "travel", "travel", "grocery", "fuel", "travel", "other", "other", "other"],
})
toy_y = pd.Series(["approve", "review", "approve", "reject", "review", "reject", "approve", "review", "reject", "approve", "review", "reject"], name="decision")

multi_pipeline = TransformPipeline(name="multiclass_target_encoding_example")
multi_pipeline.add(
    "categorical",
    "target_encode",
    columns=["merchant_segment"],
    params={"target_type": "classification", "cv_folds": 3, "cv_strategy": "stratified", "smoothing": 2.0},
)
toy_encoded = multi_pipeline.fit_transform(toy_X, y=toy_y)
print([col for col in toy_encoded.columns if col.startswith("merchant_segment")])

target_pipeline = TransformPipeline(name="target_encoding_example") target_pipeline.add( "categorical", "target_encode", columns=["EDUCATION"], params={ "target_type": "classification", # binary target -> P(positive_label | category) "positive_label": 1, "cv_folds": 5, "cv_strategy": "stratified", "smoothing": 10.0, "random_state": 42, }, ) # Training path: out-of-fold encodings, so each row is encoded by folds that did not contain it. X_train_te = target_pipeline.fit_transform(X_train[["EDUCATION"]], y=y_train) # Inference path: the saved full-training-data map is applied without refitting. X_test_te = target_pipeline.transform(X_test[["EDUCATION"]]) target_step = target_pipeline.transformers[0] description = target_step.describe() audit = pd.DataFrame({ "education": X_train["EDUCATION"].to_numpy(), "target": y_train.to_numpy(), "oof_encoded": X_train_te["EDUCATION"].to_numpy(), }) summary = ( audit.groupby("education") .agg( n=("target", "size"), raw_default_rate=("target", "mean"), mean_oof_encoding=("oof_encoded", "mean"), ) .sort_values("mean_oof_encoding", ascending=False) ) print("Target encoder audit:") print(f" mode: {description['mode']}") print(f" fit_scope: {description['fit_scope']}") print(f" output: {description['output_columns_by_input']}") print(f" test dtype: {X_test_te['EDUCATION'].dtype}") display(summary) print("\nMulticlass target encoding expands one source column into one probability channel per class:") toy_X = pd.DataFrame({ "merchant_segment": ["grocery", "grocery", "fuel", "fuel", "travel", "travel", "grocery", "fuel", "travel", "other", "other", "other"], }) toy_y = pd.Series(["approve", "review", "approve", "reject", "review", "reject", "approve", "review", "reject", "approve", "review", "reject"], name="decision") multi_pipeline = TransformPipeline(name="multiclass_target_encoding_example") multi_pipeline.add( "categorical", "target_encode", columns=["merchant_segment"], params={"target_type": "classification", "cv_folds": 3, "cv_strategy": "stratified", "smoothing": 2.0}, ) toy_encoded = multi_pipeline.fit_transform(toy_X, y=toy_y) print([col for col in toy_encoded.columns if col.startswith("merchant_segment")])

8e. Rare Category Binning: bin_rare¶

In [18]:

print("EDUCATION distribution:")
edu_dist = X_train['EDUCATION'].value_counts(normalize=True)
for cat, freq in edu_dist.items():
    print(f"  {cat:15s}: {freq:.3f} ({freq*100:.1f}%)")

print()
for threshold in [0.01, 0.10, 0.20]:
    cfg_br = TransformConfig(name=f"bin_rare_{threshold}", transformer_type="categorical",
                             method="bin_rare", columns=["EDUCATION"],
                             params={"threshold": threshold, "other_label": "other"})
    t_br = CategoricalTransformer(cfg_br)
    X_br = t_br.fit_transform(X_train)
    rare_cats = t_br.get_params()['EDUCATION_rare_categories']
    print(f"  threshold={threshold:.2f}: {len(rare_cats)} merged \u2192 {X_br['EDUCATION'].nunique()} unique remain. Merged: {rare_cats}")

print("EDUCATION distribution:") edu_dist = X_train['EDUCATION'].value_counts(normalize=True) for cat, freq in edu_dist.items(): print(f" {cat:15s}: {freq:.3f} ({freq*100:.1f}%)") print() for threshold in [0.01, 0.10, 0.20]: cfg_br = TransformConfig(name=f"bin_rare_{threshold}", transformer_type="categorical", method="bin_rare", columns=["EDUCATION"], params={"threshold": threshold, "other_label": "other"}) t_br = CategoricalTransformer(cfg_br) X_br = t_br.fit_transform(X_train) rare_cats = t_br.get_params()['EDUCATION_rare_categories'] print(f" threshold={threshold:.2f}: {len(rare_cats)} merged \u2192 {X_br['EDUCATION'].nunique()} unique remain. Merged: {rare_cats}")

EDUCATION distribution:
  University     : 0.469 (46.9%)
  Graduate       : 0.352 (35.2%)
  High_School    : 0.163 (16.3%)
  Other          : 0.016 (1.6%)

  threshold=0.01: 0 merged → 4 unique remain. Merged: []
  threshold=0.10: 1 merged → 4 unique remain. Merged: ['Other']
  threshold=0.20: 2 merged → 3 unique remain. Merged: ['High_School', 'Other']

8f. Feature Hashing: hash_encode¶

In [19]:

print("MARRIAGE unique values:", X_train['MARRIAGE'].unique())
for n_features in [4, 8, 16]:
    cfg_hash = TransformConfig(name=f"hash_{n_features}", transformer_type="categorical",
                               method="hash_encode", columns=["MARRIAGE"],
                               params={"n_features": n_features})
    t_hash = CategoricalTransformer(cfg_hash)
    t_hash.fit(X_train)
    unique_cats = X_train['MARRIAGE'].unique()
    hash_mapping = {cat: hash(str(cat)) % n_features for cat in unique_cats}
    collisions = len(unique_cats) - len(set(hash_mapping.values()))
    print(f"  n_features={n_features:3d}: {collisions} collision(s) among {len(unique_cats)} categories: {hash_mapping}")

print("MARRIAGE unique values:", X_train['MARRIAGE'].unique()) for n_features in [4, 8, 16]: cfg_hash = TransformConfig(name=f"hash_{n_features}", transformer_type="categorical", method="hash_encode", columns=["MARRIAGE"], params={"n_features": n_features}) t_hash = CategoricalTransformer(cfg_hash) t_hash.fit(X_train) unique_cats = X_train['MARRIAGE'].unique() hash_mapping = {cat: hash(str(cat)) % n_features for cat in unique_cats} collisions = len(unique_cats) - len(set(hash_mapping.values())) print(f" n_features={n_features:3d}: {collisions} collision(s) among {len(unique_cats)} categories: {hash_mapping}")

MARRIAGE unique values: ['Single' 'Married' 'Other']
  n_features=  4: 0 collision(s) among 3 categories: {'Single': 2, 'Married': 3, 'Other': 0}
  n_features=  8: 0 collision(s) among 3 categories: {'Single': 2, 'Married': 7, 'Other': 0}
  n_features= 16: 0 collision(s) among 3 categories: {'Single': 2, 'Married': 7, 'Other': 0}

---

9. Composing a Production Pipeline¶

We now build a complete TransformPipeline for the credit card default dataset, guided by the EDA in Section 4:

Feature Group	Transform	Reasoning
PAY_AMT1–PAY_AMT6	log1p	Extreme right skew, non-negative, many zeros
BILL_AMT1–BILL_AMT6	yeo_johnson	Right skew, possible negative values (credit balance)
LIMIT_BAL	log1p	Strictly positive, right-skewed credit limit
AGE	standard_scale	Roughly normal, no outlier concern
PAY_0–PAY_6	robust_scale	Ordinal integers, some extreme delay values (8, 9)
SEX, MARRIAGE	label_encode	Low cardinality (2–3 values), tree models
EDUCATION	onehot_encode	4 categories, slightly higher cardinality

In [20]:

pay_amt_cols  = [f'PAY_AMT{i}' for i in range(1, 7)]
bill_amt_cols = [f'BILL_AMT{i}' for i in range(1, 7)]
pay_cols      = ['PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6']
binary_cats   = ['SEX', 'MARRIAGE']
ohe_cats      = ['EDUCATION']

default_pipeline = (
    TransformPipeline(name="credit_default_pipeline")

    # ── Numerical transforms ────────────────────────────────────────────────
    .add("numerical", "log1p",
         columns=pay_amt_cols + ['LIMIT_BAL'],
         name="log1p_payment_amounts")

    .add("numerical", "yeo_johnson",
         columns=bill_amt_cols,
         name="yj_bill_amounts")

    .add("numerical", "standard_scale",
         columns=["AGE"],
         name="standard_scale_age")

    .add("numerical", "robust_scale",
         columns=pay_cols,
         name="robust_scale_pay_status")

    # ── Categorical transforms ──────────────────────────────────────────────
    .add("categorical", "label_encode",
         columns=binary_cats,
         params={"unknown_strategy": "use_encoded_value", "unknown_value": -1},
         name="label_encode_binary_cats")

    .add("categorical", "onehot_encode",
         columns=ohe_cats,
         params={"sparse": False, "drop": None},
         name="onehot_education")
)

print(default_pipeline)
print(f"\nTotal steps: {len(default_pipeline)}")

pay_amt_cols = [f'PAY_AMT{i}' for i in range(1, 7)] bill_amt_cols = [f'BILL_AMT{i}' for i in range(1, 7)] pay_cols = ['PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6'] binary_cats = ['SEX', 'MARRIAGE'] ohe_cats = ['EDUCATION'] default_pipeline = ( TransformPipeline(name="credit_default_pipeline") # ── Numerical transforms ──────────────────────────────────────────────── .add("numerical", "log1p", columns=pay_amt_cols + ['LIMIT_BAL'], name="log1p_payment_amounts") .add("numerical", "yeo_johnson", columns=bill_amt_cols, name="yj_bill_amounts") .add("numerical", "standard_scale", columns=["AGE"], name="standard_scale_age") .add("numerical", "robust_scale", columns=pay_cols, name="robust_scale_pay_status") # ── Categorical transforms ────────────────────────────────────────────── .add("categorical", "label_encode", columns=binary_cats, params={"unknown_strategy": "use_encoded_value", "unknown_value": -1}, name="label_encode_binary_cats") .add("categorical", "onehot_encode", columns=ohe_cats, params={"sparse": False, "drop": None}, name="onehot_education") ) print(default_pipeline) print(f"\nTotal steps: {len(default_pipeline)}")

TransformPipeline(name='credit_default_pipeline', steps=6, status=not fitted)

Total steps: 6

---

10. fit_transform on Training Data¶

In [21]:

X_train_t = default_pipeline.fit_transform(X_train, verbose=True)

print()
print(f"Input  shape  : {X_train.shape}")
print(f"Output shape  : {X_train_t.shape}")
print(f"Pipeline fitted: {default_pipeline.is_fitted}")
print()
print("Columns AFTER transformation:")
print(list(X_train_t.columns))

X_train_t = default_pipeline.fit_transform(X_train, verbose=True) print() print(f"Input shape : {X_train.shape}") print(f"Output shape : {X_train_t.shape}") print(f"Pipeline fitted: {default_pipeline.is_fitted}") print() print("Columns AFTER transformation:") print(list(X_train_t.columns))

Fitting [1/6]: log1p_payment_amounts
Fitting [2/6]: yj_bill_amounts
Fitting [3/6]: standard_scale_age
Fitting [4/6]: robust_scale_pay_status
Fitting [5/6]: label_encode_binary_cats
Fitting [6/6]: onehot_education
Transforming [1/6]: log1p_payment_amounts
Transforming [2/6]: yj_bill_amounts
Transforming [3/6]: standard_scale_age
Transforming [4/6]: robust_scale_pay_status
Transforming [5/6]: label_encode_binary_cats
Transforming [6/6]: onehot_education

Input  shape  : (24000, 23)
Output shape  : (24000, 26)
Pipeline fitted: True

Columns AFTER transformation:
['LIMIT_BAL', 'SEX', 'MARRIAGE', 'AGE', 'PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6', 'BILL_AMT1', 'BILL_AMT2', 'BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6', 'PAY_AMT1', 'PAY_AMT2', 'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6', 'EDUCATION_Graduate', 'EDUCATION_High_School', 'EDUCATION_Other', 'EDUCATION_University']

In [22]:

print("Spot check: PAY_AMT1 (log1p)")
print(f"  Before: min={X_train['PAY_AMT1'].min():.0f}, max={X_train['PAY_AMT1'].max():.0f}, skew={X_train['PAY_AMT1'].skew():.3f}")
print(f"  After : min={X_train_t['PAY_AMT1'].min():.3f}, max={X_train_t['PAY_AMT1'].max():.3f}, skew={X_train_t['PAY_AMT1'].skew():.3f}")
print()
print("Spot check: BILL_AMT1 (yeo_johnson)")
print(f"  Before: skew={X_train['BILL_AMT1'].skew():.3f}")
print(f"  After : skew={X_train_t['BILL_AMT1'].skew():.3f}")
print()
print("Spot check: EDUCATION one-hot columns")
edu_cols = [c for c in X_train_t.columns if c.startswith('EDUCATION')]
print(f"  Columns: {edu_cols}")
print(f"  Row sums: {X_train_t[edu_cols].sum(axis=1).unique()}")

print("Spot check: PAY_AMT1 (log1p)") print(f" Before: min={X_train['PAY_AMT1'].min():.0f}, max={X_train['PAY_AMT1'].max():.0f}, skew={X_train['PAY_AMT1'].skew():.3f}") print(f" After : min={X_train_t['PAY_AMT1'].min():.3f}, max={X_train_t['PAY_AMT1'].max():.3f}, skew={X_train_t['PAY_AMT1'].skew():.3f}") print() print("Spot check: BILL_AMT1 (yeo_johnson)") print(f" Before: skew={X_train['BILL_AMT1'].skew():.3f}") print(f" After : skew={X_train_t['BILL_AMT1'].skew():.3f}") print() print("Spot check: EDUCATION one-hot columns") edu_cols = [c for c in X_train_t.columns if c.startswith('EDUCATION')] print(f" Columns: {edu_cols}") print(f" Row sums: {X_train_t[edu_cols].sum(axis=1).unique()}")

Spot check: PAY_AMT1 (log1p)
  Before: min=0, max=873552, skew=15.312
  After : min=0.000, max=13.680, skew=-1.299

Spot check: BILL_AMT1 (yeo_johnson)
  Before: skew=2.691
  After : skew=-2.029

Spot check: EDUCATION one-hot columns
  Columns: ['EDUCATION_Graduate', 'EDUCATION_High_School', 'EDUCATION_Other', 'EDUCATION_University']
  Row sums: [1.]

---

11. Save, Load, and Transform the Test Set¶

In [23]:

pipeline_path = "credit_default_pipeline.pkl"

default_pipeline.save(pipeline_path)
file_size_kb = os.path.getsize(pipeline_path) / 1024
print(f"Pipeline saved \u2192 {pipeline_path}  ({file_size_kb:.1f} KB)")

loaded_pipeline = TransformPipeline.load(pipeline_path)
print(f"Pipeline loaded: {len(loaded_pipeline)} steps  |  fitted: {loaded_pipeline.is_fitted}")

X_test_t = loaded_pipeline.transform(X_test)

print(f"\nTest set: {X_test.shape} \u2192 {X_test_t.shape}")
assert list(X_train_t.columns) == list(X_test_t.columns), "Column mismatch!"
print("Column alignment: PASSED")

pipeline_path = "credit_default_pipeline.pkl" default_pipeline.save(pipeline_path) file_size_kb = os.path.getsize(pipeline_path) / 1024 print(f"Pipeline saved \u2192 {pipeline_path} ({file_size_kb:.1f} KB)") loaded_pipeline = TransformPipeline.load(pipeline_path) print(f"Pipeline loaded: {len(loaded_pipeline)} steps | fitted: {loaded_pipeline.is_fitted}") X_test_t = loaded_pipeline.transform(X_test) print(f"\nTest set: {X_test.shape} \u2192 {X_test_t.shape}") assert list(X_train_t.columns) == list(X_test_t.columns), "Column mismatch!" print("Column alignment: PASSED")

Pipeline saved → credit_default_pipeline.pkl  (2.3 KB)
Pipeline loaded: 6 steps  |  fitted: True

Test set: (6000, 23) → (6000, 26)
Column alignment: PASSED

In [24]:

X_train_t_verify = loaded_pipeline.transform(X_train)
is_identical = X_train_t.round(10).equals(X_train_t_verify.round(10))
print(f"Reproducibility check: loaded pipeline produces {'IDENTICAL' if is_identical else 'DIFFERENT'} results")

print()
print("Distribution comparison — train vs test (post-transform):")
print(f"{'Column':15s} {'train mean':>12s} {'test mean':>12s} {'train std':>12s} {'test std':>12s}")
print('-' * 60)
for col in ['PAY_AMT1', 'BILL_AMT1', 'LIMIT_BAL', 'AGE', 'PAY_0']:
    print(f"{col:15s} {X_train_t[col].mean():12.4f} {X_test_t[col].mean():12.4f} "
          f"{X_train_t[col].std():12.4f} {X_test_t[col].std():12.4f}")

X_train_t_verify = loaded_pipeline.transform(X_train) is_identical = X_train_t.round(10).equals(X_train_t_verify.round(10)) print(f"Reproducibility check: loaded pipeline produces {'IDENTICAL' if is_identical else 'DIFFERENT'} results") print() print("Distribution comparison — train vs test (post-transform):") print(f"{'Column':15s} {'train mean':>12s} {'test mean':>12s} {'train std':>12s} {'test std':>12s}") print('-' * 60) for col in ['PAY_AMT1', 'BILL_AMT1', 'LIMIT_BAL', 'AGE', 'PAY_0']: print(f"{col:15s} {X_train_t[col].mean():12.4f} {X_test_t[col].mean():12.4f} " f"{X_train_t[col].std():12.4f} {X_test_t[col].std():12.4f}")

Reproducibility check: loaded pipeline produces IDENTICAL results

Distribution comparison — train vs test (post-transform):
Column            train mean    test mean    train std     test std
------------------------------------------------------------
PAY_AMT1              6.6368       6.6027       3.2440       3.2755
BILL_AMT1         13533.5819   13716.1056   18330.9054   17488.7793
LIMIT_BAL            11.6633      11.6617       0.9400       0.9456
AGE                  -0.0000       0.0288       1.0000       1.0120
PAY_0                -0.0141      -0.0270       1.1232       1.1264

---

12. Pipeline Management: list_steps, disable_step, enable_step¶

In [25]:

steps = default_pipeline.list_steps()
print(f"{'#':>3}  {'Name':40s} {'Type':12s} {'Method':20s} {'Enabled':8s}")
print('-' * 90)
for i, step in enumerate(steps, 1):
    print(f"{i:>3}  {step['name']:40s} {step['type']:12s} {step['method']:20s} "
          f"{'Yes' if step['enabled'] else 'No':8s}")
    if step['columns']:
        print(f"     columns: {step['columns']}")

steps = default_pipeline.list_steps() print(f"{'#':>3} {'Name':40s} {'Type':12s} {'Method':20s} {'Enabled':8s}") print('-' * 90) for i, step in enumerate(steps, 1): print(f"{i:>3} {step['name']:40s} {step['type']:12s} {step['method']:20s} " f"{'Yes' if step['enabled'] else 'No':8s}") if step['columns']: print(f" columns: {step['columns']}")

  #  Name                                     Type         Method               Enabled 
------------------------------------------------------------------------------------------
  1  log1p_payment_amounts                    numerical    log1p                Yes     
     columns: ['PAY_AMT1', 'PAY_AMT2', 'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6', 'LIMIT_BAL']
  2  yj_bill_amounts                          numerical    yeo_johnson          Yes     
     columns: ['BILL_AMT1', 'BILL_AMT2', 'BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6']
  3  standard_scale_age                       numerical    standard_scale       Yes     
     columns: ['AGE']
  4  robust_scale_pay_status                  numerical    robust_scale         Yes     
     columns: ['PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6']
  5  label_encode_binary_cats                 categorical  label_encode         Yes     
     columns: ['SEX', 'MARRIAGE']
  6  onehot_education                         categorical  onehot_encode        Yes     
     columns: ['EDUCATION']

In [26]:

X_with_log   = default_pipeline.transform(X_train)
skew_with    = X_with_log['PAY_AMT1'].skew()
print(f"With log1p enabled  : PAY_AMT1 skew = {skew_with:.4f}")

default_pipeline.disable_step("log1p_payment_amounts")
X_without_log = default_pipeline.transform(X_train)
skew_without  = X_without_log['PAY_AMT1'].skew()
print(f"With log1p disabled : PAY_AMT1 skew = {skew_without:.4f}  (raw values pass through)")

default_pipeline.enable_step("log1p_payment_amounts")
X_re_enabled  = default_pipeline.transform(X_train)
print(f"After re-enabling   : PAY_AMT1 skew = {X_re_enabled['PAY_AMT1'].skew():.4f}")

X_with_log = default_pipeline.transform(X_train) skew_with = X_with_log['PAY_AMT1'].skew() print(f"With log1p enabled : PAY_AMT1 skew = {skew_with:.4f}") default_pipeline.disable_step("log1p_payment_amounts") X_without_log = default_pipeline.transform(X_train) skew_without = X_without_log['PAY_AMT1'].skew() print(f"With log1p disabled : PAY_AMT1 skew = {skew_without:.4f} (raw values pass through)") default_pipeline.enable_step("log1p_payment_amounts") X_re_enabled = default_pipeline.transform(X_train) print(f"After re-enabling : PAY_AMT1 skew = {X_re_enabled['PAY_AMT1'].skew():.4f}")

With log1p enabled  : PAY_AMT1 skew = -1.2993
With log1p disabled : PAY_AMT1 skew = 15.3121  (raw values pass through)
After re-enabling   : PAY_AMT1 skew = -1.2993

---

13. Visualising Transformations¶

In [27]:

# Interactive widget — use the dropdown to explore each transformed feature
plot_pipeline_transformations(default_pipeline, X_train, X_train_t)

# Interactive widget — use the dropdown to explore each transformed feature plot_pipeline_transformations(default_pipeline, X_train, X_train_t)

interactive(children=(Dropdown(description='feature', options=('AGE', 'BILL_AMT1', 'BILL_AMT2', 'BILL_AMT3', '…

In [28]:

fig, axes = plt.subplots(4, 2, figsize=(14, 16))
fig.suptitle("Numerical Transforms — Before vs After (Training Data)", fontsize=14, fontweight='bold')

num_pairs = [
    ('PAY_AMT1',  'log1p',       BLUE,   GREEN),
    ('BILL_AMT1', 'yeo_johnson', BLUE,   ORANGE),
    ('LIMIT_BAL', 'log1p',       BLUE,   GREEN),
    ('AGE',       'standard_scale', BLUE, ORANGE),
]

for i, (col, method, c_before, c_after) in enumerate(num_pairs):
    before = X_train[col]
    after  = X_train_t[col]
    axes[i, 0].hist(before.dropna(), bins=40, color=c_before, alpha=0.75)
    axes[i, 0].set_title(f"`{col}` — Original  [skew={before.skew():.2f}]", fontweight='bold', fontsize=10)
    axes[i, 1].hist(after.dropna(), bins=40, color=c_after, alpha=0.75)
    axes[i, 1].set_title(f"`{col}` — After `{method}`  [skew={after.skew():.2f}]", fontweight='bold', fontsize=10)

plt.tight_layout()
plt.show()

fig, axes = plt.subplots(4, 2, figsize=(14, 16)) fig.suptitle("Numerical Transforms — Before vs After (Training Data)", fontsize=14, fontweight='bold') num_pairs = [ ('PAY_AMT1', 'log1p', BLUE, GREEN), ('BILL_AMT1', 'yeo_johnson', BLUE, ORANGE), ('LIMIT_BAL', 'log1p', BLUE, GREEN), ('AGE', 'standard_scale', BLUE, ORANGE), ] for i, (col, method, c_before, c_after) in enumerate(num_pairs): before = X_train[col] after = X_train_t[col] axes[i, 0].hist(before.dropna(), bins=40, color=c_before, alpha=0.75) axes[i, 0].set_title(f"`{col}` — Original [skew={before.skew():.2f}]", fontweight='bold', fontsize=10) axes[i, 1].hist(after.dropna(), bins=40, color=c_after, alpha=0.75) axes[i, 1].set_title(f"`{col}` — After `{method}` [skew={after.skew():.2f}]", fontweight='bold', fontsize=10) plt.tight_layout() plt.show()

---

14. Pipeline Metadata Inspection¶

In [29]:

meta = default_pipeline.metadata

print("Pipeline Metadata")
print("=" * 50)
print(f"  Pipeline name    : {default_pipeline.name}")
print(f"  Created at       : {meta.created_at}")
print(f"  Fitted at        : {meta.fitted_at}")
print(f"  Number of steps  : {meta.n_transformers}")
print(f"  Input shape      : {meta.input_shape}")
print(f"  Output shape     : {meta.output_shape}")
print(f"  Is fitted        : {default_pipeline.is_fitted}")
print(f"  Column expansion : +{meta.output_shape[1] - meta.input_shape[1]} columns (from one-hot encoding)")

meta = default_pipeline.metadata print("Pipeline Metadata") print("=" * 50) print(f" Pipeline name : {default_pipeline.name}") print(f" Created at : {meta.created_at}") print(f" Fitted at : {meta.fitted_at}") print(f" Number of steps : {meta.n_transformers}") print(f" Input shape : {meta.input_shape}") print(f" Output shape : {meta.output_shape}") print(f" Is fitted : {default_pipeline.is_fitted}") print(f" Column expansion : +{meta.output_shape[1] - meta.input_shape[1]} columns (from one-hot encoding)")

Pipeline Metadata
==================================================
  Pipeline name    : credit_default_pipeline
  Created at       : None
  Fitted at        : 2026-05-06T14:27:18.945569
  Number of steps  : 6
  Input shape      : (24000, 23)
  Output shape     : (24000, 26)
  Is fitted        : True
  Column expansion : +3 columns (from one-hot encoding)

---

15. Conclusion¶

This notebook covered bitbullet.transform applied to a real-world credit default dataset with a genuine mix of transformation needs.

What We Built¶

You Wrote	BitBullet Handled
generate_feature_stats(df)	Full audit — skewness, kurtosis, missing values, cardinality
pipeline.add("numerical", "log1p", ...)	Fit on training data, store parameters, apply consistently
pipeline.add("numerical", "yeo_johnson", ...)	Power transformer for negative-valued BILL_AMT columns
pipeline.fit_transform(X_train, y=y_train) when supervised encoders are present	Sequential chaining, target-aware fitting, fit-lock after completion
pipeline.save(path)	Compressed serialisation of all fitted parameters
TransformPipeline.load(path)	Exact reconstruction — identical parameters, identical output
loaded_pipeline.transform(X_test)	Apply exact training parameters — zero leakage

Quick Decision Guide¶

PAY_AMT columns (zeros, extreme right skew)  → log1p
BILL_AMT columns (right skew, possible negatives) → yeo_johnson
LIMIT_BAL (strictly positive, right-skewed)  → log1p
AGE (roughly normal)                         → standard_scale
PAY_0–PAY_6 (ordinal integer, some extremes) → robust_scale
SEX, MARRIAGE (2–3 categories, tree model)   → label_encode
EDUCATION (4 categories, nominal)            → onehot_encode

Continue the Academy:

• 05_Model_Management.ipynb — versioning, comparison, and model governance