BitBullet:Lessons — 04: Data Transformations¶

A Deep Dive into bitbullet.transform¶

---

Raw data does not feed models cleanly. A feature with extreme right skew pushes tree splits to the extremes. An unscaled column ranging [0, 1,000,000] next to one ranging [0, 1] biases distance-based algorithms. A categorical column with 50 unique values explodes into 50 one-hot columns unless you have a smarter strategy.

bitbullet.transform is built around one critical constraint: transformers must be fit only on training data and applied identically to test data. Violating this rule is one of the most common sources of optimistic, non-reproducible models in the industry.

Dataset: default_of_credit_card_clients.xls — 30,000 Taiwanese credit card clients, 23 features after cleaning, binary target: default_payment.
Source: UCI ML Repository — Default of Credit Card Clients
Context: Predict whether a client will default on next month's payment based on credit limit, repayment history, bill statements, and demographics.

---

Notebook Roadmap¶

Section	Topic
2	Setup and imports
3	Load and clean data
4	EDA with generate_feature_stats
5	The pipeline mental model
6	Train/test split
7a–7e	Numerical transforms: log/power, scaling, quantile, winsorize, binning
8a–8f	Categorical transforms: label, one-hot, frequency, target, rare-bin, hash
9	Composing a production pipeline
10	fit_transform on train
11	Save, reload, transform test set
12	Pipeline management: list, disable, enable
13	Visualising transformations
14	Pipeline metadata inspection
15	Conclusion

2. Setup and Imports¶

In [1]:

import sys
import os
import warnings
warnings.filterwarnings('ignore')

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split

# When running this notebook from bitbullet/lessons in a local clone, prefer the local SDK source.
sdk_root = os.path.abspath("..")
if sdk_root not in sys.path:
    sys.path.insert(0, sdk_root)

from bitbullet.transform import (
    TransformPipeline,
    BaseTransformer,
    TransformConfig,
    NumericalTransformer,
    CategoricalTransformer,
    DateTimeTransformer,
    generate_feature_stats,
)
from bitbullet.transform.eda import plot_pipeline_transformations

import bitbullet
print(f"bitbullet version : {getattr(bitbullet, '__version__', 'dev')}")
print("All imports successful.")

plt.rcParams.update({
    'figure.facecolor': 'white',
    'axes.spines.top': False,
    'axes.spines.right': False,
    'axes.grid': True,
    'grid.alpha': 0.3,
    'font.size': 11,
})
BLUE   = '#2563EB'
ORANGE = '#F59E0B'
GREEN  = '#10B981'
RED    = '#EF4444'
GREY   = '#9CA3AF'

import sys import os import warnings warnings.filterwarnings('ignore') import pandas as pd import numpy as np import matplotlib.pyplot as plt import seaborn as sns from sklearn.model_selection import train_test_split # When running this notebook from bitbullet/lessons in a local clone, prefer the local SDK source. sdk_root = os.path.abspath("..") if sdk_root not in sys.path: sys.path.insert(0, sdk_root) from bitbullet.transform import ( TransformPipeline, BaseTransformer, TransformConfig, NumericalTransformer, CategoricalTransformer, DateTimeTransformer, generate_feature_stats, ) from bitbullet.transform.eda import plot_pipeline_transformations import bitbullet print(f"bitbullet version : {getattr(bitbullet, '__version__', 'dev')}") print("All imports successful.") plt.rcParams.update({ 'figure.facecolor': 'white', 'axes.spines.top': False, 'axes.spines.right': False, 'axes.grid': True, 'grid.alpha': 0.3, 'font.size': 11, }) BLUE = '#2563EB' ORANGE = '#F59E0B' GREEN = '#10B981' RED = '#EF4444' GREY = '#9CA3AF'

bitbullet version : dev
All imports successful.

3. Load and Clean the Dataset¶

Dataset: Default of Credit Card Clients Dataset
Source: UCI ML Repository — Default of Credit Card Clients
File: default_of_credit_card_clients.xls

Before running this cell: download default_of_credit_card_clients.xls from the link above and place it in the same directory as this notebook. If you store it elsewhere, update data_path in the code cell below to match your chosen location.

In [ ]:

# Update this path if you stored the file in a different location.
data_path = "default_of_credit_card_clients.xls"
# The XLS file has a descriptive first row — header is on row 1 (0-indexed)
df_raw = pd.read_excel(data_path, header=1)

print(f"Raw shape: {df_raw.shape}")
print(f"Columns: {list(df_raw.columns)}")

# Update this path if you stored the file in a different location. data_path = "default_of_credit_card_clients.xls" # The XLS file has a descriptive first row — header is on row 1 (0-indexed) df_raw = pd.read_excel(data_path, header=1) print(f"Raw shape: {df_raw.shape}") print(f"Columns: {list(df_raw.columns)}")

Raw shape: (30000, 25)
Columns: ['ID', 'LIMIT_BAL', 'SEX', 'EDUCATION', 'MARRIAGE', 'AGE', 'PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6', 'BILL_AMT1', 'BILL_AMT2', 'BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6', 'PAY_AMT1', 'PAY_AMT2', 'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6', 'default payment next month']

In [5]:

# Drop the ID column — it is a row identifier with no predictive value
df = df_raw.drop(columns=['ID'])

# Rename target to a clean identifier
df = df.rename(columns={'default payment next month': 'default_payment'})

# Map integer-coded categorical columns to descriptive strings
df['SEX'] = df['SEX'].map({1: 'Male', 2: 'Female'})
df['EDUCATION'] = df['EDUCATION'].map({
    1: 'Graduate', 2: 'University', 3: 'High_School', 4: 'Other',
    5: 'Other', 6: 'Other', 0: 'Other'   # values 0, 5, 6 are undocumented — treat as Other
})
df['MARRIAGE'] = df['MARRIAGE'].map({
    1: 'Married', 2: 'Single', 3: 'Other', 0: 'Other'
})

TARGET = 'default_payment'

print(f"Dataset shape (after cleaning): {df.shape}")
print(f"\nTarget distribution:")
counts = df[TARGET].value_counts()
print(f"  No default (0) : {counts[0]:,}  ({counts[0]/len(df)*100:.1f}%)")
print(f"  Default    (1) : {counts[1]:,}  ({counts[1]/len(df)*100:.1f}%)")
print()
display(df.head())

# Drop the ID column — it is a row identifier with no predictive value df = df_raw.drop(columns=['ID']) # Rename target to a clean identifier df = df.rename(columns={'default payment next month': 'default_payment'}) # Map integer-coded categorical columns to descriptive strings df['SEX'] = df['SEX'].map({1: 'Male', 2: 'Female'}) df['EDUCATION'] = df['EDUCATION'].map({ 1: 'Graduate', 2: 'University', 3: 'High_School', 4: 'Other', 5: 'Other', 6: 'Other', 0: 'Other' # values 0, 5, 6 are undocumented — treat as Other }) df['MARRIAGE'] = df['MARRIAGE'].map({ 1: 'Married', 2: 'Single', 3: 'Other', 0: 'Other' }) TARGET = 'default_payment' print(f"Dataset shape (after cleaning): {df.shape}") print(f"\nTarget distribution:") counts = df[TARGET].value_counts() print(f" No default (0) : {counts[0]:,} ({counts[0]/len(df)*100:.1f}%)") print(f" Default (1) : {counts[1]:,} ({counts[1]/len(df)*100:.1f}%)") print() display(df.head())

Dataset shape (after cleaning): (30000, 24)

Target distribution:
  No default (0) : 23,364  (77.9%)
  Default    (1) : 6,636  (22.1%)

	LIMIT_BAL	SEX	EDUCATION	MARRIAGE	AGE	PAY_0	PAY_2	PAY_3	PAY_4	PAY_5	...	BILL_AMT4	BILL_AMT5	BILL_AMT6	PAY_AMT1	PAY_AMT2	PAY_AMT3	PAY_AMT4	PAY_AMT5	PAY_AMT6	default_payment
0	20000	Female	University	Married	24	2	2	-1	-1	-2	...	0	0	0	0	689	0	0	0	0	1
1	120000	Female	University	Single	26	-1	2	0	0	0	...	3272	3455	3261	0	1000	1000	1000	0	2000	1
2	90000	Female	University	Single	34	0	0	0	0	0	...	14331	14948	15549	1518	1500	1000	1000	1000	5000	0
3	50000	Female	University	Married	37	0	0	0	0	0	...	28314	28959	29547	2000	2019	1200	1100	1069	1000	0
4	50000	Male	University	Married	57	-1	0	-1	0	0	...	20940	19146	19131	2000	36681	10000	9000	689	679	0

5 rows × 24 columns

Column Glossary¶

Column	Type	Description
LIMIT_BAL	numerical	Credit limit (NT dollar)
SEX	categorical	Male / Female
EDUCATION	categorical	Graduate / University / High_School / Other
MARRIAGE	categorical	Married / Single / Other
AGE	numerical	Age in years
PAY_0–PAY_6	numerical	Repayment status for Sept–April (-2=no consumption, -1=paid duly, 0=revolving credit, 1–9=months delayed)
BILL_AMT1–BILL_AMT6	numerical	Bill statement amount Sept–April (NT dollar)
PAY_AMT1–PAY_AMT6	numerical	Previous payment amount Sept–April (NT dollar)
default_payment	target	1 = defaulted next month, 0 = did not

4. EDA with generate_feature_stats¶

In [6]:

X_full = df.drop(columns=[TARGET])
y_full = df[TARGET]

stats_df, numerical_cols, categorical_cols = generate_feature_stats(X_full)

print(f"Numerical   ({len(numerical_cols)}): {numerical_cols}")
print(f"Categorical ({len(categorical_cols)}): {categorical_cols}")
print()
display(stats_df)

X_full = df.drop(columns=[TARGET]) y_full = df[TARGET] stats_df, numerical_cols, categorical_cols = generate_feature_stats(X_full) print(f"Numerical ({len(numerical_cols)}): {numerical_cols}") print(f"Categorical ({len(categorical_cols)}): {categorical_cols}") print() display(stats_df)

Numerical   (20): ['LIMIT_BAL', 'AGE', 'PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6', 'BILL_AMT1', 'BILL_AMT2', 'BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6', 'PAY_AMT1', 'PAY_AMT2', 'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6']
Categorical (3): ['SEX', 'EDUCATION', 'MARRIAGE']

	dtype	missing_count	missing_percent	unique_values	zero_count	negative_count	mean	std	skewness	kurtosis	mean_percentile	min	25%	50%	75%	max	top	freq
LIMIT_BAL	int64	0	0.0	81	0	0	167484.322667	129747.661567	0.992867	0.536263	56.98	10000.0	50000.0	140000.0	240000.0	1000000.0	-	-
SEX	object	0	0.0	2	0	0	-	-	-	-	-	-	-	-	-	-	Female	18112
EDUCATION	object	0	0.0	4	0	0	-	-	-	-	-	-	-	-	-	-	University	14030
MARRIAGE	object	0	0.0	3	0	0	-	-	-	-	-	-	-	-	-	-	Single	15964
AGE	int64	0	0.0	56	0	0	35.4855	9.217904	0.732246	0.044303	56.03	21.0	28.0	34.0	41.0	79.0	-	-
PAY_0	int64	0	0.0	11	14737	8445	-0.0167	1.123802	0.731975	2.720715	28.15	-2.0	-1.0	0.0	0.0	8.0	-	-
PAY_2	int64	0	0.0	11	15730	9832	-0.133767	1.197186	0.790565	1.570418	32.773333	-2.0	-1.0	0.0	0.0	8.0	-	-
PAY_3	int64	0	0.0	11	15764	10023	-0.1662	1.196868	0.840682	2.084436	33.41	-2.0	-1.0	0.0	0.0	8.0	-	-
PAY_4	int64	0	0.0	11	16455	10035	-0.220667	1.169139	0.999629	3.496983	33.45	-2.0	-1.0	0.0	0.0	8.0	-	-
PAY_5	int64	0	0.0	10	16947	10085	-0.2662	1.133187	1.008197	3.989748	33.616667	-2.0	-1.0	0.0	0.0	8.0	-	-
PAY_6	int64	0	0.0	10	16286	10635	-0.2911	1.149988	0.948029	3.426534	35.45	-2.0	-1.0	0.0	0.0	8.0	-	-
BILL_AMT1	int64	0	0.0	22723	2008	590	51223.3309	73635.860576	2.663861	9.806289	69.516667	-165580.0	3558.75	22381.5	67091.0	964511.0	-	-
BILL_AMT2	int64	0	0.0	22346	2506	669	49179.075167	71173.768783	2.705221	10.302946	68.426667	-69777.0	2984.75	21200.0	64006.25	983931.0	-	-
BILL_AMT3	int64	0	0.0	22026	2870	655	47013.1548	69349.387427	3.08783	19.783255	68.073333	-157264.0	2666.25	20088.5	60164.75	1664089.0	-	-
BILL_AMT4	int64	0	0.0	21548	3195	675	43262.948967	64332.856134	2.821965	11.309325	68.856667	-170000.0	2326.75	19052.0	54506.0	891586.0	-	-
BILL_AMT5	int64	0	0.0	21010	3506	655	40311.400967	60797.15577	2.87638	12.305881	69.696667	-81334.0	1763.0	18104.5	50190.5	927171.0	-	-
BILL_AMT6	int64	0	0.0	20604	4020	688	38871.7604	59554.107537	2.846645	12.270705	69.846667	-339603.0	1256.0	17071.0	49198.25	961664.0	-	-
PAY_AMT1	int64	0	0.0	7943	5249	0	5663.5805	16563.280354	14.668364	415.254743	77.59	0.0	1000.0	2100.0	5006.0	873552.0	-	-
PAY_AMT2	int64	0	0.0	7899	5396	0	5921.1635	23040.870402	30.453817	1641.631911	79.133333	0.0	833.0	2009.0	5000.0	1684259.0	-	-
PAY_AMT3	int64	0	0.0	7518	5968	0	5225.6815	17606.96147	17.216635	564.311229	79.873333	0.0	390.0	1800.0	4505.0	896040.0	-	-
PAY_AMT4	int64	0	0.0	6937	6408	0	4826.076867	15666.159744	12.904985	277.333768	77.683333	0.0	296.0	1500.0	4013.25	621000.0	-	-
PAY_AMT5	int64	0	0.0	6897	6703	0	4799.387633	15278.305679	11.127417	180.06394	77.5	0.0	252.5	1500.0	4031.5	426529.0	-	-
PAY_AMT6	int64	0	0.0	6939	7173	0	5215.502567	17777.465775	10.640727	167.16143	81.986667	0.0	117.75	1500.0	4000.0	528666.0	-	-

In [7]:

num_stats = stats_df.loc[numerical_cols, ['dtype', 'missing_percent', 'skewness', 'kurtosis',
                                                   'zero_count', 'negative_count', 'min', 'max']]
print("Numerical feature audit:")
display(num_stats)

print()
print("Categorical feature audit:")
cat_stats = stats_df.loc[categorical_cols, ['dtype', 'missing_percent', 'unique_values', 'top', 'freq']]
display(cat_stats)

num_stats = stats_df.loc[numerical_cols, ['dtype', 'missing_percent', 'skewness', 'kurtosis', 'zero_count', 'negative_count', 'min', 'max']] print("Numerical feature audit:") display(num_stats) print() print("Categorical feature audit:") cat_stats = stats_df.loc[categorical_cols, ['dtype', 'missing_percent', 'unique_values', 'top', 'freq']] display(cat_stats)

Numerical feature audit:

	dtype	missing_percent	skewness	kurtosis	zero_count	negative_count	min	max
LIMIT_BAL	int64	0.0	0.992867	0.536263	0	0	10000.0	1000000.0
AGE	int64	0.0	0.732246	0.044303	0	0	21.0	79.0
PAY_0	int64	0.0	0.731975	2.720715	14737	8445	-2.0	8.0
PAY_2	int64	0.0	0.790565	1.570418	15730	9832	-2.0	8.0
PAY_3	int64	0.0	0.840682	2.084436	15764	10023	-2.0	8.0
PAY_4	int64	0.0	0.999629	3.496983	16455	10035	-2.0	8.0
PAY_5	int64	0.0	1.008197	3.989748	16947	10085	-2.0	8.0
PAY_6	int64	0.0	0.948029	3.426534	16286	10635	-2.0	8.0
BILL_AMT1	int64	0.0	2.663861	9.806289	2008	590	-165580.0	964511.0
BILL_AMT2	int64	0.0	2.705221	10.302946	2506	669	-69777.0	983931.0
BILL_AMT3	int64	0.0	3.08783	19.783255	2870	655	-157264.0	1664089.0
BILL_AMT4	int64	0.0	2.821965	11.309325	3195	675	-170000.0	891586.0
BILL_AMT5	int64	0.0	2.87638	12.305881	3506	655	-81334.0	927171.0
BILL_AMT6	int64	0.0	2.846645	12.270705	4020	688	-339603.0	961664.0
PAY_AMT1	int64	0.0	14.668364	415.254743	5249	0	0.0	873552.0
PAY_AMT2	int64	0.0	30.453817	1641.631911	5396	0	0.0	1684259.0
PAY_AMT3	int64	0.0	17.216635	564.311229	5968	0	0.0	896040.0
PAY_AMT4	int64	0.0	12.904985	277.333768	6408	0	0.0	621000.0
PAY_AMT5	int64	0.0	11.127417	180.06394	6703	0	0.0	426529.0
PAY_AMT6	int64	0.0	10.640727	167.16143	7173	0	0.0	528666.0

Categorical feature audit:

	dtype	missing_percent	unique_values	top	freq
SEX	object	0.0	2	Female	18112
EDUCATION	object	0.0	4	University	14030
MARRIAGE	object	0.0	3	Single	15964

Reading the Stats Table — Key Observations for This Dataset¶

• PAY_AMT1–PAY_AMT6: extreme right skew (> 10). Clients who pay large lump sums dominate the tail. log1p is the right choice — safe for zeros, compresses the right tail.
• BILL_AMT1–BILL_AMT6: moderate-to-high right skew, and some negative values (credit balance). yeo_johnson handles negatives that log and log1p cannot.
• LIMIT_BAL: right-skewed credit limits. log1p reduces the long tail.
• AGE: roughly normal — standard_scale is appropriate.
• PAY_0–PAY_6: ordinal integer repayment status (-2 to 9). Treat as numerical; robust_scale is appropriate for this bounded ordinal range.
• SEX, EDUCATION, MARRIAGE: three low-cardinality categoricals, 2–4 unique values each. label_encode works for tree models; onehot_encode for linear models.

5. The Pipeline Mental Model¶

Before writing a single transform, we internalise one rule:

Transformers must be fit exclusively on training data. They must then be applied — without refitting — to validation and test data.

What TransformPipeline Remembers After Fitting¶

Transform	Fitted Parameters Stored
standard_scale	mean, std per column
robust_scale	median, IQR per column
yeo_johnson	PowerTransformer object per column
log1p	Stateless — no fitting required
winsorize	Lower and upper clip thresholds per column
label_encode	LabelEncoder object per column
onehot_encode	OneHotEncoder object per column
target_encode	Leakage-aware target statistics: OOF values from fit_transform(..., y=target) for training, full-data smoothed map for transform(...) at inference

6. Train/Test Split¶

In [8]:

X_train, X_test, y_train, y_test = train_test_split(
    X_full, y_full, test_size=0.20, random_state=42, stratify=y_full
)

print(f"Training set : {X_train.shape[0]:,} samples  (default rate: {y_train.mean():.3f})")
print(f"Test set     : {X_test.shape[0]:,} samples  (default rate: {y_test.mean():.3f})")
print()
print("Test set is sealed. No fitting step will touch it until final evaluation.")

X_train, X_test, y_train, y_test = train_test_split( X_full, y_full, test_size=0.20, random_state=42, stratify=y_full ) print(f"Training set : {X_train.shape[0]:,} samples (default rate: {y_train.mean():.3f})") print(f"Test set : {X_test.shape[0]:,} samples (default rate: {y_test.mean():.3f})") print() print("Test set is sealed. No fitting step will touch it until final evaluation.")

Training set : 24,000 samples  (default rate: 0.221)
Test set     : 6,000 samples  (default rate: 0.221)

Test set is sealed. No fitting step will touch it until final evaluation.

---

7. Numerical Transformations¶

7a. Logarithmic and Power Transforms: log1p, yeo_johnson, sqrt, boxcox¶

Decision rule for this dataset:

• PAY_AMT columns: zeros present, right-skewed → log1p
• BILL_AMT columns: possible negatives, right-skewed → yeo_johnson
• LIMIT_BAL: strictly positive, right-skewed → log1p

In [9]:

fig, axes = plt.subplots(2, 4, figsize=(18, 9))
fig.suptitle("Log / Power Transforms — Before vs After", fontsize=14, fontweight='bold')

transforms_demo = [
    ('PAY_AMT1', 'log1p',      BLUE,   GREEN),
    ('PAY_AMT2', 'log1p',      BLUE,   GREEN),
    ('BILL_AMT1','yeo_johnson', BLUE,   ORANGE),
    ('LIMIT_BAL','log1p',      BLUE,   ORANGE),
]

results_power = {}
for col, method, c1, c2 in transforms_demo:
    cfg = TransformConfig(name=f"{method}_{col}", transformer_type="numerical",
                          method=method, columns=[col], params={})
    t = NumericalTransformer(cfg)
    results_power[col] = t.fit_transform(X_train)

for i, (col, method, c1, c2) in enumerate(transforms_demo):
    before = X_train[col]
    after  = results_power[col][col]

    axes[0, i].hist(before.dropna(), bins=40, color=c1, alpha=0.8)
    axes[0, i].set_title(f"`{col}`\n(before)", fontsize=10, fontweight='bold')
    axes[0, i].text(0.97, 0.95, f'skew={before.skew():.2f}', transform=axes[0, i].transAxes,
                    ha='right', va='top', fontsize=9,
                    bbox=dict(boxstyle='round,pad=0.3', facecolor='white', edgecolor='grey', alpha=0.8))

    axes[1, i].hist(after.dropna(), bins=40, color=c2, alpha=0.8)
    axes[1, i].set_title(f"`{method}`\n(after)", fontsize=10, fontweight='bold')
    axes[1, i].text(0.97, 0.95, f'skew={after.skew():.2f}', transform=axes[1, i].transAxes,
                    ha='right', va='top', fontsize=9,
                    bbox=dict(boxstyle='round,pad=0.3', facecolor='white', edgecolor='grey', alpha=0.8))

plt.tight_layout()
plt.show()

print("Skewness reduction summary:")
for col, method, _, _ in transforms_demo:
    before_skew = X_train[col].skew()
    after_skew  = results_power[col][col].skew()
    print(f"  {col:15s} [{method:12s}]  {before_skew:+.3f} \u2192 {after_skew:+.3f}")

fig, axes = plt.subplots(2, 4, figsize=(18, 9)) fig.suptitle("Log / Power Transforms — Before vs After", fontsize=14, fontweight='bold') transforms_demo = [ ('PAY_AMT1', 'log1p', BLUE, GREEN), ('PAY_AMT2', 'log1p', BLUE, GREEN), ('BILL_AMT1','yeo_johnson', BLUE, ORANGE), ('LIMIT_BAL','log1p', BLUE, ORANGE), ] results_power = {} for col, method, c1, c2 in transforms_demo: cfg = TransformConfig(name=f"{method}_{col}", transformer_type="numerical", method=method, columns=[col], params={}) t = NumericalTransformer(cfg) results_power[col] = t.fit_transform(X_train) for i, (col, method, c1, c2) in enumerate(transforms_demo): before = X_train[col] after = results_power[col][col] axes[0, i].hist(before.dropna(), bins=40, color=c1, alpha=0.8) axes[0, i].set_title(f"`{col}`\n(before)", fontsize=10, fontweight='bold') axes[0, i].text(0.97, 0.95, f'skew={before.skew():.2f}', transform=axes[0, i].transAxes, ha='right', va='top', fontsize=9, bbox=dict(boxstyle='round,pad=0.3', facecolor='white', edgecolor='grey', alpha=0.8)) axes[1, i].hist(after.dropna(), bins=40, color=c2, alpha=0.8) axes[1, i].set_title(f"`{method}`\n(after)", fontsize=10, fontweight='bold') axes[1, i].text(0.97, 0.95, f'skew={after.skew():.2f}', transform=axes[1, i].transAxes, ha='right', va='top', fontsize=9, bbox=dict(boxstyle='round,pad=0.3', facecolor='white', edgecolor='grey', alpha=0.8)) plt.tight_layout() plt.show() print("Skewness reduction summary:") for col, method, _, _ in transforms_demo: before_skew = X_train[col].skew() after_skew = results_power[col][col].skew() print(f" {col:15s} [{method:12s}] {before_skew:+.3f} \u2192 {after_skew:+.3f}")

Skewness reduction summary:
  PAY_AMT1        [log1p       ]  +15.312 → -1.299
  PAY_AMT2        [log1p       ]  +21.124 → -1.241
  BILL_AMT1       [yeo_johnson ]  +2.691 → -2.029
  LIMIT_BAL       [log1p       ]  +1.000 → -0.517

7b. Scaling Transforms: standard_scale, robust_scale¶

In [10]:

fig, axes = plt.subplots(1, 3, figsize=(16, 5))
fig.suptitle("Scaling Transforms on `AGE`", fontsize=13, fontweight='bold')

axes[0].hist(X_train['AGE'], bins=30, color=GREY, alpha=0.8)
axes[0].set_title(f"Original\nrange [{X_train['AGE'].min():.0f}, {X_train['AGE'].max():.0f}]", fontweight='bold')

for i, (method, params, color) in enumerate([
    ('standard_scale', {}, BLUE),
    ('robust_scale',   {}, ORANGE),
], start=1):
    cfg = TransformConfig(name=f"{method}_age", transformer_type="numerical",
                          method=method, columns=["AGE"], params=params)
    t = NumericalTransformer(cfg)
    result = t.fit_transform(X_train)['AGE']

    axes[i].hist(result, bins=30, color=color, alpha=0.8)
    axes[i].set_title(f"`{method}`\nrange [{result.min():.2f}, {result.max():.2f}]", fontweight='bold')
    axes[i].set_xlabel('Scaled value')

plt.tight_layout()
plt.show()

# Show outlier effect on PAY_0 (can range from -2 to 9)
print("Repayment status (PAY_0) range in training data:")
print(f"  min={X_train['PAY_0'].min()}, max={X_train['PAY_0'].max()}, unique values: {sorted(X_train['PAY_0'].unique())}")
print("\nUsing robust_scale for PAY columns — resistant to extreme delay values (8, 9 months).")

fig, axes = plt.subplots(1, 3, figsize=(16, 5)) fig.suptitle("Scaling Transforms on `AGE`", fontsize=13, fontweight='bold') axes[0].hist(X_train['AGE'], bins=30, color=GREY, alpha=0.8) axes[0].set_title(f"Original\nrange [{X_train['AGE'].min():.0f}, {X_train['AGE'].max():.0f}]", fontweight='bold') for i, (method, params, color) in enumerate([ ('standard_scale', {}, BLUE), ('robust_scale', {}, ORANGE), ], start=1): cfg = TransformConfig(name=f"{method}_age", transformer_type="numerical", method=method, columns=["AGE"], params=params) t = NumericalTransformer(cfg) result = t.fit_transform(X_train)['AGE'] axes[i].hist(result, bins=30, color=color, alpha=0.8) axes[i].set_title(f"`{method}`\nrange [{result.min():.2f}, {result.max():.2f}]", fontweight='bold') axes[i].set_xlabel('Scaled value') plt.tight_layout() plt.show() # Show outlier effect on PAY_0 (can range from -2 to 9) print("Repayment status (PAY_0) range in training data:") print(f" min={X_train['PAY_0'].min()}, max={X_train['PAY_0'].max()}, unique values: {sorted(X_train['PAY_0'].unique())}") print("\nUsing robust_scale for PAY columns — resistant to extreme delay values (8, 9 months).")

Repayment status (PAY_0) range in training data:
  min=-2, max=8, unique values: [np.int64(-2), np.int64(-1), np.int64(0), np.int64(1), np.int64(2), np.int64(3), np.int64(4), np.int64(5), np.int64(6), np.int64(7), np.int64(8)]

Using robust_scale for PAY columns — resistant to extreme delay values (8, 9 months).

7c. Quantile Transforms: quantile_normal, quantile_uniform¶

In [11]:

fig, axes = plt.subplots(1, 3, figsize=(16, 5))
fig.suptitle("Quantile Transforms on `PAY_AMT1`", fontsize=13, fontweight='bold')

axes[0].hist(X_train['PAY_AMT1'], bins=50, color=GREY, alpha=0.8)
axes[0].set_title(f"Original\nskew = {X_train['PAY_AMT1'].skew():.2f}", fontweight='bold')

for i, (method, color, label) in enumerate([
    ('quantile_normal',  BLUE,   'quantile_normal\n\u2192 Gaussian output'),
    ('quantile_uniform', ORANGE, 'quantile_uniform\n\u2192 Uniform [0,1] output'),
], start=1):
    cfg = TransformConfig(name=f"{method}_pamt1", transformer_type="numerical",
                          method=method, columns=["PAY_AMT1"], params={"n_quantiles": 1000})
    t = NumericalTransformer(cfg)
    result = t.fit_transform(X_train)['PAY_AMT1']

    axes[i].hist(result, bins=50, color=color, alpha=0.8)
    axes[i].set_title(f"{label}\nskew = {result.skew():.4f}", fontweight='bold')
    axes[i].set_xlabel('Transformed value')

plt.tight_layout()
plt.show()
print("quantile_normal achieves near-zero skewness by construction — the most powerful normalisation available.")

fig, axes = plt.subplots(1, 3, figsize=(16, 5)) fig.suptitle("Quantile Transforms on `PAY_AMT1`", fontsize=13, fontweight='bold') axes[0].hist(X_train['PAY_AMT1'], bins=50, color=GREY, alpha=0.8) axes[0].set_title(f"Original\nskew = {X_train['PAY_AMT1'].skew():.2f}", fontweight='bold') for i, (method, color, label) in enumerate([ ('quantile_normal', BLUE, 'quantile_normal\n\u2192 Gaussian output'), ('quantile_uniform', ORANGE, 'quantile_uniform\n\u2192 Uniform [0,1] output'), ], start=1): cfg = TransformConfig(name=f"{method}_pamt1", transformer_type="numerical", method=method, columns=["PAY_AMT1"], params={"n_quantiles": 1000}) t = NumericalTransformer(cfg) result = t.fit_transform(X_train)['PAY_AMT1'] axes[i].hist(result, bins=50, color=color, alpha=0.8) axes[i].set_title(f"{label}\nskew = {result.skew():.4f}", fontweight='bold') axes[i].set_xlabel('Transformed value') plt.tight_layout() plt.show() print("quantile_normal achieves near-zero skewness by construction — the most powerful normalisation available.")

quantile_normal achieves near-zero skewness by construction — the most powerful normalisation available.

7d. Winsorization: winsorize¶

In [12]:

fig, axes = plt.subplots(1, 3, figsize=(16, 5))
fig.suptitle("Winsorization on `LIMIT_BAL` — Clipping Extreme Values", fontsize=13, fontweight='bold')

axes[0].boxplot(X_train['LIMIT_BAL'].dropna(), vert=True, patch_artist=True,
                boxprops=dict(facecolor=GREY, alpha=0.7))
axes[0].set_title(f"Original\nmax={X_train['LIMIT_BAL'].max():,}", fontweight='bold')
axes[0].set_xticks([])

for i, (lower, upper, color) in enumerate([(0.01, 0.99, BLUE), (0.05, 0.95, ORANGE)], start=1):
    cfg = TransformConfig(name=f"winsorize_{int(lower*100)}_{int(upper*100)}_bal",
                          transformer_type="numerical", method="winsorize",
                          columns=["LIMIT_BAL"], params={"lower": lower, "upper": upper})
    t = NumericalTransformer(cfg)
    t.fit(X_train)
    result = t.transform(X_train)['LIMIT_BAL']
    fitted = t.get_params()
    clip_lo, clip_hi = fitted['LIMIT_BAL_lower'], fitted['LIMIT_BAL_upper']

    axes[i].boxplot(result.dropna(), vert=True, patch_artist=True,
                    boxprops=dict(facecolor=color, alpha=0.7))
    axes[i].set_title(f"Winsorize ({int(lower*100)}%–{int(upper*100)}%)\nclipped to [{clip_lo:,.0f}, {clip_hi:,.0f}]",
                      fontweight='bold')
    axes[i].set_xticks([])

plt.tight_layout()
plt.show()

fig, axes = plt.subplots(1, 3, figsize=(16, 5)) fig.suptitle("Winsorization on `LIMIT_BAL` — Clipping Extreme Values", fontsize=13, fontweight='bold') axes[0].boxplot(X_train['LIMIT_BAL'].dropna(), vert=True, patch_artist=True, boxprops=dict(facecolor=GREY, alpha=0.7)) axes[0].set_title(f"Original\nmax={X_train['LIMIT_BAL'].max():,}", fontweight='bold') axes[0].set_xticks([]) for i, (lower, upper, color) in enumerate([(0.01, 0.99, BLUE), (0.05, 0.95, ORANGE)], start=1): cfg = TransformConfig(name=f"winsorize_{int(lower*100)}_{int(upper*100)}_bal", transformer_type="numerical", method="winsorize", columns=["LIMIT_BAL"], params={"lower": lower, "upper": upper}) t = NumericalTransformer(cfg) t.fit(X_train) result = t.transform(X_train)['LIMIT_BAL'] fitted = t.get_params() clip_lo, clip_hi = fitted['LIMIT_BAL_lower'], fitted['LIMIT_BAL_upper'] axes[i].boxplot(result.dropna(), vert=True, patch_artist=True, boxprops=dict(facecolor=color, alpha=0.7)) axes[i].set_title(f"Winsorize ({int(lower*100)}%–{int(upper*100)}%)\nclipped to [{clip_lo:,.0f}, {clip_hi:,.0f}]", fontweight='bold') axes[i].set_xticks([]) plt.tight_layout() plt.show()

7e. Percentile Binning: percentile_binning¶

In [13]:

cfg_bin = TransformConfig(name="bin_age", transformer_type="numerical",
                         method="percentile_binning", columns=["AGE"], params={"n_bins": 4})
t_bin = NumericalTransformer(cfg_bin)
X_binned = t_bin.fit_transform(X_train)
bin_edges = t_bin.get_params()['AGE_bin_edges']

fig, axes = plt.subplots(1, 2, figsize=(14, 5))
fig.suptitle("Percentile Binning on `AGE`", fontsize=13, fontweight='bold')

axes[0].hist(X_train['AGE'], bins=30, color=GREY, alpha=0.8)
for edge in bin_edges[1:-1]:
    axes[0].axvline(edge, color=RED, linestyle='--', lw=1.5, alpha=0.7)
axes[0].set_title('Original — dashed lines show bin edges', fontweight='bold')
axes[0].set_xlabel('AGE')

bin_counts = X_binned['AGE'].value_counts().sort_index()
axes[1].bar([str(b) for b in bin_counts.index], bin_counts.values, color=BLUE, alpha=0.8)
axes[1].set_title('After binning — approximately equal counts per bin', fontweight='bold')
axes[1].set_xlabel('Bin (0 = youngest quartile, 3 = oldest)')
axes[1].set_ylabel('Count')

plt.tight_layout()
plt.show()

print("Learned age quartile edges:")
names = ['Young (Q1)', 'Early-career (Q2)', 'Mid-career (Q3)', 'Senior (Q4)']
for name, (lo, hi) in zip(names, [(bin_edges[i], bin_edges[i+1]) for i in range(len(bin_edges)-1)]):
    print(f"  {name}: age [{lo:.0f}, {hi:.0f}) years")

cfg_bin = TransformConfig(name="bin_age", transformer_type="numerical", method="percentile_binning", columns=["AGE"], params={"n_bins": 4}) t_bin = NumericalTransformer(cfg_bin) X_binned = t_bin.fit_transform(X_train) bin_edges = t_bin.get_params()['AGE_bin_edges'] fig, axes = plt.subplots(1, 2, figsize=(14, 5)) fig.suptitle("Percentile Binning on `AGE`", fontsize=13, fontweight='bold') axes[0].hist(X_train['AGE'], bins=30, color=GREY, alpha=0.8) for edge in bin_edges[1:-1]: axes[0].axvline(edge, color=RED, linestyle='--', lw=1.5, alpha=0.7) axes[0].set_title('Original — dashed lines show bin edges', fontweight='bold') axes[0].set_xlabel('AGE') bin_counts = X_binned['AGE'].value_counts().sort_index() axes[1].bar([str(b) for b in bin_counts.index], bin_counts.values, color=BLUE, alpha=0.8) axes[1].set_title('After binning — approximately equal counts per bin', fontweight='bold') axes[1].set_xlabel('Bin (0 = youngest quartile, 3 = oldest)') axes[1].set_ylabel('Count') plt.tight_layout() plt.show() print("Learned age quartile edges:") names = ['Young (Q1)', 'Early-career (Q2)', 'Mid-career (Q3)', 'Senior (Q4)'] for name, (lo, hi) in zip(names, [(bin_edges[i], bin_edges[i+1]) for i in range(len(bin_edges)-1)]): print(f" {name}: age [{lo:.0f}, {hi:.0f}) years")

Learned age quartile edges:
  Young (Q1): age [21, 28) years
  Early-career (Q2): age [28, 34) years
  Mid-career (Q3): age [34, 41) years
  Senior (Q4): age [41, 75) years

---

8. Categorical Transformations¶

8a. Label Encoding: label_encode¶

In [14]:

cfg_le = TransformConfig(name="label_sex", transformer_type="categorical",
                        method="label_encode", columns=["SEX"],
                        params={"unknown_strategy": "use_encoded_value", "unknown_value": -1})
t_le = CategoricalTransformer(cfg_le)
X_le = t_le.fit_transform(X_train)

learned_classes = t_le.get_params()['SEX_classes']
print("Learned label encoding for SEX:")
for i, cls in enumerate(learned_classes):
    count = (X_train['SEX'] == cls).sum()
    print(f"  {cls!r:15s} \u2192 {i}  (n={count:,})")

cfg_le = TransformConfig(name="label_sex", transformer_type="categorical", method="label_encode", columns=["SEX"], params={"unknown_strategy": "use_encoded_value", "unknown_value": -1}) t_le = CategoricalTransformer(cfg_le) X_le = t_le.fit_transform(X_train) learned_classes = t_le.get_params()['SEX_classes'] print("Learned label encoding for SEX:") for i, cls in enumerate(learned_classes): count = (X_train['SEX'] == cls).sum() print(f" {cls!r:15s} \u2192 {i} (n={count:,})")

Learned label encoding for SEX:
  'Female'        → 0  (n=14,514)
  'Male'          → 1  (n=9,486)

8b. One-Hot Encoding: onehot_encode¶

In [15]:

cfg_ohe = TransformConfig(name="ohe_education", transformer_type="categorical",
                         method="onehot_encode", columns=["EDUCATION"],
                         params={"sparse": False, "drop": None})
t_ohe = CategoricalTransformer(cfg_ohe)
X_ohe = t_ohe.fit_transform(X_train)

print(f"Columns before OHE: {X_train.shape[1]}")
print(f"Columns after  OHE: {X_ohe.shape[1]}")
new_cols = [c for c in X_ohe.columns if c.startswith('EDUCATION')]
print(f"\nNew columns: {new_cols}")
for col in new_cols:
    print(f"  {col:30s}  mean={X_ohe[col].mean():.3f}")

cfg_ohe = TransformConfig(name="ohe_education", transformer_type="categorical", method="onehot_encode", columns=["EDUCATION"], params={"sparse": False, "drop": None}) t_ohe = CategoricalTransformer(cfg_ohe) X_ohe = t_ohe.fit_transform(X_train) print(f"Columns before OHE: {X_train.shape[1]}") print(f"Columns after OHE: {X_ohe.shape[1]}") new_cols = [c for c in X_ohe.columns if c.startswith('EDUCATION')] print(f"\nNew columns: {new_cols}") for col in new_cols: print(f" {col:30s} mean={X_ohe[col].mean():.3f}")

Columns before OHE: 23
Columns after  OHE: 26

New columns: ['EDUCATION_Graduate', 'EDUCATION_High_School', 'EDUCATION_Other', 'EDUCATION_University']
  EDUCATION_Graduate              mean=0.352
  EDUCATION_High_School           mean=0.163
  EDUCATION_Other                 mean=0.016
  EDUCATION_University            mean=0.469

8c. Frequency Encoding: frequency_encode¶

In [16]:

cfg_fe = TransformConfig(name="freq_marriage", transformer_type="categorical",
                        method="frequency_encode", columns=["MARRIAGE"],
                        params={"normalize": True})
t_fe = CategoricalTransformer(cfg_fe)
X_fe = t_fe.fit_transform(X_train)
freq_map = t_fe.get_params()['MARRIAGE_freq_map']

print("Frequency map learned from training data:")
for cat, freq in sorted(freq_map.items(), key=lambda x: -x[1]):
    print(f"  {cat:10s} \u2192 {freq:.4f}  ({freq*100:.1f}% of training rows)")

cfg_fe = TransformConfig(name="freq_marriage", transformer_type="categorical", method="frequency_encode", columns=["MARRIAGE"], params={"normalize": True}) t_fe = CategoricalTransformer(cfg_fe) X_fe = t_fe.fit_transform(X_train) freq_map = t_fe.get_params()['MARRIAGE_freq_map'] print("Frequency map learned from training data:") for cat, freq in sorted(freq_map.items(), key=lambda x: -x[1]): print(f" {cat:10s} \u2192 {freq:.4f} ({freq*100:.1f}% of training rows)")

Frequency map learned from training data:
  Single     → 0.5336  (53.4% of training rows)
  Married    → 0.4538  (45.4% of training rows)
  Other      → 0.0126  (1.3% of training rows)

8d. Target Encoding: target_encode¶

Target encoding is supervised: the encoder learns category-to-target statistics from the training target. In BitBullet, pass the target through fit(..., y=...) or pipeline.fit_transform(..., y=...) rather than storing y inside transformer params. This keeps the config declarative and the fitted state explicit.

The upgraded SDK target encoder is leakage-aware. During fit_transform it returns out-of-fold (OOF) encodings for the training rows, so a row never sees its own target. During later transform calls it uses the full training-data smoothed map, which is the state saved with the pipeline for inference. It supports regression targets, binary classification probabilities, and multiclass one-vs-rest probability columns.

In [ ]:

target_pipeline = TransformPipeline(name="target_encoding_example")
target_pipeline.add(
    "categorical",
    "target_encode",
    columns=["EDUCATION"],
    params={
        "target_type": "classification",   # binary target -> P(positive_label | category)
        "positive_label": 1,
        "cv_folds": 5,
        "cv_strategy": "stratified",
        "smoothing": 10.0,
        "random_state": 42,
    },
)

# Training path: out-of-fold encodings, so each row is encoded by folds that did not contain it.
X_train_te = target_pipeline.fit_transform(X_train[["EDUCATION"]], y=y_train)

# Inference path: the saved full-training-data map is applied without refitting.
X_test_te = target_pipeline.transform(X_test[["EDUCATION"]])

target_step = target_pipeline.transformers[0]
description = target_step.describe()

audit = pd.DataFrame({
    "education": X_train["EDUCATION"].to_numpy(),
    "target": y_train.to_numpy(),
    "oof_encoded": X_train_te["EDUCATION"].to_numpy(),
})

summary = (
    audit.groupby("education")
    .agg(
        n=("target", "size"),
        raw_default_rate=("target", "mean"),
        mean_oof_encoding=("oof_encoded", "mean"),
    )
    .sort_values("mean_oof_encoding", ascending=False)
)

print("Target encoder audit:")
print(f"  mode:       {description['mode']}")
print(f"  fit_scope:  {description['fit_scope']}")
print(f"  output:     {description['output_columns_by_input']}")
print(f"  test dtype:  {X_test_te['EDUCATION'].dtype}")
display(summary)

print("\nMulticlass target encoding expands one source column into one probability channel per class:")
toy_X = pd.DataFrame({
    "merchant_segment": ["grocery", "grocery", "fuel", "fuel", "travel", "travel", "grocery", "fuel", "travel", "other", "other", "other"],
})
toy_y = pd.Series(["approve", "review", "approve", "reject", "review", "reject", "approve", "review", "reject", "approve", "review", "reject"], name="decision")

multi_pipeline = TransformPipeline(name="multiclass_target_encoding_example")
multi_pipeline.add(
    "categorical",
    "target_encode",
    columns=["merchant_segment"],
    params={"target_type": "classification", "cv_folds": 3, "cv_strategy": "stratified", "smoothing": 2.0},
)
toy_encoded = multi_pipeline.fit_transform(toy_X, y=toy_y)
print([col for col in toy_encoded.columns if col.startswith("merchant_segment")])

target_pipeline = TransformPipeline(name="target_encoding_example") target_pipeline.add( "categorical", "target_encode", columns=["EDUCATION"], params={ "target_type": "classification", # binary target -> P(positive_label | category) "positive_label": 1, "cv_folds": 5, "cv_strategy": "stratified", "smoothing": 10.0, "random_state": 42, }, ) # Training path: out-of-fold encodings, so each row is encoded by folds that did not contain it. X_train_te = target_pipeline.fit_transform(X_train[["EDUCATION"]], y=y_train) # Inference path: the saved full-training-data map is applied without refitting. X_test_te = target_pipeline.transform(X_test[["EDUCATION"]]) target_step = target_pipeline.transformers[0] description = target_step.describe() audit = pd.DataFrame({ "education": X_train["EDUCATION"].to_numpy(), "target": y_train.to_numpy(), "oof_encoded": X_train_te["EDUCATION"].to_numpy(), }) summary = ( audit.groupby("education") .agg( n=("target", "size"), raw_default_rate=("target", "mean"), mean_oof_encoding=("oof_encoded", "mean"), ) .sort_values("mean_oof_encoding", ascending=False) ) print("Target encoder audit:") print(f" mode: {description['mode']}") print(f" fit_scope: {description['fit_scope']}") print(f" output: {description['output_columns_by_input']}") print(f" test dtype: {X_test_te['EDUCATION'].dtype}") display(summary) print("\nMulticlass target encoding expands one source column into one probability channel per class:") toy_X = pd.DataFrame({ "merchant_segment": ["grocery", "grocery", "fuel", "fuel", "travel", "travel", "grocery", "fuel", "travel", "other", "other", "other"], }) toy_y = pd.Series(["approve", "review", "approve", "reject", "review", "reject", "approve", "review", "reject", "approve", "review", "reject"], name="decision") multi_pipeline = TransformPipeline(name="multiclass_target_encoding_example") multi_pipeline.add( "categorical", "target_encode", columns=["merchant_segment"], params={"target_type": "classification", "cv_folds": 3, "cv_strategy": "stratified", "smoothing": 2.0}, ) toy_encoded = multi_pipeline.fit_transform(toy_X, y=toy_y) print([col for col in toy_encoded.columns if col.startswith("merchant_segment")])

8e. Rare Category Binning: bin_rare¶

In [18]:

print("EDUCATION distribution:")
edu_dist = X_train['EDUCATION'].value_counts(normalize=True)
for cat, freq in edu_dist.items():
    print(f"  {cat:15s}: {freq:.3f} ({freq*100:.1f}%)")

print()
for threshold in [0.01, 0.10, 0.20]:
    cfg_br = TransformConfig(name=f"bin_rare_{threshold}", transformer_type="categorical",
                             method="bin_rare", columns=["EDUCATION"],
                             params={"threshold": threshold, "other_label": "other"})
    t_br = CategoricalTransformer(cfg_br)
    X_br = t_br.fit_transform(X_train)
    rare_cats = t_br.get_params()['EDUCATION_rare_categories']
    print(f"  threshold={threshold:.2f}: {len(rare_cats)} merged \u2192 {X_br['EDUCATION'].nunique()} unique remain. Merged: {rare_cats}")

print("EDUCATION distribution:") edu_dist = X_train['EDUCATION'].value_counts(normalize=True) for cat, freq in edu_dist.items(): print(f" {cat:15s}: {freq:.3f} ({freq*100:.1f}%)") print() for threshold in [0.01, 0.10, 0.20]: cfg_br = TransformConfig(name=f"bin_rare_{threshold}", transformer_type="categorical", method="bin_rare", columns=["EDUCATION"], params={"threshold": threshold, "other_label": "other"}) t_br = CategoricalTransformer(cfg_br) X_br = t_br.fit_transform(X_train) rare_cats = t_br.get_params()['EDUCATION_rare_categories'] print(f" threshold={threshold:.2f}: {len(rare_cats)} merged \u2192 {X_br['EDUCATION'].nunique()} unique remain. Merged: {rare_cats}")

EDUCATION distribution:
  University     : 0.469 (46.9%)
  Graduate       : 0.352 (35.2%)
  High_School    : 0.163 (16.3%)
  Other          : 0.016 (1.6%)

  threshold=0.01: 0 merged → 4 unique remain. Merged: []
  threshold=0.10: 1 merged → 4 unique remain. Merged: ['Other']
  threshold=0.20: 2 merged → 3 unique remain. Merged: ['High_School', 'Other']

8f. Feature Hashing: hash_encode¶

In [19]:

print("MARRIAGE unique values:", X_train['MARRIAGE'].unique())
for n_features in [4, 8, 16]:
    cfg_hash = TransformConfig(name=f"hash_{n_features}", transformer_type="categorical",
                               method="hash_encode", columns=["MARRIAGE"],
                               params={"n_features": n_features})
    t_hash = CategoricalTransformer(cfg_hash)
    t_hash.fit(X_train)
    unique_cats = X_train['MARRIAGE'].unique()
    hash_mapping = {cat: hash(str(cat)) % n_features for cat in unique_cats}
    collisions = len(unique_cats) - len(set(hash_mapping.values()))
    print(f"  n_features={n_features:3d}: {collisions} collision(s) among {len(unique_cats)} categories: {hash_mapping}")

print("MARRIAGE unique values:", X_train['MARRIAGE'].unique()) for n_features in [4, 8, 16]: cfg_hash = TransformConfig(name=f"hash_{n_features}", transformer_type="categorical", method="hash_encode", columns=["MARRIAGE"], params={"n_features": n_features}) t_hash = CategoricalTransformer(cfg_hash) t_hash.fit(X_train) unique_cats = X_train['MARRIAGE'].unique() hash_mapping = {cat: hash(str(cat)) % n_features for cat in unique_cats} collisions = len(unique_cats) - len(set(hash_mapping.values())) print(f" n_features={n_features:3d}: {collisions} collision(s) among {len(unique_cats)} categories: {hash_mapping}")

MARRIAGE unique values: ['Single' 'Married' 'Other']
  n_features=  4: 0 collision(s) among 3 categories: {'Single': 2, 'Married': 3, 'Other': 0}
  n_features=  8: 0 collision(s) among 3 categories: {'Single': 2, 'Married': 7, 'Other': 0}
  n_features= 16: 0 collision(s) among 3 categories: {'Single': 2, 'Married': 7, 'Other': 0}

---

9. Composing a Production Pipeline¶

We now build a complete TransformPipeline for the credit card default dataset, guided by the EDA in Section 4:

Feature Group	Transform	Reasoning
PAY_AMT1–PAY_AMT6	log1p	Extreme right skew, non-negative, many zeros
BILL_AMT1–BILL_AMT6	yeo_johnson	Right skew, possible negative values (credit balance)
LIMIT_BAL	log1p	Strictly positive, right-skewed credit limit
AGE	standard_scale	Roughly normal, no outlier concern
PAY_0–PAY_6	robust_scale	Ordinal integers, some extreme delay values (8, 9)
SEX, MARRIAGE	label_encode	Low cardinality (2–3 values), tree models
EDUCATION	onehot_encode	4 categories, slightly higher cardinality

In [20]:

pay_amt_cols  = [f'PAY_AMT{i}' for i in range(1, 7)]
bill_amt_cols = [f'BILL_AMT{i}' for i in range(1, 7)]
pay_cols      = ['PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6']
binary_cats   = ['SEX', 'MARRIAGE']
ohe_cats      = ['EDUCATION']

default_pipeline = (
    TransformPipeline(name="credit_default_pipeline")

    # ── Numerical transforms ────────────────────────────────────────────────
    .add("numerical", "log1p",
         columns=pay_amt_cols + ['LIMIT_BAL'],
         name="log1p_payment_amounts")

    .add("numerical", "yeo_johnson",
         columns=bill_amt_cols,
         name="yj_bill_amounts")

    .add("numerical", "standard_scale",
         columns=["AGE"],
         name="standard_scale_age")

    .add("numerical", "robust_scale",
         columns=pay_cols,
         name="robust_scale_pay_status")

    # ── Categorical transforms ──────────────────────────────────────────────
    .add("categorical", "label_encode",
         columns=binary_cats,
         params={"unknown_strategy": "use_encoded_value", "unknown_value": -1},
         name="label_encode_binary_cats")

    .add("categorical", "onehot_encode",
         columns=ohe_cats,
         params={"sparse": False, "drop": None},
         name="onehot_education")
)

print(default_pipeline)
print(f"\nTotal steps: {len(default_pipeline)}")

pay_amt_cols = [f'PAY_AMT{i}' for i in range(1, 7)] bill_amt_cols = [f'BILL_AMT{i}' for i in range(1, 7)] pay_cols = ['PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6'] binary_cats = ['SEX', 'MARRIAGE'] ohe_cats = ['EDUCATION'] default_pipeline = ( TransformPipeline(name="credit_default_pipeline") # ── Numerical transforms ──────────────────────────────────────────────── .add("numerical", "log1p", columns=pay_amt_cols + ['LIMIT_BAL'], name="log1p_payment_amounts") .add("numerical", "yeo_johnson", columns=bill_amt_cols, name="yj_bill_amounts") .add("numerical", "standard_scale", columns=["AGE"], name="standard_scale_age") .add("numerical", "robust_scale", columns=pay_cols, name="robust_scale_pay_status") # ── Categorical transforms ────────────────────────────────────────────── .add("categorical", "label_encode", columns=binary_cats, params={"unknown_strategy": "use_encoded_value", "unknown_value": -1}, name="label_encode_binary_cats") .add("categorical", "onehot_encode", columns=ohe_cats, params={"sparse": False, "drop": None}, name="onehot_education") ) print(default_pipeline) print(f"\nTotal steps: {len(default_pipeline)}")

TransformPipeline(name='credit_default_pipeline', steps=6, status=not fitted)

Total steps: 6

---

10. fit_transform on Training Data¶

In [21]:

X_train_t = default_pipeline.fit_transform(X_train, verbose=True)

print()
print(f"Input  shape  : {X_train.shape}")
print(f"Output shape  : {X_train_t.shape}")
print(f"Pipeline fitted: {default_pipeline.is_fitted}")
print()
print("Columns AFTER transformation:")
print(list(X_train_t.columns))

X_train_t = default_pipeline.fit_transform(X_train, verbose=True) print() print(f"Input shape : {X_train.shape}") print(f"Output shape : {X_train_t.shape}") print(f"Pipeline fitted: {default_pipeline.is_fitted}") print() print("Columns AFTER transformation:") print(list(X_train_t.columns))

Fitting [1/6]: log1p_payment_amounts
Fitting [2/6]: yj_bill_amounts
Fitting [3/6]: standard_scale_age
Fitting [4/6]: robust_scale_pay_status
Fitting [5/6]: label_encode_binary_cats
Fitting [6/6]: onehot_education
Transforming [1/6]: log1p_payment_amounts
Transforming [2/6]: yj_bill_amounts
Transforming [3/6]: standard_scale_age
Transforming [4/6]: robust_scale_pay_status
Transforming [5/6]: label_encode_binary_cats
Transforming [6/6]: onehot_education

Input  shape  : (24000, 23)
Output shape  : (24000, 26)
Pipeline fitted: True

Columns AFTER transformation:
['LIMIT_BAL', 'SEX', 'MARRIAGE', 'AGE', 'PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6', 'BILL_AMT1', 'BILL_AMT2', 'BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6', 'PAY_AMT1', 'PAY_AMT2', 'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6', 'EDUCATION_Graduate', 'EDUCATION_High_School', 'EDUCATION_Other', 'EDUCATION_University']

In [22]:

print("Spot check: PAY_AMT1 (log1p)")
print(f"  Before: min={X_train['PAY_AMT1'].min():.0f}, max={X_train['PAY_AMT1'].max():.0f}, skew={X_train['PAY_AMT1'].skew():.3f}")
print(f"  After : min={X_train_t['PAY_AMT1'].min():.3f}, max={X_train_t['PAY_AMT1'].max():.3f}, skew={X_train_t['PAY_AMT1'].skew():.3f}")
print()
print("Spot check: BILL_AMT1 (yeo_johnson)")
print(f"  Before: skew={X_train['BILL_AMT1'].skew():.3f}")
print(f"  After : skew={X_train_t['BILL_AMT1'].skew():.3f}")
print()
print("Spot check: EDUCATION one-hot columns")
edu_cols = [c for c in X_train_t.columns if c.startswith('EDUCATION')]
print(f"  Columns: {edu_cols}")
print(f"  Row sums: {X_train_t[edu_cols].sum(axis=1).unique()}")

print("Spot check: PAY_AMT1 (log1p)") print(f" Before: min={X_train['PAY_AMT1'].min():.0f}, max={X_train['PAY_AMT1'].max():.0f}, skew={X_train['PAY_AMT1'].skew():.3f}") print(f" After : min={X_train_t['PAY_AMT1'].min():.3f}, max={X_train_t['PAY_AMT1'].max():.3f}, skew={X_train_t['PAY_AMT1'].skew():.3f}") print() print("Spot check: BILL_AMT1 (yeo_johnson)") print(f" Before: skew={X_train['BILL_AMT1'].skew():.3f}") print(f" After : skew={X_train_t['BILL_AMT1'].skew():.3f}") print() print("Spot check: EDUCATION one-hot columns") edu_cols = [c for c in X_train_t.columns if c.startswith('EDUCATION')] print(f" Columns: {edu_cols}") print(f" Row sums: {X_train_t[edu_cols].sum(axis=1).unique()}")

Spot check: PAY_AMT1 (log1p)
  Before: min=0, max=873552, skew=15.312
  After : min=0.000, max=13.680, skew=-1.299

Spot check: BILL_AMT1 (yeo_johnson)
  Before: skew=2.691
  After : skew=-2.029

Spot check: EDUCATION one-hot columns
  Columns: ['EDUCATION_Graduate', 'EDUCATION_High_School', 'EDUCATION_Other', 'EDUCATION_University']
  Row sums: [1.]

---

11. Save, Load, and Transform the Test Set¶

In [23]:

pipeline_path = "credit_default_pipeline.pkl"

default_pipeline.save(pipeline_path)
file_size_kb = os.path.getsize(pipeline_path) / 1024
print(f"Pipeline saved \u2192 {pipeline_path}  ({file_size_kb:.1f} KB)")

loaded_pipeline = TransformPipeline.load(pipeline_path)
print(f"Pipeline loaded: {len(loaded_pipeline)} steps  |  fitted: {loaded_pipeline.is_fitted}")

X_test_t = loaded_pipeline.transform(X_test)

print(f"\nTest set: {X_test.shape} \u2192 {X_test_t.shape}")
assert list(X_train_t.columns) == list(X_test_t.columns), "Column mismatch!"
print("Column alignment: PASSED")

pipeline_path = "credit_default_pipeline.pkl" default_pipeline.save(pipeline_path) file_size_kb = os.path.getsize(pipeline_path) / 1024 print(f"Pipeline saved \u2192 {pipeline_path} ({file_size_kb:.1f} KB)") loaded_pipeline = TransformPipeline.load(pipeline_path) print(f"Pipeline loaded: {len(loaded_pipeline)} steps | fitted: {loaded_pipeline.is_fitted}") X_test_t = loaded_pipeline.transform(X_test) print(f"\nTest set: {X_test.shape} \u2192 {X_test_t.shape}") assert list(X_train_t.columns) == list(X_test_t.columns), "Column mismatch!" print("Column alignment: PASSED")

Pipeline saved → credit_default_pipeline.pkl  (2.3 KB)
Pipeline loaded: 6 steps  |  fitted: True

Test set: (6000, 23) → (6000, 26)
Column alignment: PASSED

In [24]:

X_train_t_verify = loaded_pipeline.transform(X_train)
is_identical = X_train_t.round(10).equals(X_train_t_verify.round(10))
print(f"Reproducibility check: loaded pipeline produces {'IDENTICAL' if is_identical else 'DIFFERENT'} results")

print()
print("Distribution comparison — train vs test (post-transform):")
print(f"{'Column':15s} {'train mean':>12s} {'test mean':>12s} {'train std':>12s} {'test std':>12s}")
print('-' * 60)
for col in ['PAY_AMT1', 'BILL_AMT1', 'LIMIT_BAL', 'AGE', 'PAY_0']:
    print(f"{col:15s} {X_train_t[col].mean():12.4f} {X_test_t[col].mean():12.4f} "
          f"{X_train_t[col].std():12.4f} {X_test_t[col].std():12.4f}")

X_train_t_verify = loaded_pipeline.transform(X_train) is_identical = X_train_t.round(10).equals(X_train_t_verify.round(10)) print(f"Reproducibility check: loaded pipeline produces {'IDENTICAL' if is_identical else 'DIFFERENT'} results") print() print("Distribution comparison — train vs test (post-transform):") print(f"{'Column':15s} {'train mean':>12s} {'test mean':>12s} {'train std':>12s} {'test std':>12s}") print('-' * 60) for col in ['PAY_AMT1', 'BILL_AMT1', 'LIMIT_BAL', 'AGE', 'PAY_0']: print(f"{col:15s} {X_train_t[col].mean():12.4f} {X_test_t[col].mean():12.4f} " f"{X_train_t[col].std():12.4f} {X_test_t[col].std():12.4f}")

Reproducibility check: loaded pipeline produces IDENTICAL results

Distribution comparison — train vs test (post-transform):
Column            train mean    test mean    train std     test std
------------------------------------------------------------
PAY_AMT1              6.6368       6.6027       3.2440       3.2755
BILL_AMT1         13533.5819   13716.1056   18330.9054   17488.7793
LIMIT_BAL            11.6633      11.6617       0.9400       0.9456
AGE                  -0.0000       0.0288       1.0000       1.0120
PAY_0                -0.0141      -0.0270       1.1232       1.1264

---

12. Pipeline Management: list_steps, disable_step, enable_step¶

In [25]:

steps = default_pipeline.list_steps()
print(f"{'#':>3}  {'Name':40s} {'Type':12s} {'Method':20s} {'Enabled':8s}")
print('-' * 90)
for i, step in enumerate(steps, 1):
    print(f"{i:>3}  {step['name']:40s} {step['type']:12s} {step['method']:20s} "
          f"{'Yes' if step['enabled'] else 'No':8s}")
    if step['columns']:
        print(f"     columns: {step['columns']}")

steps = default_pipeline.list_steps() print(f"{'#':>3} {'Name':40s} {'Type':12s} {'Method':20s} {'Enabled':8s}") print('-' * 90) for i, step in enumerate(steps, 1): print(f"{i:>3} {step['name']:40s} {step['type']:12s} {step['method']:20s} " f"{'Yes' if step['enabled'] else 'No':8s}") if step['columns']: print(f" columns: {step['columns']}")

  #  Name                                     Type         Method               Enabled 
------------------------------------------------------------------------------------------
  1  log1p_payment_amounts                    numerical    log1p                Yes     
     columns: ['PAY_AMT1', 'PAY_AMT2', 'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6', 'LIMIT_BAL']
  2  yj_bill_amounts                          numerical    yeo_johnson          Yes     
     columns: ['BILL_AMT1', 'BILL_AMT2', 'BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6']
  3  standard_scale_age                       numerical    standard_scale       Yes     
     columns: ['AGE']
  4  robust_scale_pay_status                  numerical    robust_scale         Yes     
     columns: ['PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6']
  5  label_encode_binary_cats                 categorical  label_encode         Yes     
     columns: ['SEX', 'MARRIAGE']
  6  onehot_education                         categorical  onehot_encode        Yes     
     columns: ['EDUCATION']

In [26]:

X_with_log   = default_pipeline.transform(X_train)
skew_with    = X_with_log['PAY_AMT1'].skew()
print(f"With log1p enabled  : PAY_AMT1 skew = {skew_with:.4f}")

default_pipeline.disable_step("log1p_payment_amounts")
X_without_log = default_pipeline.transform(X_train)
skew_without  = X_without_log['PAY_AMT1'].skew()
print(f"With log1p disabled : PAY_AMT1 skew = {skew_without:.4f}  (raw values pass through)")

default_pipeline.enable_step("log1p_payment_amounts")
X_re_enabled  = default_pipeline.transform(X_train)
print(f"After re-enabling   : PAY_AMT1 skew = {X_re_enabled['PAY_AMT1'].skew():.4f}")

X_with_log = default_pipeline.transform(X_train) skew_with = X_with_log['PAY_AMT1'].skew() print(f"With log1p enabled : PAY_AMT1 skew = {skew_with:.4f}") default_pipeline.disable_step("log1p_payment_amounts") X_without_log = default_pipeline.transform(X_train) skew_without = X_without_log['PAY_AMT1'].skew() print(f"With log1p disabled : PAY_AMT1 skew = {skew_without:.4f} (raw values pass through)") default_pipeline.enable_step("log1p_payment_amounts") X_re_enabled = default_pipeline.transform(X_train) print(f"After re-enabling : PAY_AMT1 skew = {X_re_enabled['PAY_AMT1'].skew():.4f}")

With log1p enabled  : PAY_AMT1 skew = -1.2993
With log1p disabled : PAY_AMT1 skew = 15.3121  (raw values pass through)
After re-enabling   : PAY_AMT1 skew = -1.2993

---

13. Visualising Transformations¶

In [27]:

# Interactive widget — use the dropdown to explore each transformed feature
plot_pipeline_transformations(default_pipeline, X_train, X_train_t)

# Interactive widget — use the dropdown to explore each transformed feature plot_pipeline_transformations(default_pipeline, X_train, X_train_t)

interactive(children=(Dropdown(description='feature', options=('AGE', 'BILL_AMT1', 'BILL_AMT2', 'BILL_AMT3', '…

In [28]:

fig, axes = plt.subplots(4, 2, figsize=(14, 16))
fig.suptitle("Numerical Transforms — Before vs After (Training Data)", fontsize=14, fontweight='bold')

num_pairs = [
    ('PAY_AMT1',  'log1p',       BLUE,   GREEN),
    ('BILL_AMT1', 'yeo_johnson', BLUE,   ORANGE),
    ('LIMIT_BAL', 'log1p',       BLUE,   GREEN),
    ('AGE',       'standard_scale', BLUE, ORANGE),
]

for i, (col, method, c_before, c_after) in enumerate(num_pairs):
    before = X_train[col]
    after  = X_train_t[col]
    axes[i, 0].hist(before.dropna(), bins=40, color=c_before, alpha=0.75)
    axes[i, 0].set_title(f"`{col}` — Original  [skew={before.skew():.2f}]", fontweight='bold', fontsize=10)
    axes[i, 1].hist(after.dropna(), bins=40, color=c_after, alpha=0.75)
    axes[i, 1].set_title(f"`{col}` — After `{method}`  [skew={after.skew():.2f}]", fontweight='bold', fontsize=10)

plt.tight_layout()
plt.show()

fig, axes = plt.subplots(4, 2, figsize=(14, 16)) fig.suptitle("Numerical Transforms — Before vs After (Training Data)", fontsize=14, fontweight='bold') num_pairs = [ ('PAY_AMT1', 'log1p', BLUE, GREEN), ('BILL_AMT1', 'yeo_johnson', BLUE, ORANGE), ('LIMIT_BAL', 'log1p', BLUE, GREEN), ('AGE', 'standard_scale', BLUE, ORANGE), ] for i, (col, method, c_before, c_after) in enumerate(num_pairs): before = X_train[col] after = X_train_t[col] axes[i, 0].hist(before.dropna(), bins=40, color=c_before, alpha=0.75) axes[i, 0].set_title(f"`{col}` — Original [skew={before.skew():.2f}]", fontweight='bold', fontsize=10) axes[i, 1].hist(after.dropna(), bins=40, color=c_after, alpha=0.75) axes[i, 1].set_title(f"`{col}` — After `{method}` [skew={after.skew():.2f}]", fontweight='bold', fontsize=10) plt.tight_layout() plt.show()

---

14. Pipeline Metadata Inspection¶

In [29]:

meta = default_pipeline.metadata

print("Pipeline Metadata")
print("=" * 50)
print(f"  Pipeline name    : {default_pipeline.name}")
print(f"  Created at       : {meta.created_at}")
print(f"  Fitted at        : {meta.fitted_at}")
print(f"  Number of steps  : {meta.n_transformers}")
print(f"  Input shape      : {meta.input_shape}")
print(f"  Output shape     : {meta.output_shape}")
print(f"  Is fitted        : {default_pipeline.is_fitted}")
print(f"  Column expansion : +{meta.output_shape[1] - meta.input_shape[1]} columns (from one-hot encoding)")

meta = default_pipeline.metadata print("Pipeline Metadata") print("=" * 50) print(f" Pipeline name : {default_pipeline.name}") print(f" Created at : {meta.created_at}") print(f" Fitted at : {meta.fitted_at}") print(f" Number of steps : {meta.n_transformers}") print(f" Input shape : {meta.input_shape}") print(f" Output shape : {meta.output_shape}") print(f" Is fitted : {default_pipeline.is_fitted}") print(f" Column expansion : +{meta.output_shape[1] - meta.input_shape[1]} columns (from one-hot encoding)")

Pipeline Metadata
==================================================
  Pipeline name    : credit_default_pipeline
  Created at       : None
  Fitted at        : 2026-05-06T14:27:18.945569
  Number of steps  : 6
  Input shape      : (24000, 23)
  Output shape     : (24000, 26)
  Is fitted        : True
  Column expansion : +3 columns (from one-hot encoding)

---

15. Conclusion¶

This notebook covered bitbullet.transform applied to a real-world credit default dataset with a genuine mix of transformation needs.

What We Built¶

You Wrote	BitBullet Handled
generate_feature_stats(df)	Full audit — skewness, kurtosis, missing values, cardinality
pipeline.add("numerical", "log1p", ...)	Fit on training data, store parameters, apply consistently
pipeline.add("numerical", "yeo_johnson", ...)	Power transformer for negative-valued BILL_AMT columns
pipeline.fit_transform(X_train, y=y_train) when supervised encoders are present	Sequential chaining, target-aware fitting, fit-lock after completion
pipeline.save(path)	Compressed serialisation of all fitted parameters
TransformPipeline.load(path)	Exact reconstruction — identical parameters, identical output
loaded_pipeline.transform(X_test)	Apply exact training parameters — zero leakage

Quick Decision Guide¶

PAY_AMT columns (zeros, extreme right skew)  → log1p
BILL_AMT columns (right skew, possible negatives) → yeo_johnson
LIMIT_BAL (strictly positive, right-skewed)  → log1p
AGE (roughly normal)                         → standard_scale
PAY_0–PAY_6 (ordinal integer, some extremes) → robust_scale
SEX, MARRIAGE (2–3 categories, tree model)   → label_encode
EDUCATION (4 categories, nominal)            → onehot_encode

Continue the Academy:

• 05_Model_Management.ipynb — versioning, comparison, and model governance