Evaluate

Structured post-training evaluation for classification and regression models. Functions return typed report objects with .to_dict() for metadata pipelines. ModelEvaluator wraps both with a unified config-driven interface.

Configuration

bitbullet.evaluate.config.EvaluationConfig dataclass

Configuration for model evaluation.

Example::

config = EvaluationConfig(
    name="shipping_delay_eval",
    threshold=0.372,          # from Youden's J during training
    optimize_threshold=False,
    generate_plots=False,     # True → also set plot_output_dir
    generate_report=False,
)

Evaluator

bitbullet.evaluate.evaluator.ModelEvaluator

Evaluate a binary classifier on a held-out test set.

One call returns the full metrics suite, optional plots, and an optional text/JSON report.

Example::

from bitbullet.evaluate import ModelEvaluator, EvaluationConfig

config = EvaluationConfig(
    name="shipping_delay_eval",
    threshold=trainer_state.optimal_threshold,
)
evaluator = ModelEvaluator(config)
results   = evaluator.evaluate(y_test, y_pred_proba)
print(results)

evaluate(y_true, y_pred_proba, threshold=None)

Run evaluation.

Parameters:

Name	Type	Description	Default
y_true	ndarray	True binary labels (0 / 1).	required
y_pred_proba	ndarray	Predicted probabilities for the positive class.	required
threshold	Optional[float]	Override the config threshold for this call only.	None

Returns:

Type	Description
EvaluationResults	EvaluationResults with all metrics populated.

bitbullet.evaluate.evaluator.EvaluationResults dataclass

Container for a complete evaluation run.

Classification

bitbullet.evaluate.classification.ClassificationMetricsReport dataclass

Complete numeric classification report.

Attributes are intentionally JSON-friendly so the report can be attached to model metadata, saved by SDK users, or consumed by higher-level services.

to_dict()

Return a JSON-friendly dictionary.

bitbullet.evaluate.classification.evaluate_classification(y_true, y_pred=None, y_pred_proba=None, threshold=0.5, labels=None)

Calculate structured binary or multiclass classification metrics.

Parameters:

Name	Type	Description	Default
y_true		Ground-truth labels.	required
y_pred		Optional hard predictions. If omitted, predictions are derived from probabilities.	None
y_pred_proba		Optional class probabilities.	None
threshold	float	Binary positive-class threshold used when deriving predictions from probabilities.	0.5
labels	Optional[List[Any]]	Optional label order. Defaults to sorted unique labels.	None

Returns:

Type	Description
ClassificationMetricsReport	ClassificationMetricsReport with serializable metrics and curve data.

Regression

bitbullet.evaluate.regression.RegressionMetricsReport dataclass

Complete numeric regression report.

to_dict()

Return a JSON-friendly dictionary.

bitbullet.evaluate.regression.evaluate_regression(y_true, y_pred, *, n_features=None, include_values=True)

Calculate structured regression metrics.

Parameters:

Name	Type	Description	Default
y_true		Ground-truth continuous target values.	required
y_pred		Predicted continuous target values.	required
n_features	Optional[int]	Optional feature count used for adjusted R-squared.	None
include_values	bool	Whether to include predictions and residuals in the report. Disable this for very large evaluation sets.	True

Returns:

Type	Description
RegressionMetricsReport	RegressionMetricsReport with serializable metrics and summaries.