Conformal Prediction Kit

Problem#

A point prediction says what the model expects, but not how much to trust it. Most uncertainty methods need a specific model family or distributional assumptions.

What it does#

Conformal prediction turns any fitted model into one that outputs an interval (regression) or a set of labels (classification). Given exchangeable data, the result contains the truth at least 1 - alpha of the time, for any model, sample size or data distribution. A poorly calibrated model still gets valid coverage. It pays for it with wider intervals.

• Split conformal regression with constant-width intervals.
• Normalised intervals that widen where the data are noisier (heteroskedastic data).
• Prediction sets for classification.

Usage#

from conformal_kit import SplitConformalRegressor

calibration_pred = model.predict(X_calibration)
conformal = SplitConformalRegressor(alpha=0.1).fit(y_calibration, calibration_pred)
lower, upper = conformal.predict_interval(model.predict(X_test))

alpha=0.1 asks for 90% coverage.

Engineering#

MIT licensed, with tests. It comes from the same concern as my forecasting research: reporting how far a model can be trusted, not only how accurate it is on average.