Contents
Which is correct way to calibrate classifiers with scikit?
There are two things mentioned in the CalibratedClassifierCV docs that hint towards the ways it can be used: base_estimator: If cv=prefit, the classifier must have been fit already on data. cv: If “prefit” is passed, it is assumed that base_estimator has been fitted already and all data is used for calibration.
What is the correct way to use calibratedclassifiercv?
Also, if CalibratedClassifierCV should only be fit on models fit on a different training set, why would it’s default options be cv=3, which will also fit the base estimator? Does the cross validation handle the disjoint rule on its own? Question: what is the correct way to use CalibratedClassifierCV?
Why does calibratedclassifiercv underperform a direct base estimator?
I noticed that sklearn’s new CalibratedClassifierCV seems to underperform the direct base_estimator when the base_estimator is GradientBoostingClassifer, (I haven’t tested other classifiers). Interestingly, if make_classification ‘s parameters are: then the CalibratedClassifierCV seems to be the slight outperformer (log loss evaluation).
What’s the point of using a calibration classifier?
The point of using a calibration classifier is to come up with a probability prediction that behaves a little smoother than would a normal classifier. It is not to enhance your base estimator’s performance. So there’s no guarantee that the probabilities or log-loss would be the same (same neighbourhood, but not the same).
How is calibratedclassifiercv used in probability calibration?
CalibratedClassifierCV uses a cross-validation approach to ensure unbiased data is always used to fit the calibrator. The data is split into k (train_set, test_set) couples (as determined by cv ).
When to use isotonic or sigmoid calibration in scikit?
Can be ‘sigmoid’ which corresponds to Platt’s method (i.e. a logistic regression model) or ‘isotonic’ which is a non-parametric approach. It is not advised to use isotonic calibration with too few calibration samples (<<1000) since it tends to overfit. Determines the cross-validation splitting strategy.
How to calibrate sklearn.svm estimators using probabilities?
This is the method implemented when probabilities=True for sklearn.svm estimators. Already fitted classifiers can be calibrated via the parameter cv=”prefit”. In this case, no cross-validation is used and all provided data is used for calibration.