Contents
What is model uncertainty in machine learning?
Uncertainty means working with imperfect or incomplete information. Uncertainty is fundamental to the field of machine learning, yet it is one of the aspects that causes the most difficulty for beginners, especially those coming from a developer background.
What is the uncertainty for ML?
In the graduated cylinder shown in Figure 1, the mL graduations are marked and can be read with certainty. All graduated glassware is read with one estimated digit, so this measurement is recorded correctly to the nearest 0.1 mL, with an understood uncertainty of ± 0.1 mL.
What is the uncertainty of a 500ml beaker?
Minimum uncertainty is ± 5 mL. The markings may be off as much as 5%, but this has no impact on our measurements.
Why is uncertainty estimation important in machine learning?
For designing machine learning (ML) models as well as for monitoring them in production, uncertainty estimation on predictions is a critical asset. It helps identify suspicious samples during model training in addition to detecting out-of-distribution samples at inference time.
How to estimate uncertainty in a neural network?
In the last part of our series on uncertainty estimation, we addressed the limitations of approaches like bootstrapping for large models, and demonstrated how we might estimate uncertainty in the predictions of a neural network using MC Dropout.
Which is the best method for estimating uncertainty?
We see that the Mean-Variance estimation method produces the intervals with the smallest width, which results in a reduction of it’s PICP score. MC Dropout, and Quantile Regression produce very wide intervals, leading to a perfect PICP score.
How are uncertainty methods used in deep learning?
Most uncertainty methods are model-specific by design (Mondrian forest trees, quantile regression trees for non-deep learning algorithms) or incorporated during the initial model training through an additional branch or specific loss ( [ DeVries] and [ Corbière et al. ]).