What is a confidence interval in machine learning?
Confidence intervals are a way of quantifying the uncertainty of an estimate. They can be used to add a bounds or likelihood on a population parameter, such as a mean, estimated from a sample of independent observations from the population.
How do you find the confidence interval in machine learning?
If you want to calculate the 95% confidence interval, then the Z-critical value is 1.96. That means, the total area under the curve for a distance of 1.96 standard deviations from the center of the standard normal distribution on either side is 0.95, where the total area under the curve is taken as 1.0.
How are confidence intervals used in machine learning?
Instead, the confidence interval provides bounds on a population parameter, such as a mean, standard deviation, or similar. In applied machine learning, we may wish to use confidence intervals in the presentation of the skill of a predictive model.
How is a confidence interval different from a tolerance interval?
A confidence interval is different from a tolerance interval that describes the bounds of data sampled from the distribution. It is also different from a prediction interval that describes the bounds on a single observation. Instead, the confidence interval provides bounds on a population parameter, such as a mean, standard deviation, or similar.
What happens when confidence intervals for two models overlap?
If the confidence intervals for two models significantly overlap, this is an indication of (statistical) equivalence between the two and might provide a reason to favor the less complex or more interpretable model. — Page 416, Applied Predictive Modeling, 2013.
How is the confidence interval for a classification method calculated?
That a confidence interval is a bounds on an estimate of a population parameter. That the confidence interval for the estimated skill of a classification method can be calculated directly. That the confidence interval for any arbitrary population statistic can be estimated in a distribution-free way using the bootstrap.