Contents
Which is the correct definition of the Brier score?
Original definition by Brier. Although the above formulation is the most widely used, the original definition by Brier is applicable to multi-category forecasts as well as it remains a proper scoring rule, while the binary form is only proper for binary events.
The second term is known as refinement, and it is an aggregation of resolution and uncertainty, and is related to the area under the ROC Curve. The Brier Score, and the CAL + REF decomposition, can be represented graphically through the so-called Brier Curves, where the expected loss is shown for each operating condition.
When did Glenn Brier come up with the Brier score?
The set of possible outcomes can be either binary or categorical in nature, and the probabilities assigned to this set of outcomes must sum to one (where each individual probability is in the range of 0 to 1). It was proposed by Glenn W. Brier in 1950.
When to use the Brier score for uni dimensional predictions?
For uni dimensional predictions, it is strictly equivalent to the mean squared error as applied to predicted probabilities. The Brier score is applicable to tasks in which predictions must assign probabilities to a set of mutually exclusive discrete outcomes or classes.
Which is the best scoring rule for classification?
The two most commonly used proper scoring rules are the quadratic error measure, i.e., mean squared error or Brier score, and the logarithmic scoring rule, which is a linear translation of the log likelihood for a binary outcome model (Bernoulli trials).
Why do we use different scoring rules for logarithmic loss?
The authors show that choice of probability cutoff is equivalent to a choice of the relative cost of false-positive and false-negative classifications. They then provide a way to tailor loss functions to meet different choices of relative costs. So the choice of scoring rule might best take the eventual use of the model into account.