What property does a classifier have when its predicted confidence levels match the empirical frequency of correctness, so that predictions made with 80% confidence are right about 80% of the time?

Show answer

Calibration

A well-calibrated model's probabilities can be trusted as genuine uncertainty estimates, which matters for decision-making and deferral. Calibration is commonly measured with reliability diagrams or expected calibration error and improved with techniques such as temperature scaling.

Also asked as:

What term describes a model whose output probabilities accurately reflect the actual likelihood of the prediction being correct?

Would you remember this in 6 months?

Trivia Forge uses spaced repetition so facts stick. Free, no signup needed.

Turn on Machine Learning practice →

More Machine Learning trivia

What regularization technique penalizes the sum of the absolute values of the weights, often leading to sparse weight vectors?
What specific divergence measure is used in Variational Autoencoders (VAEs) to ensure the learned latent distribution stays close to a prior distribution?
What specific type of Autoencoder is designed to generate new data by learning the parameters of a probability distribution?
What term describes an agent achieving high reward by exploiting flaws or loopholes in a misspecified reward function instead of accomplishing the task its designer intended?
What term describes the phenomenon where a model performs well on training data but poorly on unseen test data?
What term describes the tendency of a neural network to abruptly lose previously learned capabilities when it is trained on a new task or data distribution?

See all 98 →