Choosing the Right Metric to Evaluate a Betting Model

The metric that misleads most betting models is win rate, because it ignores both the price paid and the confidence of each forecast. Evaluate forecasts with proper scoring rules such as log loss or Brier score against the market's margin-free probabilities, check calibration, judge bets by closing line value, and read ROI only over large samples.

The metric the video means: win rate

The video is a 30-second short from the Predictive Play channel. The metric it calls destructive is win rate, the share of picks that win. Its argument: a model hitting 70 percent while backing heavy favourites can still lose over a season, a couple of upsets wipe out many small wins, and a 52 percent model at plus odds beats a 65 percent model on bad lines. Its recommendation is to focus on beating the closing price.

The arithmetic holds up. At decimal odds of 2.10 (plus 110), a 52 percent hit rate returns about 9 percent on stakes. At 1.45, a 65 percent hit rate loses about 6 percent. But replacing win rate with a single other number is not quite enough. Each evaluation metric answers a different question, and a sound process uses several, each at the stage where it is informative.

What each metric actually answers

Win rate (accuracy). How often the favoured side of the model's pick won. It ignores the price paid and the confidence stated, so it cannot tell a profitable model from an unprofitable one.

ROI. Profit per unit staked. It is the quantity you ultimately care about, but it is very noisy. At odds near 1.91, the standard error of ROI is about 95 percent divided by the square root of the number of bets: roughly 4.2 points after 500 bets and 3 points after 1,000. A small edge is invisible inside that.

Log loss. The average negative log of the probability the model gave to what actually happened. It rewards confident correct forecasts and punishes confident wrong ones severely.

Brier score. The mean squared difference between forecast probability and outcome (1 or 0). It is bounded and less harsh on extreme misses than log loss, and it splits into reliability, resolution and uncertainty terms, which helps diagnose why a model scores as it does.

Calibration. Whether outcomes rated at 70 percent happen about 70 percent of the time. Checked with a reliability plot or a binned error measure.

Closing line value. Whether the prices taken beat the market's final, most informed prices. It measures the decision, not the forecast, and settles far faster than ROI.

Why proper scoring rules beat accuracy

A scoring rule is proper when a forecaster gets the best expected score by reporting their true belief. Log loss and the Brier score are proper. Accuracy is not: it only checks which side of 50 percent a forecast falls on, so a model that says 51 percent and one that says 99 percent get identical credit. In betting, that difference decides the stake and whether a price looks like value at all.

Illustrative example: same accuracy, different models

Two models pick the same favourite in ten games, and the favourite wins six. Model A rates each at 60 percent; Model B rates each at 90 percent. Both have 60 percent accuracy.

Model A's log loss is 0.673 and its Brier score 0.240. Model B's log loss is 0.984 and its Brier score 0.330. A coin flip scores 0.693 and 0.250. Model B is worse than guessing by both proper rules, yet indistinguishable by accuracy, and it is the one that would see huge value in every favourite and bet accordingly.

Score against the market, not a coin flip

Beating 0.693 log loss proves almost nothing in sports, because the market already does. The useful baseline is the market's own margin-free probability for each game, ideally from the closing line at a sharp book. The question is whether the model's forecasts score better than the market's on the same games.

Compute the log loss difference game by game and look at its average and standard error. Being paired, this comparison is far more sensitive than comparing two totals. If the model cannot beat the market's probabilities anywhere, it has no information beyond the price, however good its accuracy looks. If it beats them only in certain markets or time windows, those are the candidates worth testing further.

The devig method matters here. Different methods spread the margin differently, especially on longshots, and the baseline should use the one that is best calibrated on your own markets.

Choosing the metric at each stage

During development: log loss and calibration on out-of-sample seasons, compared with the market's probabilities. These are proper, use every game rather than only bets, and expose overconfidence.

When deciding what to bet: expected value from calibrated probability against the offered price, minus a margin for estimation error. Accuracy plays no role here.

In production: closing line value first, because it gives a verdict in hundreds of bets; ROI eventually, with its confidence interval reported next to it.

Throughout: fix the metric before looking at results. Choosing whichever metric, threshold or subset looks best after the fact is its own form of overfitting, and it is how a win rate ends up in the headline.

Every one of these needs prices next to outcomes. Parlay Ledger provides historical odds, game results and player statistics across bookmakers in a unified format, so forecasts can be scored against market probabilities and closing lines on the same games.

Frequently asked questions

What metric should you use to evaluate a betting model?
Use more than one. Log loss and calibration on out-of-sample games, compared with the market's margin-free probabilities, show whether forecasts carry information. Closing line value shows whether bets beat the market's final price. ROI is the end goal but needs thousands of bets to read. Win rate alone should never decide anything.
Why is win rate a misleading metric for betting models?
Win rate ignores the price paid and the confidence of each forecast. Favourites win often and pay little, so a model that picks them can show a high win rate and still lose. In the video's example, 52 percent at plus odds beats 65 percent on short prices, which the break-even arithmetic confirms.
Is log loss or Brier score better for sports models?
Both are proper scoring rules, so either rewards honest probabilities. Log loss punishes confident misses heavily, which suits betting because overconfidence drives bad stakes. The Brier score is bounded and decomposes into reliability and resolution, which helps diagnosis. Many modellers report both, always against the market's probabilities as the baseline.
How many bets does ROI need to be meaningful?
Many. At odds near 1.91, the standard error of ROI is about 95 percent divided by the square root of the bet count, so roughly 4.2 points after 500 bets and 3 points after 1,000. An edge of 2 to 3 percent stays within noise for a long time, which is why closing line value is used earlier.