How to Track Model Performance Properly
Record the prediction, the timestamp, the price available at that moment, the closing price, and the outcome. Profit and loss alone cannot distinguish a good method from a lucky one, because outcomes are noisy and a small sample of them tells you almost nothing.
What the record has to contain
Most records store the decision and the result. That is the minimum useful format and it cannot answer the question you actually care about, which is whether the method works.
The prediction itself, as a probability rather than a pick. A record of picks cannot be checked for calibration, and calibration is the only property you can measure on a sample this small.
The timestamp, to the minute. Without it you cannot reconstruct what prices were available, which makes every downstream comparison approximate.
The price at that moment, as recorded, not as remembered.
The closing price for the same market. This is the column most records omit and the most valuable one they could add, for reasons below.
The outcome, which is the noisiest column in the file and the one people over-weight.
Whatever the model saw, or a reference to the input snapshot. Without this you cannot rerun a past decision under a revised model and get a meaningful comparison.
Predictions you did not act on. A record of only acted-upon decisions is a selected sample, and selected samples produce biased estimates of everything.
Why the format matters more than the discipline
People usually diagnose a useless record as a failure of consistency, and respond by resolving to log more carefully. That rarely helps, because the problem is structural. A record missing timestamps and prices cannot be analysed no matter how faithfully it was kept, while a record with those columns is informative even if it has gaps in it. Fix the schema first, then worry about how completely you fill it in.
Why closing price is the useful column
Outcomes are binary and noisy. A method with a genuine small edge and a method with none produce indistinguishable outcome records over hundreds of decisions. That is not a claim about anyone's discipline, it is a property of variance.
Comparing your price to the closing price gives you something with far better statistical properties.
It is continuous, so every record contributes information rather than one bit.
It resolves quickly, at market close rather than at settlement, which for futures markets can be months.
It measures the thing you can control. Whether you identified a mispricing before the market corrected is a question about your process. Whether the event then occurred is largely not.
A record showing consistently better prices than close, over a reasonable sample, is evidence about method. A record showing profit over the same sample might be evidence about method or might be variance, and nothing in the record distinguishes them.
This is a measurement argument, not a prescription about what to do with any particular market.
On sample size
The intuition that a few hundred records is a meaningful sample is wrong by roughly an order of magnitude for outcome-based evaluation. Price-based evaluation needs far less, because it measures a continuous quantity with much lower variance. This is the practical reason the extra column is worth the effort of capturing.
Segmenting the record
An aggregate number is an average over things that behave differently, and the average conceals which parts of the method are carrying it.
By market type. Main lines and derivative markets have different pricing quality and different amounts of attention on them. Performance in one says little about the other.
By time before event. A method that works on early prices and not on late ones is a different method from one that works throughout, and the distinction matters for how you would use it.
By your own stated confidence. This is the calibration check. Group by predicted probability and compare to observed frequency. If the group you called seventy percent lands near seventy percent, the model is calibrated in that region. If it lands at fifty, it is not, and knowing that is more actionable than any profit figure.
By source of the input data. When one source degrades, performance on predictions that depended on it degrades first, and segmenting is how you notice before the aggregate moves.
Keep the record append only, the way you would keep any other measurement log. A record you revise after the fact is a record of what you would like to have believed.
This page describes measurement practice and is not betting advice.
Frequently asked questions
- What should a model performance record contain?
- The prediction as a probability, a timestamp to the minute, the price available at that moment, the closing price for the same market, the outcome, and a reference to the inputs the model saw. Also the predictions you considered and did not act on.
- Why is profit and loss not enough?
- Because outcomes are binary and noisy. A method with a genuine small edge and one with no edge produce indistinguishable outcome records over hundreds of decisions. That is a property of variance, not a comment on anyone's discipline or record keeping.
- Why record the closing price?
- It is continuous rather than binary, so every record contributes information. It resolves at market close rather than at settlement. And it measures whether you identified a mispricing before the market corrected, which is a question about your process rather than about luck.
- How should the record be segmented?
- By market type, by how long before the event the prediction was made, by your own stated confidence level, and by which data source fed the inputs. An aggregate averages over things that behave differently and hides which parts are working.