Where Language Models Fit in Predictive Modelling

Extracting structure from text. Injury reports, press conference transcripts, weather notes, local coverage. These contain information that matters and arrive as prose, which is exactly the transformation these models perform well. Output a structured record with defined fields and let a conventional model use it.

Entity reconciliation. The single most costly unglamorous problem in multi-source sports data is that nothing is named consistently. Proposing that two differently written names refer to the same player or club is a task these models handle well, with a human or a rule confirming. Not deciding, proposing, with the confirmation step preserved.

Triage of flagged anomalies. A statistical method detects that something is unusual. A language model is useful for reading the surrounding context and suggesting what kind of unusual it is: a data error, a genuine event, a definition change. That turns a queue of alerts into a ranked queue with explanations, which is a real operational improvement.

Code and query assistance for the analysts doing the work, which is not specific to this domain but is where most of the day-to-day value actually accrues.

Notice the pattern. In each case the model converts something unstructured into something a conventional method can consume, and the conventional method does the quantitative work.

The tempting application is to describe a situation and ask for a likelihood. It produces a number, the number sounds reasonable, and it is not what it appears to be.

It is not calibrated. A statistical model's outputs can be checked: over many predictions, do the things called forty percent occur about forty percent of the time. A language model's stated probability has no such guarantee, and in practice these outputs cluster on round numbers and shift with phrasing that should not matter.

It is not stable. Rewording a question changes the answer. Anything whose output moves with prompt phrasing rather than with evidence cannot be the basis of a quantitative estimate.

It cannot be decomposed. A statistical model can be interrogated: which inputs drove this, what happens if one changes. A generated probability comes with a fluent explanation that may or may not describe how the number was produced, and the explanation's plausibility is not evidence.

It may already know the outcome. For historical evaluation this is fatal. A model asked about a past event may have encountered its result during training, so any backtest is contaminated in a way that is difficult to detect and impossible to fully rule out.

That last point is worth stating plainly, because it invalidates the most natural way anyone would try to validate the approach.

Detection versus explanation

Treat outputs as features, not as answers. Anything extracted becomes an input to a model that is itself validated. The extraction step gets its own accuracy measurement against a labelled sample.

Keep a confirmation step wherever a mistake propagates. Entity mapping is the clear case. A mismap produces plausible wrong data that flows silently through everything downstream, which is worse than an obvious gap.

Version the prompt and the model. Outputs change when either changes, so results produced under different versions are not comparable and a track record spanning a change is not a track record.

Measure extraction quality separately. Precision and recall on a held-out labelled set, reported alongside the model that consumes the output. Without it you cannot tell whether a downstream problem is the model or the input.

Never let generated text become an unlabelled input. If a value came from a model rather than from a source, that provenance belongs in the record, because analysis that cannot distinguish observed from inferred values will eventually treat one as the other.

The summary: use them to make messy inputs usable, keep the quantitative work in methods that can be calibrated, and record which is which.

This page describes modelling practice and is not betting advice.