How We Validate a Predictive Model

A predictive model that has never been tested is not a model. It is an opinion with a spreadsheet attached. Validation is the step that separates a disciplined system from a guess, and it is the part of our work we take most seriously.

What backtesting means in plain terms

Backtesting is simple to describe and hard to do well. You take the model, feed it historical data as if it were live, and record what it would have predicted. Then you compare those predictions to what actually happened.

The catch: the past must look exactly like the future will. If your historical data contains information that would not have been available at the time of each prediction, the test is invalid. This is the classic "data leakage" failure — a model that scores brilliantly in the lab and does nothing in production.

The metrics we care about

Accuracy is where most people start, and it is where most people stop. It is not enough. We track a small set of measures that together tell the real story.

Accuracy — how often the model is right. Useful, but it hides the detail. A model can be 80% accurate and still be useless if the 20% it gets wrong are the important cases.

Precision — of the times the model makes a confident claim, how often is that claim correct. This matters when you act selectively: if you only act on the strongest signals, you want those signals to be reliable.

Calibration — does the model mean what it says. If a model assigns 70% probability to an outcome, that outcome should happen roughly 70% of the time across many predictions. A well-calibrated model is honest about its own uncertainty.

Maximum drawdown — the worst sustained decline in a results curve. Even a good model has losing stretches. Understanding how deep those stretches can go is what keeps expectations realistic and keeps a system running through the bad weeks instead of being abandoned.

Why validation is non-negotiable

Models fail in predictable ways. They overfit the past, memorizing noise instead of learning structure. They leak information. They drift as the world changes. Without a validation process, none of these failures are visible until real outcomes — and real stakes — expose them.

Validation is not a one-time event. It is a loop. New data arrives, the model is re-tested, and the evidence either confirms the model or forces a change. The goal is never a perfect model. It is a model whose weaknesses are known, measured and controlled.

We would rather run a modest model we understand than a complex one we cannot validate.

Read the next piece in this series: how the data actually gets from raw sources into the model.