How we learned to be honest before trying to be clever

Engine 1 used roughly 26 factors: records, Elo ratings, age and reach differences, recent form, layoffs, and basic fight statistics. Its job was not to revolutionise sports forecasting. Its job was to establish whether fighter history contained a real signal.

In chronological testing—learning from older fights and predicting newer ones—it achieved approximately 58.4% accuracy. That was not spectacular, but it was meaningfully better than random guessing.

A separate, much smaller validation slice appeared to score above 75%, with a calibrated figure near 79%. Those were not valid measures of future performance. The model had been adjusted using the same small group of fights on which it was judged. The honest walk-forward result remained 58.4%.

We added richer defensive statistics: striking defence, takedown defence, strikes absorbed, and submission activity. Validation accuracy jumped to almost 87%. Some forecasts became virtually 0% or 100%.

It looked like a breakthrough. It was actually data leakage. A historical fight from 2015 could receive a fighter's present-day career average—containing results from fights that had not yet happened in 2015. The model was predicting the past with information from the future.

We withdrew the affected models and returned to the safer Engine 1 feature set. Suspiciously good results stopped being a reason to celebrate and became a reason to investigate.

Engine 2 was less a finished new brain and more a data-foundation project. We realised richer statistics could only be used safely if they were preserved as they existed before each fight.

We began preserving fight-by-fight and round-by-round information: strikes, takedowns, submission attempts, control time, target areas, fighting positions, and late-round output. This made it possible to study cardio and defence without importing the future into the past.

Some Engine 2-labelled artifacts were produced, but later withdrawn because of leakage or evaluation problems. Engine 2 never earned a clean new public benchmark. Its true contribution was the point-in-time data foundation used by later engines.

Engine 3 reorganised the project into a modular system supporting several model families, a combined ensemble, finish-method predictions, probability calibration, and red-versus-blue mirroring.

Its original development test reported approximately 64.4% for the independent model and 68.6% for an odds-aware version, compared with roughly 66.2% for the Vegas benchmark used in that test.

We interpreted that too enthusiastically. The comparison used a limited holdout, not the later multi-period walk-forward system. Calibration remained too close to evaluation data, and two intended model families could not run in the original environment. Engine 3 was an architectural success, but declaring that it had beaten the market was premature.

Engine 4 expanded to roughly 90 factors and attempted to combine multiple models, recent-fight weighting, specialised division models, and probability calibration. On paper, it was sophisticated.

The audit found that the complete Engine 4 training process had never successfully finished in production. A short timeout repeatedly killed weekly training, while the surrounding process continued without making the failure visible. The site was still serving an older, simpler artifact—marketed as Engine 4, but closer to Engine 2.5.

We also discovered broken red/blue mirroring, an incorrect calibration metric, calibration on test fights, recurring future-data leakage, flawed payout calculations, and missing odds counted as false market disagreements.

Engine 5 rebuilt the measurement system and tested each complicated Engine 4 component separately. Several sophisticated ideas made performance worse: learned model stacking, division-specific routing, extra calibration, and cohort adjustments. We retired them.

The simpler global model was more dependable. On the main recent window it achieved approximately 60.7% accuracy, while the market achieved approximately 67.2%. The model contained a small additional signal, but it fell below thresholds written before the result was known.

We followed the rules: no claim that Fight Detective beats the market, no model-driven BET labels, and customer probabilities anchored to the betting line with the bookmaker's margin removed. The independent model remained a research pick.

Engine 5.1 then improved operational truthfulness—showing whether a job was dispatched, running, successful, failed, or cancelled—without changing a single prediction.

A historical replay asks what a model would have predicted across old fights. A live record proves that a prediction existed before the event. Those are not the same form of evidence.

Engine 6 separated them. Retrospective backtests remained research. The live record would count only predictions generated, locked, and published before an official deadline. It also corrected the market benchmark by requiring valid prices for both fighters and removing the bookmaker's margin consistently.

Engine 6.1 gave each prediction a lifecycle: generated privately, published before an immutable deadline, superseded only through a recorded action, and settled through an append-only result history. Corrections create revisions instead of erasing what came before.

The release was initially blocked because the database and application were temporarily at different stages. Running the new prediction code too early could have hidden real upcoming picks. We stopped and required a controlled lifecycle test rather than accepting the risk.

Engine 6.1 did not make the model smarter. It made the evidence harder to manipulate, including by accident.

Engine 6.2 returned to the modelling question. Engine 5 had been trained on an older portion of history and then carried forward. We tested whether the same LightGBM family improved when repeatedly refitted as new completed events became available.

The exact deployed Engine 5 artifact faced the refitted challenger on 2,151 identical fights across nine chronological periods. Entire UFC events stayed together. Engine 6.2 won all nine periods.

Its average log-loss improvement—penalising wrong answers and misplaced confidence—was approximately 0.0177 per fight, with the lower end of the event-based confidence range still positive. Calibration measures also improved. The governed verdict was B-WIN: the current-information refit beat the stale Engine 5 control.

The first production launch still failed safely. Reversing red and blue on the enriched live feature frame produced a symmetry error of about 6.5 percentage points. We fixed the transformation and added regression tests. The second release reduced that error to effectively zero before activating the model trained on 7,169 completed fights.

We explicitly overrode the originally planned 50-fight shadow period for immediate activation, and recorded that decision. Engine 6.2 proved it was a better refitting method than the frozen Engine 5 control. It did not prove that it beats the betting market or guarantees profit. That requires prospective live evidence.