What a backtest audit checks and how it reaches a verdict. All three cases are our own strategies: one stopped by the audit, one allowed with conditions, and one whose headline number was withdrawn.
| Case | Reported number | Reading after audit | Verdict |
|---|---|---|---|
| A US long/short equity | Backtest Sharpe 2.92 | Indistinguishable from zero skill once the true trial count is applied; 91 days of paper trading at Sharpe 0.63, realized beta 0.52 | ✕Not ready for live |
| B Commodity futures | Clears 0.50 in both eras | The recent era clears by only 0.0038; 63% of its return comes from 2024–25 | ◐Conditional: forward test only |
| C Momentum with a brake | Lower of two eras 0.5589 | Ranks 8th of 63 rebalance start dates; median 0.3876 | ↓Headline withdrawn |
The research loop picked this book automatically in June 2026: four price-based factors, a linear model, long one end and short the other. The exported backtest record looked excellent, and the book went into paper trading.
| Check | Reading | Judgment |
|---|---|---|
| Return basis | The backtest booked a 5-day forecast label as if it were a daily return (the old basis, used before 2026-06-25). With a holding period h above 1, this inflates the Sharpe by roughly √h. By a rule-of-thumb conversion, 2.92 is about 1.30. This is a conversion, not a re-run. | Not reliable |
| Trial count | At export the Sharpe was deflated for only 2 trials, giving a probability of 0.998. By that round the system had tried 2,688 model × strategy combinations in total. Deflated for 2,688, the best of that many zero-skill trials would also print 2.93, and the probability falls to 0.495: a coin toss. | Indistinguishable from luck |
| Forward performance | Paper trading 2026-07-13 to 11-18, 91 return days: Sharpe 0.63, annual volatility 9.2%, max drawdown −5.5%. That is 22% of the backtest, below the 25% overfitting line. Against the converted 1.30 it is about 48% (a conversion, not a re-run). With only 91 days this line misfires often, so it is a flag, not a reason on its own. | Flag |
| Sequential test | Looks at 63, 84, 105 … 252 days, with a boundary that falls as the sample grows. At day 84, t = 0.57 against a boundary of 1.95; the next look is at day 105. | Not passed |
| Market exposure | Average net exposure on the books was only 2%, which looks market-neutral, but the realized beta to SPY was 0.52 (correlation 0.79). SPY fell 4.85% over the period. With the market removed the Sharpe is 2.22, t = 1.34: not significant over 91 days. | Not neutral |
The backtest number was inflated in two systematic ways, the return basis and the selection count. After correcting both, what is left cannot be told apart from zero skill. The paper record cannot settle anything yet, and the realized beta is far above a market-neutral design (our later research bar is |beta| ≤ 0.3). Our own promotion check held the book at "not ready", and it never went live.
Three commodity signals already in the literature (hedging pressure, basis momentum, skewness) are combined equal-weight into one book, priced in whole contracts on US$1M of design capital, and hedged against the equal-weight market. The rules and data were frozen before the run, and it ran once.
| Year | Return |
|---|
| Check | Reading | Judgment |
|---|---|---|
| Pre-registration | Plan and verdict rule frozen before the run (2026-11-02 03:01 UTC), data snapshot checksummed, run once at 03:30 UTC; all 10 validity clauses held. | Sound |
| Two eras | Both eras clear 0.50, and both beat the best single leg by more than 0.10. The recent era clears the bar by only 0.0038. | Pass, thin margin |
| Concentration in time | Sharpe over 2013–2022 is −0.09. Max drawdown −30%: peak 2014-09, trough 2021-12, recovered only in 2024-06. | Main risk |
| Cost and contracts | Costs doubled: 0.41. Standard contracts only, no micros: 0.40. Best 5 days removed: 0.43. All three are below the bar. | Fails stress |
| Diversification | Average pairwise correlation of the three legs' daily returns is 0.08 (recent era), and the combination does beat the best single leg (0.35). | Sound |
| Selection bias | Both eras are the eras in which these three signals were originally found, so strictly there is no out-of-sample evidence yet. | Unresolved |
| Rebalance phase | A new cohort every day, held 5 days with overlap. There is no fixed rebalance day, so there is no phase problem. | Not applicable |
| Forward record | The reading rule was frozen before the first forward return. As of 11-16 there are only 11 return days; the first reading is at day 63. By simulation, even with a true Sharpe of 0.50, the chance of reading "effective" within a year is only 35%. | In progress |
The process is beyond reproach; the numbers are fragile. Forward testing can continue, but the book must not be described as "a Sharpe 0.50 strategy". The accurate description is: the recent era barely clears the bar, mostly on 2024–25, and out-of-sample evidence is still accumulating. Any capital decision waits for the forward reading.
This book rebalances every 63 trading days. By default the backtest starts rebalancing on day 0, but starting on day 1, day 2 … day 62 is just as reasonable. Reporting one start date means picking one result out of 63.
| Start day | Result | Rank |
|---|
| Check | Reading | Judgment |
|---|---|---|
| Reported number | Lower of two eras 0.5589, above the 0.50 bar: a pass under the pre-registered rule. | Pass by the rule |
| Rebalance phase | Phase 0 ranks 8th of 63; the median is 0.3876; only 24% of start dates clear the bar. | Level withdrawn |
| Mechanism | The brake itself helps at every start date and in every era. That part stands. | Stands |
| Follow-up trial | The same brake on a book that forms a cohort every day and holds the cohorts with overlap, so it has no phase. Pre-registered, then run: 0.4946, below the bar. | Family closed |
The original verdict stands, because the rule is the rule. The sentence "this is a 0.56 book" is withdrawn. For a strategy that rebalances less often than daily, the headline number should be the median across all start dates, not the default one. Many published factor indices have the same problem, known in the industry as rebalance timing luck.
We apply these methods to ourselves first. Our research registry holds 63 registered hypotheses: 49 closed by rules fixed in advance, 4 kept, 4 shelved, and 6 still candidates or in build. Of the 3 passes, 2 did not survive stricter checks later: one turned negative in the holdout period, and the other is case C above.