Plumbline · Independent backtest audit · Sample

Backtest Audit Sample

What a backtest audit checks and how it reaches a verdict. All three cases are our own strategies: one stopped by the audit, one allowed with conditions, and one whose headline number was withdrawn.

Data as of 2026-11-18 Report date 2026-11-19 Every number cites its source (end of each case) 中文版
This is a demonstration of method, not investment advice. The data, code and conclusions in all three cases come from our own research records. A client audit uses the return or position data the client supplies.

The three cases at a glance

CaseReported numberReading after auditVerdict
A US long/short equityBacktest Sharpe 2.92Indistinguishable from zero skill once the true trial count is applied; 91 days of paper trading at Sharpe 0.63, realized beta 0.52✕Not ready for live
B Commodity futuresClears 0.50 in both erasThe recent era clears by only 0.0038; 63% of its return comes from 2024–25◐Conditional: forward test only
C Momentum with a brakeLower of two eras 0.5589Ranks 8th of 63 rebalance start dates; median 0.3876↓Headline withdrawn

What this audit checks

Return basisHow the backtest books returns, and whether that matches how live trading books them
Look-aheadWhether the backtest used information it could not yet have had; from weights and returns, without the code
Trial countHow many variants were tried before this one was picked; the Sharpe deflated for the true count (Deflated Sharpe)
Two erasWhether the result holds in a period that played no part in the selection
Rebalance phaseHow much the result moves if rebalancing starts on a different day
Concentration in timeWhether one or two years carry the return; what is left without the best few days
Cost and capacityWhat is left with costs doubled, and when trading whole shares or whole contracts
Market exposureThe realized beta; what is left once the market is removed
Pre-registrationWhether the verdict rule was fixed before the results were seen
Forward recordHow much evidence has arrived since going live, and whether it is enough to decide
Case A · US long/short equity (S&P 500 members, daily rebalance)

A backtest Sharpe of 2.92 that is luck under honest counting

✕Not ready for live

The research loop picked this book automatically in June 2026: four price-based factors, a linear model, long one end and short the other. The exported backtest record looked excellent, and the book went into paper trading.

Backtest Sharpe (export record)2.92Annual return 110.9%, max drawdown −36.4%
Best of 2,688 zero-skill trials2.93The bar implied by the true trial count
Paper Sharpe (91 return days)0.63t-stat 0.38, cumulative +1.97%
CheckReadingJudgment
Return basisThe backtest booked a 5-day forecast label as if it were a daily return (the old basis, used before 2026-06-25). With a holding period h above 1, this inflates the Sharpe by roughly √h. By a rule-of-thumb conversion, 2.92 is about 1.30. This is a conversion, not a re-run.Not reliable
Trial countAt export the Sharpe was deflated for only 2 trials, giving a probability of 0.998. By that round the system had tried 2,688 model × strategy combinations in total. Deflated for 2,688, the best of that many zero-skill trials would also print 2.93, and the probability falls to 0.495: a coin toss.Indistinguishable from luck
Forward performancePaper trading 2026-07-13 to 11-18, 91 return days: Sharpe 0.63, annual volatility 9.2%, max drawdown −5.5%. That is 22% of the backtest, below the 25% overfitting line. Against the converted 1.30 it is about 48% (a conversion, not a re-run). With only 91 days this line misfires often, so it is a flag, not a reason on its own.Flag
Sequential testLooks at 63, 84, 105 … 252 days, with a boundary that falls as the sample grows. At day 84, t = 0.57 against a boundary of 1.95; the next look is at day 105.Not passed
Market exposureAverage net exposure on the books was only 2%, which looks market-neutral, but the realized beta to SPY was 0.52 (correlation 0.79). SPY fell 4.85% over the period. With the market removed the Sharpe is 2.22, t = 1.34: not significant over 91 days.Not neutral

Verdict

The backtest number was inflated in two systematic ways, the return basis and the selection count. After correcting both, what is left cannot be told apart from zero skill. The paper record cannot settle anything yet, and the realized beta is far above a market-neutral design (our later research bar is |beta| ≤ 0.3). Our own promotion check held the book at "not ready", and it never went live.

Sources
Export record: storage/paper_trading/ready/PB_0610_E289BD19.json (2026-06-10, round 422) Basis change: commit 138525f, 2026-06-25, default return basis changed to one_period Cumulative trial count: storage/state/dsr_trial_counter.json, 2,688 through round 422 Re-deflation: evaluation/deflated_sharpe.redeflate_with_trials (closed-form back-out, N 2 → 2,688) Paper NAV: storage/paper_trading/nav/PB_0610_E289BD19_29454fdbc222_nav.csv, from 2026-07-10 Sequential test: storage/live_trading/candidates/latest_live_promotion_candidates.json (2026-11-18T01:53Z) SPY: USEquityDaily.daily_bars, price return, dividends excluded
Case B · Commodity futures cross-section (23 markets, 5-day hold)

Cleared the bar by the rules, but only just

◐Conditional: forward test only

Three commodity signals already in the literature (hedging pressure, basis momentum, skewness) are combined equal-weight into one book, priced in whole contracts on US$1M of design capital, and hedged against the equal-weight market. The rules and data were frozen before the run, and it ran once.

Recent era 2008–2026, net Sharpe (hedged)0.5038t-stat 2.17, bar 0.50
Early era 1992–2007, net Sharpe (hedged)0.9466t-stat 3.78
Without 2024–250.2163% of the recent era's return comes from these two years
Recent-era return by calendar year (sum of daily returns, net of cost, hedged). 2026 runs through September 30.
Show as a table
YearReturn
CheckReadingJudgment
Pre-registrationPlan and verdict rule frozen before the run (2026-11-02 03:01 UTC), data snapshot checksummed, run once at 03:30 UTC; all 10 validity clauses held.Sound
Two erasBoth eras clear 0.50, and both beat the best single leg by more than 0.10. The recent era clears the bar by only 0.0038.Pass, thin margin
Concentration in timeSharpe over 2013–2022 is −0.09. Max drawdown −30%: peak 2014-09, trough 2021-12, recovered only in 2024-06.Main risk
Cost and contractsCosts doubled: 0.41. Standard contracts only, no micros: 0.40. Best 5 days removed: 0.43. All three are below the bar.Fails stress
DiversificationAverage pairwise correlation of the three legs' daily returns is 0.08 (recent era), and the combination does beat the best single leg (0.35).Sound
Selection biasBoth eras are the eras in which these three signals were originally found, so strictly there is no out-of-sample evidence yet.Unresolved
Rebalance phaseA new cohort every day, held 5 days with overlap. There is no fixed rebalance day, so there is no phase problem.Not applicable
Forward recordThe reading rule was frozen before the first forward return. As of 11-16 there are only 11 return days; the first reading is at day 63. By simulation, even with a true Sharpe of 0.50, the chance of reading "effective" within a year is only 35%.In progress

Verdict

The process is beyond reproach; the numbers are fragile. Forward testing can continue, but the book must not be described as "a Sharpe 0.50 strategy". The accurate description is: the recent era barely clears the bar, mostly on 2024–25, and out-of-sample evidence is still accumulating. Any capital decision waits for the forward reading.

Sources
Plan (frozen): docs/plans/ai_quant_lab_commodity_futures_combo_h5_v1_plan.md, anchor commit 0946cb4 Verdict: storage/research/futures_combo_h5_v1/result.json (snapshot sha256 eef5cb10…) Readout: storage/diagnostics/futures_combo_h5_v1_readout.json (by year, concentration, drawdown) Forward rule: docs/plans/ai_quant_lab_futures_paper_book_plan.md, frozen commit d6b54a3; power in its §5 Forward reading: storage/futures_paper/readings/reading_20261118.json (11 return days)
Case C · 63-day momentum + exogenous market brake

0.5589 was one of the luckier of 63 rebalance start dates

↓Headline withdrawn

This book rebalances every 63 trading days. By default the backtest starts rebalancing on day 0, but starting on day 1, day 2 … day 62 is just as reasonable. Reporting one start date means picking one result out of 63.

The result for each of the 63 rebalance start dates (net Sharpe, lower of the two eras), high to low. Blue is phase 0, the one originally reported.
Show as a table
Start dayResultRank
CheckReadingJudgment
Reported numberLower of two eras 0.5589, above the 0.50 bar: a pass under the pre-registered rule.Pass by the rule
Rebalance phasePhase 0 ranks 8th of 63; the median is 0.3876; only 24% of start dates clear the bar.Level withdrawn
MechanismThe brake itself helps at every start date and in every era. That part stands.Stands
Follow-up trialThe same brake on a book that forms a cohort every day and holds the cohorts with overlap, so it has no phase. Pre-registered, then run: 0.4946, below the bar.Family closed

Verdict

The original verdict stands, because the rule is the rule. The sentence "this is a 0.56 book" is withdrawn. For a strategy that rebalances less often than daily, the headline number should be the median across all start dates, not the default one. Many published factor indices have the same problem, known in the industry as rebalance timing luck.

Sources
Phase band: storage/diagnostics/regime_overlay_momentum_v2_phase_band.json (2026-09-16, rule commit ef36408, outcome withdrawn) Follow-up trial: registry regime-overlay-overlap-momentum-v3, 2026-09-17, no_gain, lower of two eras 0.4946

What this audit can and cannot do

What the client provides

Our own record

We apply these methods to ourselves first. Our research registry holds 63 registered hypotheses: 49 closed by rules fixed in advance, 4 kept, 4 shelved, and 6 still candidates or in build. Of the 3 passes, 2 did not survive stricter checks later: one turned negative in the holdout period, and the other is case C above.