Lessons learned · how we audit · what's bullshit and what's real
🔬 What's actually predictive in our universe (Russell 3000, 2018-2026)
944 feature × horizon combinations tested with 5+ years of point-in-time data. After deflating the inflation from overlapping returns, only 23 real signals survive. They cluster into 6 themes:
| Theme | Effect | Top features | Mean |IC| |
|---|---|---|---|
| Size premium | Small caps outperform large | mcap_pit, ev_pit, shares_diluted_*, raw net_income |
19.6% |
| Value premium | Cheap stocks outperform expensive | ps_ttm_pit, pe_ttm_pit, sales_yield_pit |
14.2% |
| Vol-as-risk-premium | High annualised vol → high return at 252d | vol_252d, vol_90d |
19.6% |
| Quality penalty | High-margin stocks UNDERPERFORM | net_margin_pit, gross_margin_pit, cfo_q_pct_quarters_positive_8q |
10.6% |
| Cash penalty | High cash hoards → lower returns | cash |
5.5% |
| Momentum-uptrend | Far above 52w low → higher returns | dist_52w_low, wq_alpha042 |
12.5% |
Most-counterintuitive finding: high-margin / consistently-profitable companies systematically underperform. This is the classic value-vs-quality tension — quality is already priced rich. Don't naïvely buy "best business" stocks expecting outperformance.
Industry-standard caveat: raw IR values look gigantic (e.g. 87 for size effect at 252d) because we sample daily with 252-day forward returns — consecutive observations share 251/252 of their forward window. Effective IR (deflated by √horizon) keeps the rankings honest. After deflation, the largest signal has IR = 5.5, well above Grinold's "great" threshold of 0.75.
What's NOT predictive (out of 800 noise features): most RSI variants, most chart patterns, most WorldQuant alphas, MACD, stochastic oscillators, raw EPS, raw revenue, most growth rates (YoY/QoQ), Piotroski sub-scores, Bollinger band width. Browse the full bullshit list →
⚠ Case study: the look-ahead bug that turned a Sharpe-3.37 winner into a -0.30 loser
During autonomous strategy research on May 3 2026, an in-house cross-asset trend-follower posted an apparent Sharpe of 3.37 over 21 years. After a mandatory honest audit, the entire result evaporated:
| Version | Sharpe | Sortino | CAGR | MaxDD |
|---|---|---|---|---|
| Originally claimed (BUGGED) | 3.37 | 5.59 | +28.5% | -4.3% |
| Fix 1: signal lagged 1 day | 0.53 | 0.69 | +3.8% | -22.4% |
| Fix 2: + 10 bps round-trip cost | 0.17 | 0.22 | +0.3% | – |
| Fix 3: + 5% annual borrow on lev>1 | 0.08 | 0.10 | +0.3% | -29.9% |
| Fix 4: pessimistic 20bps + 6% borrow | -0.30 | -0.39 | -2.5% | -44.5% |
The bug: the signal was computed at price[t] and then the strategy earned return[t] the same day — the strategy was already "in" the trade before the signal could possibly have been observed. After lagging the signal by 1 day so the strategy earns return[t+1] only, the edge disappears.
# BUGGY: in_trend[i] = spy_p[i] > spy_sma200[i] # uses TODAY's close position[i] = (spy_r[i], tlt_r[i]) # earns TODAY's return — leakage # CORRECT: in_trend[i] = spy_p[i-1] > spy_sma200[i] # uses YESTERDAY's close
Full forensics: reports/HONEST_AUDIT.md. The production website backtests at /strategies are not affected — they use a different engine that strictly enforces forward returns (audit: reports/PRODUCTION_BACKTEST_AUDIT.md).
✅ The 10-point honest-audit checklist
Every claim of a working strategy must clear all 10:
- Date-permutation test — shuffle the order of forward returns and recompute Sharpe. If it stays positive, you have leakage.
- Manual signal-vs-return alignment — print last 5 days of signal alongside prices. Confirm signal[t] uses prices[≤ t-1] and return[t] is forward.
- Cost model matches rebalance frequency — daily-rebal needs daily turnover cost, not per-flip.
- Borrow cost on leverage > 1 — explicit annual rate × fraction of time levered.
- Forward-return is explicit —
shift(-1)join, not implicit time alignment. - Sanity-check Sharpe > 2 — assume bug until proven otherwise.
- Random-baseline test — re-shuffle the SIGNAL keeping returns. Random-version Sharpe should be far below actual.
- Bootstrap CI lower-bound > 0.5 — block-bootstrap to preserve autocorrelation; lower bound > 0.5 = real edge.
- Out-of-sample lockbox — last 12-18 months never tuned on; one-shot evaluation.
- Survivorship + look-ahead in fundamentals — point-in-time membership; report-availability lag.
📊 How we decide what's predictive
The IC page labels each feature as REAL, WEAK, NOISE, or SUSPECT. The thresholds are industry-standard:
| Verdict | Threshold | Source |
|---|---|---|
| ✅ REAL | |IC IR| ≥ 0.5 ∧ |t-stat| ≥ 3 ∧ |mean IC| ≥ 2% ∧ hit rate ≥ 55% ∧ N dates ≥ 200 | Grinold's fundamental law of active management (IR > 0.5 = good, > 0.75 = great) |
| 🟡 WEAK | |IC IR| ≥ 0.2 ∧ |t-stat| ≥ 1.96 ∧ hit ≥ 52% ∧ N dates ≥ 100 | Statistically significant at 95% but borderline |
| ⚠ SUSPECT | |t-stat| > 5 with |IR| < 0.1 | Degenerate cross-section — signal is constant or near-constant |
| ⚪ NOISE | Doesn't clear weak threshold | Not predictive |
📚 The 70-mistake catalog (stuff that wrecks backtests)
Compiled from López de Prado, Bailey, Harvey-Liu-Zhu, Hou-Xue-Zhang, Jensen-Kelly-Pedersen, White, Hansen, Lo, Linnainmaa-Roberts, Bessembinder. Citations verified live.
- Same-day signal/return overlap (this caused our Sharpe 3.37 → 0.53)
- Restated fundamentals (using post-restatement values pre-restatement)
- Survivorship in benchmarks (current S&P 500 ≠ historical)
- Universe selection driven by current relevance
- Fundamental data lagged less than report-available date
- Cross-section ranking using future N
- Vol scaling with full-sample variance
- Calendar effects from futures-roll dates
- Z-score normalisation across full panel
- Probability of backtest overfitting (Bailey-LdP)
- Reporting only winning strategies (publication bias)
- Sub-period selection (cherry-picking 2009-2019)
- Hyperparameter optimisation without test set
- Feature selection in-sample
- Walking-forward without holding out final fold
- Strategy library survivorship
- Reality Check / SPA tests not applied
- Bid-ask spread ignored
- Volume-weighted impact (Almgren-Chriss)
- Borrow cost on shorts not modelled
- Tax drag (long-term vs short-term)
- Slippage at open/close auctions
- Commission tiers vs flat-rate
- Cash drag from rebalance windows
- Cost asymmetry by side (taker vs maker)
- Hyperparameter trial count vs deflated Sharpe
- Walk-forward parameter drift
- Curve-fit thresholds (e.g. RSI<30 specifically)
- Overlapping signals counted as independent
- Implicit factor exposure not controlled
- Train/test contamination through normalisation
- Naive standard errors (assume IID returns)
- Sharpe ratio bias from non-normal returns (Lo)
- Multiple-testing corrections not applied (Bonferroni / BH)
- Ignoring autocorrelation (Politis-Romano blocks)
- Treating skew/kurt as second-order
- Stationarity assumed without testing
- No probabilistic Sharpe ratio
- No deflated Sharpe ratio
- No PBO (probability of backtest overfit)
- No CPCV (combinatorial purged cross-validation)
- No paper-trading window before live
- No out-of-sample lockbox
- No random-baseline percentile
- No factor decomposition (Fama-French 5)
- No drawdown-relative-to-CAGR test (Calmar)
- No skew/kurt reporting
Full catalog with detection tests + prevention patterns + verified citations: ~/.claude/projects/-home-miche/memory/feedback_backtest_mistakes_catalog.md (70 items, 24 sources, 11 categories).
🧪 Even the "real" signals can be sector bets
A signal can have high IC because it accidentally aligns with sector membership. Financials skew small-cap, mega-tech skews large-cap — so the apparent "size effect" might just be "tech outperformed financials over 2018-2026." Test: subtract the per-(date, sector) mean from feature and forward return, then recompute IC. The retained ratio tells you what fraction is true cross-sectional alpha.
We ran this on our top REAL signals (May 3 2026 sweep). Only 4 of 6 retained >75% of their IC after sector-neutralising:
| Feature | Raw IC | SN IC | Retained | True alpha? |
|---|---|---|---|---|
mcap_pit (size) | -19.6% | -7.6% | 39% | Mostly sector bet |
sales_yield_pit (low P/S) | +14.3% | +12.8% | 89% | Yes |
earnings_yield_pit (low P/E) | −0.3% | +0.1% | 35% | Was already noise |
net_margin_pit (quality penalty) | −10.2% | −7.7% | 76% | Yes |
vol_252d | +19.7% | +17.4% | 88% | Yes |
dist_52w_low (uptrend) | +11.8% | +11.9% | 101% | Pure cross-section |
wq_alpha042 | +11.2% | +9.5% | 85% | Yes |
Lesson: the size effect (smaller-cap outperforms in our universe) is mostly because financial-sector stocks happen to be smaller and underperformed mega-cap tech in this regime. Don't bet on size as a stock-picking signal — it's a sector bet in disguise. Do bet on dist_52w_low, sales_yield_pit, and vol_252d, which retained their predictive power even after sector-neutralisation.
🔬 What our standard validation suite catches — and what it missed
A surprising finding from the May 3 audit: the standard validation suite (random-baseline, OOS lockbox, walk-forward, FF5 regression, bootstrap CI) all PASSED on the bugged strategy. Every metric looked clean because the leakage was identical across train, test, and random-permutation samples — they all measured the same biased stream.
What caught the bug: manual code review + date-permutation test on the input signal+price stream (not just the output return stream — that one is invariant under permutation since it preserves mean and std).
Action taken: built skills/honest_audit.py + saved a feedback memory mandating the audit checklist before any backtest result is reported.