Method

How we test things

Backtesting is easy to do and very easy to fool yourself with. These six rules exist because each of them is a way people routinely arrive at a confident wrong answer.

1. Register the hypothesis before touching the data

What counts as success — the gate, the horizon, the statistic — is fixed in writing before any result is visible. A test whose bar is set afterwards is not a test.

2. Compute achievable power first, then set the bar

Some questions cannot be answered by the amount of history that exists. Working out what is detectable before choosing a threshold stops us from setting bars that decades of data could never clear, and from believing results that clear bars set too low.

3. One attempt, and no rescues

A candidate gets a single pre-registered run. If it fails, it is retired — no parameter tuning, no shifted horizon, no quietly better variant. Most published strategy research is the survivor of an unreported search; this is the rule that prevents that.

4. Adversarial review by someone who did not do the work

Every material result is challenged by a separate process that receives a frozen evidence bundle and is asked to refute it. The author may answer questions but cannot direct the outcome.

5. Preserve the failures

Negative results are kept and published rather than quietly dropped. A research record that contains only successes is not a record.

6. Say what is not known

Where evidence is missing or a rate is unverified, the honest output is a refusal rather than an estimate. A confident wrong number is indistinguishable from a right one to the person reading it.

The cost of doing this

Testing this way rejects almost everything. That is the intended behaviour — the base rate for active investment strategies genuinely is close to zero, and a method that approved things at a comfortable rate would be measuring its own leniency rather than the strategies.

It also means the honest conclusion is often unexciting: hold a broadly diversified portfolio, rebalance on a rule, and be sceptical of anything promising more. We would rather publish that than a more interesting claim we cannot support.