Walk-forward vs. out-of-sample
Was the result produced by rules fixed before the evaluated data was observed?
A holdout protects one untouched period. Walk-forward analysis repeats training and evaluation through time. A forward record timestamps a live declaration before its outcome. They are not interchangeable.
THE PRESSURE TEST
Move the convenient assumptions.
Ask what was fixed, when it was fixed, and which observations were allowed to influence the next decision. The procedure matters more than the label.
- Identify the exact boundary between development and evaluation data.
- Record every retraining or parameter-selection rule before the next window.
- Keep failed windows and abandoned variants in the evidence record.
- Separate simulated out-of-sample results from timestamped forward outcomes.
Calling the last year 'out of sample' is weak protection if that year was inspected repeatedly while the strategy changed. A smaller untouched window can carry more evidence than a larger window that quietly influenced development.
The check stops here.
A closed-trade export cannot reconstruct the research process or prove that a holdout stayed untouched. Verify can test the submitted record and seal future declarations; it cannot manufacture missing development history.
SEE THE CALCULATION, NOT A CLAIM