Overfitting means a model exploits features of its development data that do not reliably repeat on new data. Searching many variants increases the selection problem.
Count attempts, not just winners
Every added filter or parameter choice is another opportunity to capture historical accidents. Record rejected variants and reasons for changes. Without that history, it is difficult to assess how heavily a result was selected after the fact.
Examine a region, not just one point
One outstanding parameter set among weak neighbors warrants investigation. Check whether small, justifiable changes reverse the behavior. This is sensitivity analysis, not proof of future stability.
Bound the research question
Define the hypothesis and rejection criteria before a run. Adding markets or periods must not make poor results disappear from the report. A failed experiment can remain valuable as documented evidence.
A practical example
Thought experiment: test 200 filter combinations and present only the best. Readers may see one planned experiment. A complete log discloses all 200 attempts, explains selection and then separates new evaluation data.
For your next test
- Write down hypotheses and rejection criteria first.
- Disclose number of attempts and parameter ranges.
- Do not repeatedly use untouched data for selection.
Common question
Does a successful forward test prove robustness?
No. It adds evidence but remains a limited observation under particular market conditions.
Sources & further reading
Educational content, not investment advice. Numerical examples are hypothetical, not results of a REVENQOR system.
Continue learning
What makes a backtest useful? →
Lookahead bias: when a test knows the future →
Explore Systems Lab ↗