Backtesting.

Every trading strategy sounds plausible when someone tells it to you. "Buy when the price bounces off the 0.618 Fibonacci level." "Enter when RSI drops below 30." "Follow the smart money." The problem is that a plausible story and a money-making strategy are two different things. Backtesting is the bridge — or the chasm — between them.

This article explains what backtesting is, why it matters for retail investors on the Indonesia Stock Exchange (IDX), and — the part most often overlooked — how backtests can lie, and how to read them honestly.

The big picture: a backtest is an experiment, not a prophecy

A backtest, at its simplest, means taking a buy-sell rule, running it against historical price data, and calculating what would have happened had that rule been followed exactly. The result is a number — return, maximum drawdown, win rate — not an opinion.

This is what separates an evidence-based approach from market pseudoscience. A pattern that "always fits after the fact" can never be honestly tested, because there's always a new explanation for every outcome. A proper backtest forces you to commit to a rule before seeing the results, and then accept whatever comes out. It can prove your favorite strategy is actually useless — and that's a feature, not a bug.

But there's one uncomfortable truth to hold onto from the start:

A good backtest does not guarantee a good future.

Markets are non-stationary and reflexive — the rules of the game change, and the more people who trade on the same signal, the thinner its edge becomes. Even the most honest backtest only gives you a distribution of possibilities, not a certainty.

Why IDX retail investors should care

Without backtesting, you're testing a strategy with real money in real time — the most expensive way to learn there is. Backtesting gives you three things:

• A fast filter. Many strategies that sound great fail the moment they're tested against data. Better to know now than after a loss.
• Realistic expectations. A backtest shows how deep and how long losses can run, so you don't panic at the worst possible moment.
• Discipline. A tested rule is easier to stick to than a hunch, especially when the market is pressing on your emotions.

Reading backtest results honestly

A headline "300% profit" number means nothing without context. Here's what actually matters:

• Return vs. benchmark. Compare it against simply buying and holding the IHSG (the Jakarta Composite Index). Beating the market is far harder than merely making money.
• Maximum drawdown. The deepest decline from peak to trough. A strategy that returns 40% but once fell 60% along the way will make most people quit before the gains arrive.
• Sample size (number of trades). A strategy with only 8 trades over 5 years doesn't have enough evidence behind it. Luck needs very little data; statistics need a lot.
• Consistency across periods. Is the profit spread across many years, or did it all come from one lucky break in 2020?
• Risk-adjusted return (e.g., Sharpe ratio). Return per unit of risk (Sharpe, 1994). Two strategies with the same return but different volatility are not equivalent.

Seven traps that make a backtest lie

This is where most backtests — including the ones used to sell "miracle systems" — fall apart.

1. Overfitting (a forced-fit curve). If you tweak parameters until the results look perfect in the past, you haven't found a strategy — you've memorized history. A strategy with 10 parameters that all happen to line up almost always fails on new data. Bailey et al. (2014) even showed that impressive simulated performance is easy to achieve just by trying a small number of configurations — and the more configurations you try, the greater the chance the result is illusory.

2. Look-ahead bias. Using information that wasn't actually available at the moment the decision was made. A classic example: using that day's closing price to "decide" to buy that same morning. In the real world, you don't yet know the closing price.

3. Survivorship bias. Testing only on stocks that still exist today, while ignoring issuers that were delisted or went bankrupt. This makes results look far rosier than reality, because the failures have already been "erased" from the data — an effect strong enough to create an illusion of consistency (Brown et al., 1992).

4. Data snooping / p-hacking. Trying 200 strategies, then showing off the single best one as if it were a discovery. Statistically, out of 200 random attempts, something is bound to look brilliant — purely by chance. Sullivan et al. (1999) demonstrated this while testing thousands of technical rules at once; Harvey et al. (2016) found a similar problem among hundreds of "factors" claimed to predict returns — most fail to hold up once corrected for multiple testing.

5. Ignoring real IDX costs. This is the silent killer. Broker buy/sell fees, sales tax, the bid-ask spread, and slippage all erode returns. A strategy that trades frequently can look profitable on paper but lose money after costs.

6. Ignoring liquidity and market mechanics. A backtest often assumes you can buy or sell any amount at any price. On the IDX, second- and third-tier stocks can be thin — a large order moves the price by itself. Then there's ARA/ARB (upper/lower auto-rejection): when your signal says to buy on a day the stock is locked at its upper limit, in reality you don't get filled at all.

7. Regime change (non-stationarity). A strategy that wins in a trending market can be destroyed in a sideways one. The data from 2014–2019, the 2020 crisis, and the post-2021 period behave almost like three different markets. A backtest on one regime is no guarantee for the next. This is the core of the Adaptive Markets Hypothesis: a strategy's efficiency and edge evolve as conditions and market participants' behavior change (Lo, 2004).

How to backtest correctly

If the traps above sound alarming, here's the good news: all of them can be avoided with discipline.

• Split your training and test data (out-of-sample). Design the strategy on only part of the data, then test it on a portion you've never seen. If the results collapse on the test data, your strategy was only memorizing the past (Bailey et al., 2014).
• Walk-forward testing. Test it on a rolling basis: train on one time window, test on the next, shift forward, repeat. This mimics how a strategy is actually used over time (López de Prado, 2018).
• Include real costs. Buy/sell fees, tax, and a conservative slippage assumption. If a strategy is only profitable without costs, it isn't profitable.
• Test for robustness. A good strategy isn't fragile to small changes in its parameters. If nudging one number slightly turns a profit into a loss, that's a sign of overfitting.
• Quantify uncertainty. Instead of a single return figure, think in ranges: a poor scenario (P10), a middle one (P50), and a good one (P90). A mature investor plans for P10, rather than hoping for P90. This way of thinking — a distribution, not a single point — is at the core of what separates serious analysis from a sweet promise.

A checklist before you trust a backtest

Before putting money behind any strategy — your own or one someone else is selling — ask:

1. Was the rule defined before seeing the results, or fitted afterward?
2. Was it tested on data not used to design it?
3. Have costs, tax, slippage, and liquidity been accounted for?
4. How many trades — enough to be statistically meaningful?
5. Is the profit consistent across years, or does it depend on one event?
6. How deep is the drawdown, and can you actually stomach it emotionally?
7. Does the presenter show the failures too, or only the best result?

If a "system" can't answer these questions openly, it isn't a tested strategy — it's a story being sold.

Closing

Backtesting is neither a crystal ball nor a scam — it's a discipline. Used honestly, it tells you whether a strategy would have survived real market conditions, at what cost, and with what risk. Used dishonestly (tuned after the fact, stripped of costs, cherry-picked periods), it tells you nothing except what you wanted to hear.

This is the standard we hold ourselves to at Sobat Investor: state the methodology plainly, disclose the traps, and report uncertainty as it is — not as we wish it were.

References & Further Reading

This article is educational and does not constitute investment advice. Investment decisions and their risks are entirely your own responsibility. Past performance does not guarantee future results.