Is backtesting a system reliable?

I've notice that people say that backtesting a system does not give very reliable output and it is better to paper trade a system for a while. Why is this?

The backtesting its self is reliable. The numbers produced are as correct as the data you use to simulate the results and the coding used to perform the backtest. There is likely to be some slippage (where you can't actually buy/sell at the exact price you want to when trading live) but usually this would be minimal.

Whether the backtest provides simliar results compared to trading live will depend on the quality of the system design. If a system is poorly designed or over optimised, past results may be fantasic, but when you trade it, may fail hopelessly. This is why paper trading is important... To confirm or reject a back tested idea.

Paper trading, or forward testing provides real results, and shows how the system actually trades live. But the backtesting is vital to get to that stage. Without it you'd be stabbing at thin air.

Its due to outliers and inversion economics.

Once you master those backtesting makes sense.

gg

whipflip15 said:

I've notice that people say that backtesting a system does not give very reliable output and it is better to paper trade a system for a while. Why is this?

Click to expand...

I don't mind helping you with your questions, I trade over 4 systems using completely backtested data over 20 years on futures and stocks on the ASX and the US, and international commodities and index / fx - futures.

Are you able to quote the references, via a url, where people disagree with this type of method ?

Systems will trade within their blueprint provided the system is trading in a period which is not far enough outside of the data set which the testing was conducted to create a trading environment foreign to it.

weird said:

Are you able to quote the references, via a url, where people disagree with this type of method ?

Click to expand...

I don't remember any particular instances where I read it, but if I read it again I'll take note.

tech/a said:

Systems will trade within their blueprint provided the system is trading in a period which is not far enough outside of the data set which the testing was conducted to create a trading environment foreign to it.

Click to expand...

Are you saying that if the backtest is over a period with a similar market environment then the system will obtain those results when traded?

whipflip15 said:

I've notice that people say that backtesting a system does not give very reliable output and it is better to paper trade a system for a while. Why is this?

Click to expand...

As was stated earlier, back testing output is relative to data input.

That being for EOD (end of day) data -- High, Low, Open, Close and Total Traded Volume. Obviously the 4 prices need suitable volume for the back test to have a greater reliability, i.e. your order quantity was available at those prices.

So a back test buys/sells on the Open, High, Low or Close price data without an inkling of what volume was available at these prices. To overcome the volume at price issue, stocks can be screened for high turnover and the assumption made that a transaction would have been possible.

A feature that Amibroker has is a setting to buy only a certain percentage of the traded bar volume, i.e. 5%.

I found back testing useful for gaining a greater understanding of indicators and market gyrations but as far as back testing trading systems go, I gave up chasing my tail. Market experience is the best back tested trading system in my opinion. (that is if we learn the lessons )

Just look at the thousands of systems, expert advisors, signals, recommendations, designing software etc. that people try to sell. Plenty on the treadmill fuelling an industry within itself.

whipflip15 said:

Are you saying that if the backtest is over a period with a similar market environment then the system will obtain those results when traded?

Click to expand...

Similar yes.