Is there a point to backtesting and if so for how long?
This is my question. It may seem like a silly question, but i'll explain how i've come to this dilemma.
I've been backtesting automated strategies for about a year now on ES (emini s&p 500) 1 minute daily data. I'm using ninja trader, which is good for me as it uses c#.net as it's programming environment, which i've had a lots of experience with so i've been able to test hundreds if not thousands of approaches in the time....
At the start i began just using a few years worth of data, however what i found was that i was able to optimise a particular strategy for that time frame, however when i let it run over the full 10 years worth of ES data the result would be overall negative over the time frame.
Intuitively, I put this down to the problem of overfitting the test data - ie: the algorithm learns how to run on the training set data, but is not good for data unseen (in this case over the different timeframes).
I then began trying to develop algorithms that would work over the entire 10 years of the data.
Recently, i contacted a developer of a "system" which i'll let remain nameless, which i backtested and found that it didn't seem to work. After contacting him I got a bunch of fairly abrupt responses, but his view was mainly summarised by the following:
"First of all going back to 1998 is really not the proper way to test any system. its anti productive. Markets were different, volatility was different and you will never find anything that will work into the future based on a great past track history,"
This seems interesting to me. I would like to know what everyone thinks about this.
My thoughts are :
1) Backtesting is productive, allowing for development and refinement of algorithms based on real data
2) I agree that markets are different in the past, however I was trying to work towards a strategy that would work accross these changing markets, for exmple, using volatility... - is this not a good approach to aim for?
So in essence i think backtesting is valuable, but i'm now questioning whether it's a waste of time to find an approach that works on old historical data that maybe is useless..... If backtesting is good to do, what timeframe should be used - ie: just use say the last 2 years worth of ES data??