I am currently developing my first mechanical trading system with AmiBroker and in the process of testing. I am coming to realise, that I am likely to spend more effort testing the system than I have building it.
I am at the stage of wanting to do some walk forward testing on the system and currently pretty green in this area. I have been reading up on WF testing, have read Howard's book, Quant. Trading, ordered Robert Pardo's book The Evaluation and Optimization of Trading Strategies, and also ordered Howard's other book Modeling Trading System Performance + also been reading a lot of forum material.
While I have defined an objective function, one thing that struck me tonight is ideally the objective function should incorporate a value that represents the selected parameter set's stability level.
Everywhere I have read, there is a consistent message that you should use a set of parameters that sits on a plateau rather than a single sharp peak. This makes perfect sense (I think). However, if you are doing an automated walk forward analysis in AmiBroker, how can this be done? and is what I am proposing make sense? I have 5 parameters in my system and a relatively small space ~1.2Million, so I am using the covariance matrix adaptation evolution strategy. Any thoughts on finding plateaus and stable parameters?