• Australian (ASX) Stock Market Forum

Ideal backtesting

Discussion in 'Trading Strategies/Systems' started by jjbinks, Sep 25, 2016.

  1. jjbinks

    jjbinks Well-Known Member

    Posts:
    54
    Likes Received:
    4
    Joined:
    May 14, 2013
    Hi guys,

    I have recently got into trying to develop a mechanical system for trading in Amibroker.

    I haved coded the system and am optimising parameters but was wondering what the best way to do out of sample testing is.

    Curently I am developing my code on data from 2000-2010. Would it be adequate to then just test 2010-2016 results to see if the results are comparable. Or is there a better way to do this?

    Thank you and much appreciated
     
  2. captain black

    captain black Well-Known Member

    Posts:
    997
    Likes Received:
    55
    Joined:
    Oct 24, 2005
    The walk forward timeframe you use for testing the validity or stability of a system is also one of the parameters that can be tested. Using the stepped walk forward in Amibroker you could do a 1 year in-sample / 1 year out-of-sample walk forward test from 2000 up to 2016. You'll be able to plot an IS and OOS comparison of the equity curves. Perform a series of IS and OOS walk forward tests over varying IS and OOS periods and see how much variation there is between each IS and OOS equity run. If the variation of the runs is high then there's a higher probability of the system being unstable.

    The length(s) of your IS and OOS periods should be determined by the frequency of trades. For a short term system you would use different testing periods than you would for a longer term trend following system for example.
     
  3. jjbinks

    jjbinks Well-Known Member

    Posts:
    54
    Likes Received:
    4
    Joined:
    May 14, 2013
    Is there a downside in simply setting aside data to manually test your strategy.

    e.g. i haven't used any data from 2010-2016 in testing my current strategy. So wants I finish tweaking it with data from 2000-2010 i will test it from the 2010-2016 to see if results are acceptable?
     
  4. captain black

    captain black Well-Known Member

    Posts:
    997
    Likes Received:
    55
    Joined:
    Oct 24, 2005
    That's one way to approach the robustness of your system. Going forward with a strategy I like to know what the best parameters for measuring the health of the system are. One of the ways to measure going forward (ie. once your live trading it) is to measure your performance once again by how far the variation between previous IS and OOS tests have been compared to the variation of IS testing and your live results.

    For example, if your walk forward analysis has shown that using a 1 year step forward test shows that to be the optimal walk forward period for testing then continuing to measure your systems performance against the previously tested variations will show whether the system is still tracking it's expected performance or whether it's "broken".

    I have a short term momentum system I trade on the ASX that I use a 3 month reassessment period to test it's robustness against historical variances in IS and OOS samples. My longer term trend following systems (daily and weekly) are reassessed annually.

    The downside of doing just a single walk forward run ( 2010 - 2016) is that it gives you very little statistical information going forward to use to measure the health of your system.
     
  5. jjbinks

    jjbinks Well-Known Member

    Posts:
    54
    Likes Received:
    4
    Joined:
    May 14, 2013
    thanks for your help so far.

    I'm just having trouble making sense of the walk forwards results.

    I have set it to optimise over 4 years and then do OOS testing in the following year.

    Do decide on whether the system is worth using based on how many OOS years are profitable?

    In terms of parameters, if I do decide to go ahead with the system then would I work out the parameters from the last 4 years uptil now?


    Thanks again
     
  6. captain black

    captain black Well-Known Member

    Posts:
    997
    Likes Received:
    55
    Joined:
    Oct 24, 2005
    The decisions I make about how robust the system is likely to be going forward are based on a number of different types of statistical analysis, but one of those is the variations between my IS and OOS sample results. I measure the spread of the results using variance and standard deviation and compare these values historically to how my system is performing going forward.
     
  7. captain black

    captain black Well-Known Member

    Posts:
    997
    Likes Received:
    55
    Joined:
    Oct 24, 2005
    Apologies, missed this question.

    Once you've established the optimal IS and OOS time period then the parameters to use going forward are from your most recent IS test. A walk forward test is simply a representation of what would happen in real trading.
     
  8. captain black

    captain black Well-Known Member

    Posts:
    997
    Likes Received:
    55
    Joined:
    Oct 24, 2005
    I'll also add that for IS testing I prefer to use the CMAE optimization engine rather than an exhaustive search. It tends to find broad plateaus in its results rather than singular "best result" data points.

    My early testing on systems when I first started using Amibroker was with Particle Swarm Optimisation using Fred Tonetti's IO addon. A PSO optimisation engine is also available to use natively in Amibroker now but I've found the CMAE to be my favourite weapon of choice :)
     
  9. Wysiwyg

    Wysiwyg Everyone wants money

    Posts:
    7,091
    Likes Received:
    7
    Joined:
    Aug 8, 2006
    To me, any period not in the system build is OOS. I test if the strategy is over optimised by doing manual runs (no parameter optimising as in the built in walk forward)) over any period outside the test period and look at what the Index was doing in the periods of drawdown. The drag on long only stock systems by a range bound or downtrending Index is frustrating. Obviously stocks rise in these sub optimal periods but your code still has to pick them.
    However I am cynical of the backtest scene. A lot of fancy tricks for a rough idea of how things might go if the market conditions suit the system from the present moving forward.

    This link explains what I mean.

    http://www.futuresmag.com/2015/09/01/dont-build-better-backtest
     
  10. captain black

    captain black Well-Known Member

    Posts:
    997
    Likes Received:
    55
    Joined:
    Oct 24, 2005
    Data mining and curve fitting are certainly 2 important factors to consider in the health of a system. That gets back to my point earlier to the OP about having a statistically significant number of data points in the IS and OOS testing. If the variance between the IS and OOS data is increasing over time then you need to look at whether the system is being curve-fitted or it's broken or a number of other factors. A single WF run wont provide that data.

    The next "WF" run is using real money, I'd rather have confidence in a system's health by using statistical techniques than have it shown to me using real money. :2twocents
     
  11. Boggo

    Boggo Well-Known Member

    Posts:
    2,813
    Likes Received:
    9
    Joined:
    Mar 28, 2006
  12. captain black

    captain black Well-Known Member

    Posts:
    997
    Likes Received:
    55
    Joined:
    Oct 24, 2005
    Nice find Boggo, I've got most of Howard Bandy's books related to system design and statistical validation but it's always good to see it from another perspective.

    I notice the charts you post in other threads are in Amibroker now rather than Metastock, hope the transition went smoothly :)

    I liked this quote from the Amazon link to the book:

     
  13. Gringotts Bank

    Gringotts Bank Well-Known Member

    Posts:
    4,796
    Likes Received:
    7
    Joined:
    Jun 30, 2007
    I trust the process, but with equities I found I massively underestimated the effect of slippage.

    An equity backtest which returns 100%pa (brokerage included but not slippage) will return about 30%, in real life trading - that's my experience anyway. This is with a liquidity filter included.
     
  14. Boggo

    Boggo Well-Known Member

    Posts:
    2,813
    Likes Received:
    9
    Joined:
    Mar 28, 2006
    I've got Bandy's books too but in some areas they get a bit too scientific for me but that is me rather than the books.
    Bandy's 'Foundations of Trading' is a nice concise summary of his work though and in some ways is similiar to Davey's book.

    Took me a while to let go of Metastock. I spent many late nights learning how to get the best from it and was reluctant to give it up. With some prompting from many people I went with Amibroker and am sorry now that I didn't do it years ago.
    Just the backtesting and optimisation alone leaves Metastock for dead.

    Cheers
     
  15. captain black

    captain black Well-Known Member

    Posts:
    997
    Likes Received:
    55
    Joined:
    Oct 24, 2005
    I'm still working my way through it but from what I've read so far I agree that it would be a great read for someone who's looking for a concise summary of statistical analysis/modeling of trading systems. I promised Howard a review of the book so I'd better finish it soon!

    Yeah, I started out with EzyCharts in the beginning and when I was looking for something better I'm forever thankful I went down the Amibroker path.
     
  16. Roller_1

    Roller_1 Well-Known Member

    Posts:
    163
    Likes Received:
    1
    Joined:
    Sep 13, 2013
    What sort of filters were you using??
     
  17. Gringotts Bank

    Gringotts Bank Well-Known Member

    Posts:
    4,796
    Likes Received:
    7
    Joined:
    Jun 30, 2007
    They vary a bit, but say I want to take a $5k position, I'd want daily turnover of at least $300k. $10k->$500k and so on. I know the ratio is too big, and that's the reason for the slippage. As you'd be aware, a stock with a daily turnover of 500k can easily turnover 50k the very next day with a huge widening of the spread thrown in for good measure, so... that's the risk you take trading outside the top 20.
     
  18. Roller_1

    Roller_1 Well-Known Member

    Posts:
    163
    Likes Received:
    1
    Joined:
    Sep 13, 2013
    You could limit slippage by using the opening auction for buy/sells or just limit orders. I have a few partial fills but nothing too drastic, using similar filters
     
  19. jjbinks

    jjbinks Well-Known Member

    Posts:
    54
    Likes Received:
    4
    Joined:
    May 14, 2013
    thanks guys.
    Definitely cleared up some confusion I have.

    Shall continue my search for a good strategy!
     
  20. Wysiwyg

    Wysiwyg Everyone wants money

    Posts:
    7,091
    Likes Received:
    7
    Joined:
    Aug 8, 2006
    A simple way to check the usefulness of a an entry signal is to create a shape for the signal on the chart, set your chart back several periods and see what happens after each signal. Saves running unnecessary backtests by quickly showing the ensuing behaviour.
     
    Last edited: Dec 25, 2016
Loading...

Share This Page