Curve fitting theory and data samples - Aussie Stock Forums

Results 1 to 7 of 7
  1. #1

    Default Curve fitting theory and data samples

    Hey All,

    I finished my third live trading system about a year ago and have been trading it with some success for the past six months. However, when I first starting designing and testing it I was still learning the ropes and as such, I ignored in-sample and out-of-sample data periods. So in essence, the system was tested over the entire data set available. Iíve just read Howard Bandyís Quantitative Trading Systems and as a result Iíve lost the nerve to trade my current system. To be honest, I was using discretion to pick between trades and sometimes ignoring trades if they looked too volatile anyway so perhaps my confidence in the code wasnít 100% in the first place. My question however, is this: are there situations where itís acceptable to not have in and out-of sample periods.

    I canít see how the system Iím trading can be curve-fit and I was hoping somebody could enlighten me. I say this because of the following:
    • The entry code is relatively simple; there are no complex functions or indicators used and there are relatively few actual entry conditions
    • The system produces a large number of trades (about 400 a year, 4,000 over the last ten)
    • The holding period is about 2 weeks per trade
    • It returns a (genuine, I think) 35% per year and is VERY realistic (overly harsh slippage on entry and exit, accurately modeled brokerage, very restrictive volume and liquidity filters, no forward looking etc)
    • The equity curve is consistent in slope over the past decade, with no major draw downs (11% ish max DD) or volatility (this is partly due to the large number of trades; I can limit position sizes and risk and I think this has a major effect on the overall DD)

    I donít see how it can be dangerous to trade this system and Iíd like other peopleís opinions (especially yours Mr Bandy if possible). I understand I can wait six months and walk-forward test or perhaps alter the conditions a bit and see if thereís a major change in results, but Iím more interested in the theory of it. If something produces a consistent profit over a sufficiently large number of trades and your equity curve is smooth, I donít see how it can be curve fit.

    To me it seems like if the frequency of trades is high enough over an extended period of time, there wouldnít be a single static piece of code that would fit that data set well enough to produce consistent performance.

    Am I wrong?

    By the way, I definitely recommend the book. Nice to see somebody apply some analytical rigor to an industry/genre that seems to be full of people spouting unfounded BS.

    Any comments appreciated.

  2. #2
    skc's Avatar
    Join Date
    Aug 2008
    The dark side
    Blog Entries

    Default Re: Curve fitting theory and data samples

    Not a systems expert but a couple of thoughts come to mind.

    How many variables to you have? If you have 1 or 2 then the possibility of curve fit is greatly reduced. If you have 5 to 8 then watch out.

    Can you get some data on other markets to test? E.g. US/UK markets, futures etc?

    Can you articulate in words, what the system is doing? i.e. what is the edge if you want to tell somebody what your system does? Or is it just a forumla that happened to work?

  3. #3

    Default Re: Curve fitting theory and data samples

    I've never tested any of my systems with out of sample data from the same bourse for the same reason you did.
    I also developed most of mine well before I read about out of sample.
    Since then all have been tested walk forward in R/T and my first 8 yrs ago has 8 yrs of data to look at that it wasnt tested on.

    My comments are
    (1) Try other bourses with similar stock types.
    (2) Wouldn't worry about it if your live trading is within the limits of your test results.
    (3) Even testing out of sample data isn't fail safe.All you need to have happen is the data your trading going forward is different enough to the in and out of sample data to cause the system to trade outside of its blue print!

    Have you Montecarlo tested the systems to get a better idea of the upper lower and median values of your systems---that way you can judge a little clearer how the system is performing live.

  4. #4

    Default Re: Curve fitting theory and data samples

    Just alter your entry parameters slightly and see what happens to the equity curve. If it falls apart, it's not robust.

    I believe OOS testing is vital if you're using any indicators as entry signals, particularly if they have been optimized. But I doubt you're doing that. If you're using more 'organic' entry signals, and your equity curve is very even and smooth, I agree OOS is not necessary.

    You do realise that you need not wait another 6 months? Just use the data you already have. OOS is an optimization process, so you can just ask your software to test 6 months, then OOS test 6 months on whatever data you already have.

  5. #5

    Default Re: Curve fitting theory and data samples

    Even easier, just take all your entry parameters and run an optimization on all of them together. You'll quickly see how parameter tweaking affects profitability, DD, SD etc.

  6. #6

    Default Re: Curve fitting theory and data samples

    I've seen systems with and without OOS testing do both well and fail. Bottom line watch the equity curve and when it does not meet your requirements optimize your system. That sequence works quite well.

  7. #7

    Default Re: Curve fitting theory and data samples

    Thanks for the replies guys.

    Skc, the system is essentially a trend follower but looks for a specific type of trend behaviour. The codeís made up of 4 filters (volume and index based) and 4 similar variables (again, fairly straight-forward code).

    Iím aware of ways to get around the problem, Iím just after peopleís opinions on how important you think it is during system design. A lot of the more technical stuff Iíve read on systems design implies itís an absolute necessity for all types of systems design, but this doesnít fit with what Iíd expect.

    I can understand it if your trying to trade a 20% win rate, 14 avg R/R system with a trade log of 200 trades Ė obviously small changes in the code that result in the trades changing could have massive effects on the outcome. In addition, the more restrictive/exhaustive the code (and the lower the trade frequency), the easier it is to find a single bit of code that will fit the data set very well.

    But again, I think the relevance of OOS testing would be directly related to the number of trades in the log, the type of system youíre trading, the win rate and RR, and the smoothness of your equity curve Ė it doesnít seem as clear cut as ďyou must always use OOS testing regardless of the situationĒ.

    Tech, the systemís been Monte Carlo tested. In live testing itís been trading well within the bounds expected. The main problem Iím having is that it generally buys as a counter-intuitive (read: scary) point in a trend Ė and this means you need to have a lot of confidence in it. Iím going to do some more testing over the coming weeks on different markets and might play about with the parameters to see the effect.

    Hey GB, I realize I donít have to wait six months but because Iíve tweeked the thing based on current data, Iíd need six(?) months of fresh data to consider it OOS testing.

    Thanks for the input guys. Iíll let you know how I go.

Similar Threads

  1. Charting Equity Curve in Xcel with IB Import
    By chops_a_must in forum Brokers
    Replies: 7
    Last Post: 26th-January-2010, 09:19 PM
  2. Inverted yield curve
    By jj1929 in forum International Markets
    Replies: 7
    Last Post: 7th-May-2008, 10:11 PM
  3. Trading the Equity Curve - Amibroker
    By Sir Burr in forum Software and Data
    Replies: 7
    Last Post: 4th-May-2008, 12:38 PM
  4. Inverted Yield Curve... Again
    By wayneL in forum International Markets
    Replies: 9
    Last Post: 26th-January-2007, 12:42 PM
  5. What is the learning curve on Stockval software?
    By ggumpshots in forum Software and Data
    Replies: 1
    Last Post: 5th-September-2006, 10:45 AM


Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
Aussie Stock Forums