New posts

Reply to thread

Message

Hi Nizar --

The question is how long to make the in-sample period.

The markets we are modeling are very dynamic, are non-stationary, and are changed by every profitable trade that is made. What works keeps changing, and what used to work will never work again.

The choice of how long to make the in-sample period is often tricky. Begin by ignoring advice that by using a very long period the system will be able to recognize more possible conditions. Very long in-sample periods result in systems that are unable to recognize anything. Yes, you want your system to be able to recognize that a bear market has started and stop taking long trades. But including data back to 1980 so that it can "see" October 1987 will not help.

Think about applying a standard moving average to a set of data -- add up the values and divide by the count. The resulting number represents the average over the entire period and has a lag of one-half the number of data points. If a trading system is based on moving averages, the longer the lag is, the later the signal will be. If a moving average is being fit to a long data series, it will not fit any of it very closely -- if it is being fit to a short data series, the fit will be better.

So, my advice is to make the in-sample period as short as possible, consistent with producing good out-of-sample results. Too long an in-sample period and the system will not perform well over any part of it. Too short an in-sample period and the system will be curve-fit to the in-sample data and will not perform well out-of-sample.

I am assuming that you are developing the model using a walk-forward process, so that you have more than one in-sample period. If there is just one in-sample period, then there is just one out-of-sample period. You will need to find additional out-of-sample data to validate the system. You may be more disciplined than I am, but I cannot resist changing my models after I have seen the out-of-sample results. It is OK to do that, provided you realize that you just transformed the out-of-sample data into in-sample data. And in-sample results have no value in terms of predicting the out-of-sample performance.

Experiment until you find out what works.

Thanks for listening,
Howard
www.quantitativetradingsystems.com

<blockquote data-quote="howardbandy" data-source="post: 195810" data-attributes="member: 9179">Hi Nizar --The question is how long to make the in-sample period.The markets we are modeling are very dynamic, are non-stationary, and are changed by every profitable trade that is made.&nbsp; What works keeps changing, and what used to work will never work again.&nbsp; The choice of how long to make the in-sample period is often tricky.&nbsp; Begin by ignoring advice that by using a very long period the system will be able to recognize more possible conditions.&nbsp; Very long in-sample periods result in systems that are unable to recognize anything.&nbsp; Yes, you want your system to be able to recognize that a bear market has started and stop taking long trades.&nbsp; But including data back to 1980 so that it can &quot;see&quot; October 1987 will not help.Think about applying a standard moving average to a set of data -- add up the values and divide by the count.&nbsp; The resulting number represents the average over the entire period and has a lag of one-half the number of data points.&nbsp; If a trading system is based on moving averages, the longer the lag is, the later the signal will be.&nbsp; If a moving average is being fit to a long data series, it will not fit any of it very closely -- if it is being fit to a short data series, the fit will be better.So, my advice is to make the in-sample period as short as possible, consistent with producing good out-of-sample results.&nbsp; Too long an in-sample period and the system will not perform well over any part of it.&nbsp; Too short an in-sample period and the system will be curve-fit to the in-sample data and will not perform well out-of-sample.&nbsp; I am assuming that you are developing the model using a walk-forward process, so that you have more than one in-sample period.&nbsp; If there is just one in-sample period, then there is just one out-of-sample period.&nbsp; You will need to find additional out-of-sample data to validate the system.&nbsp; You may be more disciplined than I am, but I cannot resist changing my models after I have seen the out-of-sample results.&nbsp; It is OK to do that, provided you realize that you just transformed the out-of-sample data into in-sample data.&nbsp; And in-sample results have no value in terms of predicting the out-of-sample performance.Experiment until you find out what works.Thanks for listening,Howard<a href="http://www.quantitativetradingsystems.com" target="_blank">www.quantitativetradingsystems.com</a></blockquote>

[QUOTE="howardbandy, post: 195810, member: 9179"] Hi Nizar -- The question is how long to make the in-sample period. The markets we are modeling are very dynamic, are non-stationary, and are changed by every profitable trade that is made. What works keeps changing, and what used to work will never work again. The choice of how long to make the in-sample period is often tricky. Begin by ignoring advice that by using a very long period the system will be able to recognize more possible conditions. Very long in-sample periods result in systems that are unable to recognize anything. Yes, you want your system to be able to recognize that a bear market has started and stop taking long trades. But including data back to 1980 so that it can "see" October 1987 will not help. Think about applying a standard moving average to a set of data -- add up the values and divide by the count. The resulting number represents the average over the entire period and has a lag of one-half the number of data points. If a trading system is based on moving averages, the longer the lag is, the later the signal will be. If a moving average is being fit to a long data series, it will not fit any of it very closely -- if it is being fit to a short data series, the fit will be better. So, my advice is to make the in-sample period as short as possible, consistent with producing good out-of-sample results. Too long an in-sample period and the system will not perform well over any part of it. Too short an in-sample period and the system will be curve-fit to the in-sample data and will not perform well out-of-sample. I am assuming that you are developing the model using a walk-forward process, so that you have more than one in-sample period. If there is just one in-sample period, then there is just one out-of-sample period. You will need to find additional out-of-sample data to validate the system. You may be more disciplined than I am, but I cannot resist changing my models after I have seen the out-of-sample results. It is OK to do that, provided you realize that you just transformed the out-of-sample data into in-sample data. And in-sample results have no value in terms of predicting the out-of-sample performance. Experiment until you find out what works. Thanks for listening, Howard [url]www.quantitativetradingsystems.com[/url] [/QUOTE]

Verification