Australian (ASX) Stock Market Forum

Approaching system development

Joined
Nov 30, 2011
Posts
9
Reactions
0
Hi all,

I'm relatively new to system trading and I'm struggling along to learn AmiBroker. I've read a smattering of books on system development and remain somewhat uncertain as to the best way to approach this task.

In _Trading Systems_ by Tomasini and Jaekle (2009), discussion is given to varying parameter values while leaving others constant and looking for plateaus rather than spikes on hypothetical equity curves. In case anyone has the book, Figure 2.2 looks very simple but I wonder if [this one graph is] deceptively so.

Suppose you have a MA crossover system with a slow length and a fast length. Suppose further that you wish to test slow lengths between 5-14 and fast lengths between 50-100 (increments of five). Like Figure 2.2, you could keep the slow length (e.g. at 5) while plotting net profits vs. the fast length. Should you repeat that plot for each of the 10 possible slow lengths, though? Now you're looking at 10 graphs for each possible fast length--that's 500 total graphs!

Add a third parameter to the system and you can see how the complexity multiplies.

What if some slow lengths produce plateau regions as the fast lengths are varied while other slow lengths produce spiky plots as the fast lengths are varied? Ultimately it will be a subjective decision as to whether the trader feels confident enough based on the % of plots that look favorable (i.e. plateau vs. spike) but--am I even on the right track in beginning to understand this?

Thanks,
Mark
 
Joined
Jun 13, 2007
Posts
838
Reactions
107
Hi Mark --

I have written extensively on the subject of trading system development, including many posts to Aussie Stock Forums. Search using my name.

Visit my website for articles about the design, testing, validation, and analysis of trading systems:
http://www.blueowlpress.com/WordPress/

Visit the website of one of my books, "Introduction to AmiBroker," and download a copy. It is free. It has chapters on installing AmiBroker and databases, a set of ten exercises that will help you learn about AmiBroker and get up to speed with it, and introductions to the design of trading systems.

For more on the design, testing, and validation of trading systems, look into my book "Quantitative Trading Systems."
http://www.quantitativetradingsystems.com/

Sign up for the Yahoo AmiBroker Forum, and read postings there:
http://finance.groups.yahoo.com/group/amibroker/

Best regards,
Howard
 
Joined
Jun 13, 2007
Posts
838
Reactions
107
Hi all,

In _Trading Systems_ by Tomasini and Jaekle (2009), discussion is given to varying parameter values while leaving others constant and looking for plateaus rather than spikes on hypothetical equity curves. In case anyone has the book, Figure 2.2 looks very simple but I wonder if [this one graph is] deceptively so.

Suppose you have a MA crossover system with a slow length and a fast length. Suppose further that you wish to test slow lengths between 5-14 and fast lengths between 50-100 (increments of five). Like Figure 2.2, you could keep the slow length (e.g. at 5) while plotting net profits vs. the fast length. Should you repeat that plot for each of the 10 possible slow lengths, though? Now you're looking at 10 graphs for each possible fast length--that's 500 total graphs!

Add a third parameter to the system and you can see how the complexity multiplies.

What if some slow lengths produce plateau regions as the fast lengths are varied while other slow lengths produce spiky plots as the fast lengths are varied? Ultimately it will be a subjective decision as to whether the trader feels confident enough based on the % of plots that look favorable (i.e. plateau vs. spike) but--am I even on the right track in beginning to understand this?

Thanks,
Mark
Hi Mark --

You are describing the process known as "evolutionary operation." I discuss it in "Quantitative Trading Systems." It is appropriate for systems that are "well behaved," which trading systems sometimes are. Whether a system is well behaved or not is determined by the multi-dimensional shape of the objective function being searched. If the contours of equal values of the objective function are convex, it is well behaved. (In two dimensions, this can be plotted, as you have noted.) If not convex, evolutionary operation may work or may not.

Your comment about the complexity increasing is called "curse of dimensionality." There is a posting on my blog site that discusses this:
http://www.blueowlpress.com/WordPress/2011/11/optimization-and-the-curse-of-dimensionality/

AmiBroker has both exhaustive and non-exhaustive search. If you want to evaluate the objective function at every combination of parameters, do nothing special and the search will be exhaustive and will do that. If you are looking for the combination of parameters that result in the maximum value of the objective function, and do not necessarily want to see the results of all of the combinations, add this line to your AmiBroker code:
OptimizerSetEngine("cmae");

Your searches will then use the Covariance Matrix Adaptation Evolutionary Strategy to search. Read more about it here:
http://www.amibroker.com/guide/h_optimization.html

CMAE tames the curse of dimensionality -- at least unless you go wild with an enormous search space. Think first, compute second.

Best regards,
Howard
 
Joined
Nov 30, 2011
Posts
9
Reactions
0
Hi Mark --

You are describing the process known as "evolutionary operation." I discuss it in "Quantitative Trading Systems." It is appropriate for systems that are "well behaved," which trading systems sometimes are. Whether a system is well behaved or not is determined by the multi-dimensional shape of the objective function being searched. If the contours of equal values of the objective function are convex, it is well behaved. (In two dimensions, this can be plotted, as you have noted.) If not convex, evolutionary operation may work or may not.

Your comment about the complexity increasing is called "curse of dimensionality." There is a posting on my blog site that discusses this:
http://www.blueowlpress.com/WordPress/2011/11/optimization-and-the-curse-of-dimensionality/

AmiBroker has both exhaustive and non-exhaustive search. If you want to evaluate the objective function at every combination of parameters, do nothing special and the search will be exhaustive and will do that. If you are looking for the combination of parameters that result in the maximum value of the objective function, and do not necessarily want to see the results of all of the combinations, add this line to your AmiBroker code:
OptimizerSetEngine("cmae");

Your searches will then use the Covariance Matrix Adaptation Evolutionary Strategy to search. Read more about it here:
http://www.amibroker.com/guide/h_optimization.html

CMAE tames the curse of dimensionality -- at least unless you go wild with an enormous search space. Think first, compute second.

Best regards,
Howard

Very helpful. You talk about it on p.43 and this helps to organize
my thinking on the subject.

Do you have any suggestions about studying two-dimensional
graphs of parameter values vs. the objective function? My
rudimentary thought was exporting to Excel and generating them
there but I'm open to other ideas (especially if AmiBroker itself
could do them).

Thanks,
Mark
 
Joined
Jun 13, 2007
Posts
838
Reactions
107
Hi Mark --

Studying the response surface of the objective function as variables change is a valuable development technique. You can do that one or two variables at a time to get a feeling for the importance and sensitivity of variables. If you are working with one or two variables at a time, AmiBroker can display the results. I often do that in early stages of designing a system, but seldom to the extent of exporting data to another analysis program such as Excel or R.

During development you will be running in-sample searches and out-of-sample tests. These will tell you a lot about how your system behaves in various conditions. Eventually, as the system development nears completion, you will make some walk forward tests. It is the out-of-sample performance from walk forward tests that indicate whether your system is likely to be profitable.

Regards,
Howard
 
Joined
Nov 30, 2011
Posts
9
Reactions
0
Hi Mark --

Studying the response surface of the objective function as variables change is a valuable development technique. You can do that one or two variables at a time to get a feeling for the importance and sensitivity of variables. If you are working with one or two variables at a time, AmiBroker can display the results. I often do that in early stages of designing a system, but seldom to the extent of exporting data to another analysis program such as Excel or R.

During development you will be running in-sample searches and out-of-sample tests. These will tell you a lot about how your system behaves in various conditions. Eventually, as the system development nears completion, you will make some walk forward tests. It is the out-of-sample performance from walk forward tests that indicate whether your system is likely to be profitable.

Regards,
Howard


How can I use AB to plot the objective function vs. different values of a single parameter?

Also, how can I use AB to plot % returns vs. MFE or MAE? How can I use AB to plot a scatter plot of % return of each trade in the backtest?

The only plotting I see AB is able to do is a 3D plot of one selected function vs. two optimized parameters (under the Optimize menu).

Thanks,
Mark
 
Joined
Jun 13, 2007
Posts
838
Reactions
107
Hi Mark --

To plot objective function versus one parameter, use two parameters, setting one to a constant value and varying the one you want to look at.

For the other plots, you have two choices.
1. Compute the indicator you want and use the Plot statement.
2. Export and analyze in an external program.

MAE and MFE appear trade-by-trade on the trade report. There is a one dimensional plot of them on the Report's Chart tab. Note that you can access the MAE and MFE from the custom backtester within AmiBroker.

For research, but not for live trading, you can look ahead (create an intentional future leak) to see what the MAE and MFE will be. Decide what you want to know, calculate it, then use the Ref function with a positive offset.

The scatter plots would be Percent Gain, MFE, or MAE relative to what?

If there are features you want that AmiBroker does not now have, submit a feature request.

Thanks,
Howard
 
Joined
Nov 12, 2007
Posts
1,628
Reactions
44
Anyone know much about Sharpe ratios? What do they mean and what's a good one, like if I have a system that has a sharpe ratio of 2, that means? I've looked it up but they are about as clear as mud, need a 5 year old explanation :D
 
Joined
Jun 30, 2007
Posts
6,175
Reactions
615
Anyone know much about Sharpe ratios? What do they mean and what's a good one, like if I have a system that has a sharpe ratio of 2, that means? I've looked it up but they are about as clear as mud, need a 5 year old explanation :D
from Amibroker help section:

Sharpe Ratio of trades - Measure of risk adjusted return of investment. Above 1.0 is good, more than 2.0 is very good. More information http://www.stanford.edu/~wfsharpe/art/sr/sr.htm . Calculation: first average percentage return and standard deviation of returns is calculated. Then these two figures are annualized by multipling them by ratio (NumberOfBarsPerYear)/(AvgNumberOfBarsPerTrade). Then the risk free rate of return is subtracted (currently hard-coded 5) from annualized average return and then divided by annualized standard deviation of returns.
 
Joined
Nov 12, 2007
Posts
1,628
Reactions
44
from Amibroker help section:

Sharpe Ratio of trades - Measure of risk adjusted return of investment. Above 1.0 is good, more than 2.0 is very good. More information http://www.stanford.edu/~wfsharpe/art/sr/sr.htm . Calculation: first average percentage return and standard deviation of returns is calculated. Then these two figures are annualized by multipling them by ratio (NumberOfBarsPerYear)/(AvgNumberOfBarsPerTrade). Then the risk free rate of return is subtracted (currently hard-coded 5) from annualized average return and then divided by annualized standard deviation of returns.
Hmm...was reading elsewhere that it needs to be more like 5 to be decent. This system I have developed has a Sharpe of 2, just curious as to what the difference is between say 2 and 5 was and what impact it has on the equity curve if any, doesn't seem to be related to the curve at all?

curve2.jpg
 
Joined
Jun 30, 2007
Posts
6,175
Reactions
615
Hmm...was reading elsewhere that it needs to be more like 5 to be decent. This system I have developed has a Sharpe of 2, just curious as to what the difference is between say 2 and 5 was and what impact it has on the equity curve if any, doesn't seem to be related to the curve at all?

View attachment 54528
Thanks for the reply GB.

It's risk-adjusted return, so a Sharpe of 5 will most likely be 1) smoother and 2) steeper than a Sharpe of 2. If it's not as steep, then it will be very smooth (ie, negligible draw downs). If it's very steep, then the draw downs will be slightly higher also. It would be impossible to create a system with a Sharpe of 5 for a liquid instrument, imo.
 
Joined
Nov 12, 2007
Posts
1,628
Reactions
44
Thanks for the reply GB.
:confused::confused:

What the? You want me to go around thanking you like a servant for every post you make? You didn't have to copy and paste from Amibroker, obviously I'm thankful. Was just keeping the discussion going anyway, not arguing or anything....

Thanks for making my night interesting Sam and giving me something to answer :p:
 
Top