System Robustness

GreatPig · 15 August 2007

_ExY_ said:
This method seems to resemble monte carlo whereas instead of randomly selecting trades at random your simply rejecting them(is there any difference?!)

It is a form of Monte Carlo testing. The way I see it, randomly rejecting trades is much the same as randomly picking trades from the available signals, as long as you don't reject so many that the sparsity of signals biases the results too much.

Normally trades are selected based on preference rules, when more than one is available at a time. Randomising which of those trades is selected goes part way towards Monte Carlo testing, but unless your system gives lots of signals, there may be certain signals which will always be picked up simply because they are the only signals available at that time, especially in a shorter-term system. If one that's always picked up is a huge winner, then all your Monte Carlo results will be biased by that same trade, as you may never get a backtest without that trade in it. By randomly rejecting signals though, you will eventually get backtests where that trade is rejected, and will thus see some results that don't include it.

Also, this allows you to test your selection preference rules by keeping those rules (rather than randomising the selection) and simply rejecting some trades at random to ensure you get a variety of portfolios. At least then if a signal is rejected, the next most preferred one will be chosen rather than just a random one.

No idea about MS/Tradesim programming, but in AmiBroker I set a variable called MonteCarlo to some percentage (actually a fraction between 0-1) and then use the formula:

Buy = Buy AND Random() > MonteCarlo;

The larger you make MonteCarlo, the less chance the random value will be larger than it and thus the more chance that the buy signal will be wiped out.

Note though that if you're using the scale-in or scale-out values in the Buy array, to retain those values you'd need to write it as:

Buy = IIf(Random() > MonteCarlo, Buy, False);

although I'm not sure that you'd want to randomly drop scale-out signals.

I'm not saying that this is a perfect way of doing Monte Carlo testing, but I think it's better than just randomising the signal selection at any bar (which in AmiBroker means setting the PositionScore variable to Random).

Cheers,
GP

theasxgorilla · 16 August 2007

_ExY_ said:
How can the information gained by this approach be anymore useful in testing the 'resilience' of the system?

This method seems to resemble monte carlo whereas instead of randomly selecting trades at random your simply rejecting them(is there any difference?!)

I actually still don't get monte carlo analysis...do you reject any trades when you do monte carlo analysis? Or do you just reorder them and observe things like Max DD?

The answer to the first question...my thoughts only...if you reject trades at random, and you have a large enough sample of trades to be considered significant, and through rejecting say 5% of the trades you notice that your CAGR or Max DD or some other relevant measure consistently takes a big hit, then it could be the case that your system is too dependent upon the opportunity factor present in your sample data, and may not perform as well with a reduced opportunity factor in the future.

ASX.G

tech/a · 16 August 2007

theasxgorilla said:
I actually still don't get monte carlo analysis...do you reject any trades when you do monte carlo analysis? Or do you just reorder them and observe things like Max DD?

ASX
The best discription I have come across is this.
MonteCarlo Analysis.
Imagine giving 20000 people (As many as you wish to test). Your trading method and all its rules.You give them all the same amount of Money and you send them off for X years (Your test period) to trade your system. After that period you have them come back to you and you tabulate results.

The answer to the first question...my thoughts only...if you reject trades at random, and you have a large enough sample of trades to be considered significant, and through rejecting say 5% of the trades you notice that your CAGR or Max DD or some other relevant measure consistently takes a big hit, then it could be the case that your system is too dependent upon the opportunity factor present in your sample data, and may not perform as well with a reduced opportunity factor in the future.

ASX.G

Trades are not rejected. As with most systems there is not enough capital to trade EVERY signal.So if I start trading your system today and someone else tommorrow OR of the 5 trades selected I take a different on to 4 other people and each time my porfolio is buying a stock I choose differently to others,then the landscape of my portfolio and that of many others is likely to be radically different.
From constituents to time held to stopped out trades---etc.

So rather than one singular portfolio of trades being tested many 1000s can be tested.I have seen a system which has a 97% success rate (97% of portfolios are profitable,3% are losers) over 20000 portfolios.
Would you trade it?
Why/why not?
(Question is to anyone interested).

CFD · 16 August 2007

Although it's an extreme example I guess it would still depend on the R/R, maxDD, how many years it took to achieve, and what else it involved. If it was back fitted over two years of data, run it over a different two years data.

Good to see you calling a system, a system. Now if only we could call optimisation something else.

nizar · 16 August 2007

tech/a said:
So rather than one singular portfolio of trades being tested many 1000s can be tested.I have seen a system which has a 97% success rate (97% of portfolios are profitable,3% are losers) over 20000 portfolios.
Would you trade it?
Why/why not?
(Question is to anyone interested).

Definately not.
I was testing a system last nite. 99.98% profitability, 0.02% loss.
1 portfolio from the 5000 returned a loss over the testing period.

I wouldnt trade it if somebody paid me!

I could well be that unlucky bastard that doesnt make money!

When i test systems i look for the maximum max.DD over the 5000 or 10000 portfolios, and the minimum return. If i am happy with both of these then i may consider that system.
Im pretty hard to please though.
And of course i ignore the last 5 years in all backtests.
I dont wanna be kidding myself.

tech/a · 16 August 2007

And of course I ignore the last 5 years in all backtests.

Now why on earth would you do that?
Your designing a LONG trading system.
You want it to catch trends and profit from them.
So you cut out the very period of trading your wanting to be involved with.

Thats like designing a boat and testing it on a freeway!!!

nizar · 16 August 2007

Tech/a.
See the last sentence of my previous post.

bingk6 · 16 August 2007

GreatPig said:
Normally trades are selected based on preference rules, when more than one is available at a time. Randomising which of those trades is selected goes part way towards Monte Carlo testing, but unless your system gives lots of signals, there may be certain signals which will always be picked up simply because they are the only signals available at that time, especially in a shorter-term system. If one that's always picked up is a huge winner, then all your Monte Carlo results will be biased by that same trade, as you may never get a backtest without that trade in it. By randomly rejecting signals though, you will eventually get backtests where that trade is rejected, and will thus see some results that don't include it.

Hi GP,

Very sound logic I think. The only thing that I could argue against it is that for a fully mechanical trader, who follows his buy and sell rules to the letter, in the situation that you described above, he'll always be in a position to take that hugh winner because the funds will be available. Therefore the list of trades in each portfolio should always include that hugh winner because in real life, with funds available, that trade would be taken. If you were trying to simulate how a number of portfolios would perform over a period of time, randomly discarding trades would in all probability give you unrealistic results (for better or for worst).

Having said that, where I believe randomly discarding some trades comes into its own would be that it gives you a very good idea as to how robust your system is, in terms of its ability to generate good returns across a range of stocks, across a range of different time frames. It enables you to guage whether your system performance was improved significantly as a result of some huge winners or adversely affected by a few shockers. It allows the extremes to be factored out somewhat and is good for evaluating the efficiency of the entry and exits on their own. However, it does not allow you to accurately simulate how a portfolio would perform over a period of time, which IMHO, is a different kettle of fish again, as I explained in the first paragraph.

tech/a · 16 August 2007

GP and Bingk6

Good points from both.

Nizar
Why do you think testing a bullish system through a bullish period is kidding yourself?
Surely you'd want to know if it out performed the index.
Smoothness of curve
How long trades were held (did you catch most of the trend available)
Drawdowns/stops,leverage during bullish periods.

Are suggesting that developement of a short system using the last 5 yrs would be best?

GreatPig · 16 August 2007

bingk6 said:
he'll always be in a position to take that hugh winner

Yes, but that's a historic winner which may not normally occur and may not occur again. If the system only ends up with positive expectancy because of that trade, then if such a trade doesn't occur again during his trading life, his system won't make money.

A lot of ifs and buts of course, but just something to be wary of. I think a robust system shouldn't be highly reliant on any particular trade or small number of trades.

Cheers,
GP

nizar · 16 August 2007

GreatPig said:
Yes, but that's a historic winner which may not normally occur and may not occur again. If the system only ends up with positive expectancy because of that trade, then if such a trade doesn't occur again during his trading life, his system won't make money.

A lot of ifs and buts of course, but just something to be wary of. I think a robust system shouldn't be highly reliant on any particular trade or small number of trades.

GP.
Great post there.

Tech/a.
I see your point.
I am mainly testing across the period 1992-2002 so i miss out on the last 5 years. This is my main testing period.

The worst portfolio from the 5000 (the no. of monte carlo simulations i am performing) has to be postive in this period and has to provide a reasonable return. Max.dd from the worst portfolio has to be acceptable as well.

When i then forward trade the system, the results were astronomical. along the lines of 85%pa

The reason I do this is because any longterm trend following system will obviously outperform in periods of outstanding bullishness (like 2003-2007).

I'd rather expect less from the system and for the results to suprise to the upside than the other way around.

Will this bullish streak continue?
I dont know but i dont want to give myself a false dream.

Those that were testing and designing systems during 2001-2002 were fortunate in the sense that the period where they began trading hugely outperformed the period in which the system was tested in.

1992-2002 was in no means bearish, but its more slow and steady.

Though i must say at this stage the returns when tested from 1993-1998 are less than impressive.

The stockmarket will have periods of bullishness and strong trends. The key is not lose too much (or make money from other systems??) in between those periods.

tech/a · 16 August 2007

GP

Is this not covered taking out the best and worst trades in the test.
Particularly if one trade as you say makes 20% or more of the profit.

Nizar.
There is only a few ways of investigating the possibility of a bull or bear market period.
I think the answer is to have various systems.And to load up when a bullish period occurs.
These generally last for prolonged periods unlike Corrective moves (When viewed against the back drop of history).
I dont think your kidding yourself if you realise what periods your testing through.
What may happen though is a skewed result with regard to strings of Winners/Drawdowns and R/R (I dont see this skewing as being way way over
the top.)

bingk6 · 16 August 2007

GreatPig said:
Yes, but that's a historic winner which may not normally occur and may not occur again. If the system only ends up with positive expectancy because of that trade, then if such a trade doesn't occur again during his trading life, his system won't make money.

A lot of ifs and buts of course, but just something to be wary of. I think a robust system shouldn't be highly reliant on any particular trade or small number of trades.

Cheers,
GP

It is always better to check for the performance of the system with the extremes removed, so that we are only checking the "bread and butter" stuff. Is there anyway in Amibroker to strip off say the top performing 2% and bottom 2% of trades for any stock when you are performing portolio level testing. The only way I can think off is to exclude a particular symbol from testing altogether, but thats not really what I am looking for as all trades for that symbol is removed.

GreatPig · 16 August 2007

tech/a said:
Is this not covered taking out the best and worst trades in the test.

Probably similar, but I think taking out random trades would be easier to code.

Cheers,
GP

_ExY_ · 16 August 2007

GreatPig said:
I think a robust system shouldn't be highly reliant on any particular trade or small number of trades.

bingk6 said:
It is always better to check for the performance of the system with the extremes removed, so that we are only checking the "bread and butter" stuff.

Why? This doesn’t really answer Nizars original question. Nizars question is more about what is the likelihood of a trading system replicating its results in a different period. So does it matter if you’re testing to see how these extremes are removed? Well its not going to matter if they are there in the future. All of the tests proposed so far simply access how the system will perform if these outliers aren’t there. They are not about accessing weather or not the system will be able to pickup these in the future, which should be the aim when designing these tests (if these types of tests are devised they will be able to measure to some extent the robustness of the system).

Also Nizars question is also about weather or not to deem a system exhibiting statistically significant behaviour based upon its results. You are trying to access weather or not these outliers (well the behaviour of all trades and from those trades the behaviour of the system) picked up by your system are something that is likely to be picked up in the future.

GreatPig said:
If one that's always picked up is a huge winner, then all your Monte Carlo results will be biased by that same trade, as you may never get a backtest without that trade in it. By randomly rejecting signals though, you will eventually get backtests where that trade is rejected, and will thus see some results that don't include it.

I know that this may seem repetitive but this still doesn’t tell us anything we already know and doesn’t address Nizars original question. I could turn this line of thought around and say I have a system that has an expected mean < 0, but should I still trade it because the system may pickup because I may pickup a winning trade that makes the system suitable for trading? What im trying to say is, given we have a system that is profitable because of a small % of outlier trades, and perform Monte Carlo where several portfolio are produced and now lets say that one of these portfolios gives an unfavorable result. Now lets take this unfavorable portfolio and imagine that this was the actual result we got originally prior to performing the Monte Carlo analysis where we reject N trades. I could say well should I reject the system? Because it may pickup some favorable trades in the following N trades.

nizar · 16 August 2007

rnr said:
An alternative to Tech’s method is as follows:-

a) From the Trade Log Window note the unique trade numbers you wish to exclude from this simulation,

b) go to the Trade Database Window and remove the tick from the box of the trade numbers noted in a) above,

c) then press Start Simulation and the new results will now exclude those trades.

GP.
Its pretty easy to remove the outliers, refer to rnr's post above.

ExY.
Great post.

tech/a · 16 August 2007

_ExY_ said:
Why? This doesn’t really answer Nizars original question. Nizars question is more about what is the likelihood of a trading system replicating its results in a different period. So does it matter if you’re testing to see how these extremes are removed? Well its not going to matter if they are there in the future. All of the tests proposed so far simply access how the system will perform if these outliers aren’t there. They are not about accessing weather or not the system will be able to pickup these in the future, which should be the aim when designing these tests (if these types of tests are devised they will be able to measure to some extent the robustness of the system).

Speaking for myself.
I want a system which doesnt need an outlier move to make it profitable.
I'm looking for a smooth equity curve rather than one which has a spike.
If I take out the spike and the end result of further testing is a smooth curve with a strong set of numbers,I can be reasonably sure that the system will also trade similar outlier moves. If it doesnt then its still acceptable.
If the outlier is the only reason for the system being profitable over a 10 year period I wont want to trade it for 10 yrs in the hope another outlier turns the system into profit.

Also Nizars question is also about weather or not to deem a system exhibiting statistically significant behaviour based upon its results. You are trying to access weather or not these outliers (well the behaviour of all trades and from those trades the behaviour of the system) picked up by your system are something that is likely to be picked up in the future.

See above.

I know that this may seem repetitive but this still doesn’t tell us anything we already know and doesn’t address Nizars original question. I could turn this line of thought around and say I have a system that has an expected mean < 0, but should I still trade it because the system may pickup because I may pickup a winning trade that makes the system suitable for trading? What im trying to say is, given we have a system that is profitable because of a small % of outlier trades, and perform Monte Carlo where several portfolio are produced and now lets say that one of these portfolios gives an unfavorable result. Now lets take this unfavorable portfolio and imagine that this was the actual result we got originally prior to performing the Monte Carlo analysis where we reject N trades. I could say well should I reject the system? Because it may pickup some favorable trades in the following N trades.

I see your point and yes it could.
But do you really want to take a punt that the portfolio your trading wont be the portfolio that makes a loss.
Systems developement is about removing the "Punt" factor.

GreatPig · 16 August 2007

_ExY_ said:
So does it matter if you’re testing to see how these extremes are removed?

Randomly removing buy signals is not just about removing outliers. As it randomly removes any signal, there will be some backtests where the outliers are still present. It's primarily to help ensure that different backtests will take different portfolio paths, and that after lots of backtests, a large variety of portfolios using the same signals will have been tested. As far as outliers go, it's mainly to help ensure that none of them are always included in every portfolio backtest.

this still doesn’t tell us anything we already know and doesn’t address Nizars original question.

It wasn't a response to Nizar's original question, rather to your more-recent question.

GP

tech/a · 16 August 2007

GP

Never done that myself but can see the merits.

It's primarily to help ensure that different backtests will take different portfolio paths,

_ExY_ · 16 August 2007

Just saw this.

tech/a said:
So rather than one singular portfolio of trades being tested many 1000s can be tested.I have seen a system which has a 97% success rate (97% of portfolios are profitable,3% are losers) over 20000 portfolios.
Would you trade it?
Why/why not?

If the Sum((FinalCapital - InitialCapital) for all portfolios)>0 then you should trade it.

tech/a said:
I see your point and yes it could.
But do you really want to take a punt that the portfolio your trading wont be the portfolio that makes a loss.
Systems development is about removing the "Punt" factor.

Well I guess in the example used by nizar you would monte carlo it and find out if Sum(All portfolio outcomes)>0, if so you should trade it.

GreatPig said:
It wasn't a response to Nizar's original question, rather to your more-recent question.

Sorry for my hastyness and thankyou for disclosing the details of your technique.

System Robustness

GreatPig

Pigs In Space

theasxgorilla

Problem solved... next bubble.

tech/a

No Ordinary Duck

CFD

nizar

tech/a

No Ordinary Duck

nizar

bingk6

tech/a

No Ordinary Duck

GreatPig

Pigs In Space

nizar

tech/a

No Ordinary Duck

bingk6

GreatPig

Pigs In Space

_ExY_

nizar

tech/a

No Ordinary Duck

GreatPig

Pigs In Space

tech/a

No Ordinary Duck

_ExY_

Similar threads

System Robustness

Pigs In Space

Problem solved... next bubble.

No Ordinary Duck

No Ordinary Duck

No Ordinary Duck

Pigs In Space

No Ordinary Duck

Pigs In Space

No Ordinary Duck

Pigs In Space

No Ordinary Duck

Similar threads

Privacy & Transparency

Privacy & Transparency