Backtested Systems, Lies, Damned Lies, & Statistics

It was this quote from tech/a that really seemed to highlight the belief that mechanical systems somehow present a trading plan, or methodology that based on statistics and mathematics, offered what is commonly referred to as the edge

Discretionary trading at best is Maverick style.
There is much talk about plans and I often see plans that are like jelly ---constantly changing.


Sound plans have statistics on entry exit,stops and position size variables so that when trading you KNOW wether what you are doing is within your results---reslts that determine profitability if you trade a particular plan.

Click to expand...

We see that within this definition, discretionary trading has been portrayed as trading without a consistent plan, viz. trading on emotion.

Contrasted with the mechanical advantage, or statistically tested methodology that results that determine profitability if you trade a particular plan.

Therefore it would really behoove anyone who is contemplating trading a mechanical methodology to closely examine some or all of the assumptions that underlie the NUMBERS

To give us a benchmark, and a point of comparable standards, I shall utilize Medical Research gold standards as the benchmark.
In medical research statistical modeling is used extensively in the research of new drugs, diagnostic criteria, treatment methodologies and everything inbetween.

Why medicine?
Several reasons, and all related to the assessment of methodological quality.

1...It utilizes human beings, who though possessed of the same, or similar anatomy & physiology, have very different outcomes due to many variables
2...Medical research is required to be predictive.
3...Medical research must be predictive for the majority, within clearly defined confidence parametres and examination of causation to be accepted (Correlation, regression, and causation)
4...The statistical techniques utilized are in many instances the same as utilized in these backtested mechanical methodologies
5...Shoddy research can be hazardous to your health, or wealth in trading terms.
6...The increasing use of Surrogate endpoints
7...Paired data, tails, and outliers
8...Study designs

jog on
d998

As with anything, starting at the beginning is probably the best place to make a start.

Was the study original?
Using my favorite mechanical system as a benchmark, the ubiquitous TechTrader, we can ask the question.

I personally think no, it wasn't, it had many similarities to the TURTLE system Now this is not a criticism (not yet anyway). Was the methodology (possibly undisclosed) examined in anyway for variables that were similar or the same within TechTrader?

From this starting point we can ask the following questions;

Is this study larger, continued for longer, or otherwise more substantial than previous studies (studies equating to trading methodologies)

Is the methodology of this study any more rigorous (in particular, does it cover any specific methodological criticisms of previous studies)

Will the numerical results of this study add significantly to a meta-analysis of previous studies?

Is the population studied different in any way (has the study examined capitalization structures, duration of listing, economic environment, market environment, inflation, interest, alternate investment classes etc)

I'll give tech/a some time to have a think about some of the preliminary questions before continuing any further.

jog on
d998

Hi ducati

Sorry but my attention span at this time of the morning is limited to posts no more than half a screen in length but I get the jist of what you're saying - I think

But anyway, I'm a firm believer in having a written investment/trading plan which should be paper traded (especially for traders) before committing real funds.

Imo a sound plan should include but not be limited to things like:

1) Your objectives - long and/or short term

2) What markets and types of securities you will trade/invest in - shares, bonds, options, hybrids, fixed interest etc etc

3) How you will search/scan for investment/trading candidates - technical/fundamental or other criteria.

4) How you will determine entry/exit points - technical/fundamental criteria or a combination of both

5) How you will determine the position size for each investment/trade.

6) How you will manage risk - diversification, stop losses etc etc

Now how complicated or simple someone makes their plan is solely up to them - but imo a key purpose of a plan is to remove emotion as much as possible from decision making by making the plan as comprehensive as required to suit ones objectives and circumstances.

I think it is very important to paper trade a plan for however long it takes (and that will vary according to each individual) until the person is satisfied the plan is generating the objectives/returns specified in the plan before committing funds.

cheers

bullmarket

bullmarket

While I acknowledge the effort that you have taken in your answer, and certainly the information that you have provided is within its context materially correct, you have I'm afraid sort of missed the point of the thread.
This may in point of fact be my error.

The purpose of the thread is as follows;
Mechanically traded systems claim statistical evidence of their efficacy.
We know this from the methodology that is followed within the software that takes thousands of data inputs and spews back a statistically derived ratio.

These ratios form the basis of a trading plan, and delineated via further calculations, one example being the expectancy calculation which is as (1+($win/$loss)*(#wins/#losses)-1)

These calculated ratios, and equation based outcomes infer that the trading methodology is in point of fact robust, logical, tested, mathematically determined, and therefore provides one of the most cliched trading phrases ever promulgated...........an edge

The purpose of this thread is to examine the methodology behind the numbers to examine just how reliable, robust, accurate, or manipulated that they actually are. After all before you place all your cash, and leverage yourself higher than might be prudent, wouldn't you examine the assumptions and facts behind the software?

jog on
d998

ok no problem ducati

ducati916 said:

bullmarket


Mechanically traded systems claim statistical evidence of their efficacy.
We know this from the methodology that is followed within the software that takes thousands of data inputs and spews back a statistically derived ratio.

These ratios form the basis of a trading plan, and delineated via further calculations, one example being the expectancy calculation which is as (1+($win/$loss)*(#wins/#losses)-1)

These calculated ratios, and equation based outcomes infer that the trading methodology is in point of fact robust, logical, tested, mathematically determined, and therefore provides one of the most cliched trading phrases ever promulgated...........an edge


jog on
d998

Click to expand...

re the above extract from your post the way I look at it is that mechanical systems, or whatever you want to call them, with their various ratios or whatever are largely just a way of monitoring your trading performance and keeping an eye on your trading performance to see if your plan's objectives are consistantly being met is critical for obvious reasons.

Keeping a record in a spreadsheet or whatever of things like:

1) number of trades
2) number of winning trades
3) number of losing trades
4) $ profit per winning trade
5) $ loss per losing trade
6) time in both winning and losing trades etc etc etc

and calculating various ratios from the above and other data to then compare against preset bench-marks is an important part of portfolio and trading performance monitoring imo.

Now whether doing any or more of the above gives someone an 'edge' is simply up to one's own perception of what they call an edge so to speak.

But using the above as a base starting point to monitor trading performance should certainly help identify potential flaws in someone's trading strategy sooner rather than later when compared to someone whose record keeping and/or monitoring of their performance is not as thorough.

And finally, all of what I mention above imo should be included in someone's trading plan as part of their portfolio/trading performance monitoring process....and hence the importance imo of paper trading a plan for however long it takes prior to commiting funds so that your trading performance monitoring process can show you if your trading strategies are generating the returns you expected.

cheers

bullmarket

ooops....mrs bullmarket is calling I'll log in tonight if you would like to discuss further....cheers..

I sense a crowd gathering ......

Duc,

Interesting, and I'm probably going to find myself agreeing with you here. Watching with interest....

I'm tied up with business issues today and will become "involved" when I can.

Suprising to some I'm with duc to a degree on some matters.

bullmarket

re the above extract from your post the way I look at it is that mechanical systems, or whatever you want to call them, with their various ratios or whatever are largely just a way of monitoring your trading performance and keeping an eye on your trading performance to see if your plan's objectives are consistantly being met is critical for obvious reasons.

Click to expand...

Yes, this is true.
However you are running way ahead of the curve at the moment.
Implicit within your statement is that the trading plan built with a statistically backtested result is legitimate because it has been built mathematically or statistically, which assumes the statistical methodology is of a high methodological design.

Lets reverse this for a moment.
I as a new trader have an idea of a methodology that I wish to trade.
I formulate a trading plan.
I trade the plan.
I record the results.
Early returns are most unsatisfactory, I give up on this plan, and go to another plan and repeat.

This is your method, but you qualify it by saying paper-trade it, so that if its horrible, you don't lose too much cash.

tech/a and others are saying;
1...Statistically, you have no idea whether your failed plan, is in reality no good, because the sample size is inconsequential, and it may just be in drawdown

And finally, all of what I mention above imo should be included in someone's trading plan as part of their portfolio/trading performance monitoring process....and hence the importance imo of paper trading a plan for however long it takes prior to commiting funds so that your trading performance monitoring process can show you if your trading strategies are generating the returns you expected.

Click to expand...

2...To test any idea by empirically, is so slow and drawn-out that by the time you have enough data to statistically draw meaningful conclusions, you'll be too old to care, to rich too care, or busted out living in a dustbin.

Therefore, enter the sandman..............
Computer software that runs monte-carlo simulations and all sorts of nifty timesaving bolt-ons, and hey presto, instant results.

This is where I come in....................
If you are going to base your trading plan on statistically generated results you must first assess the methodological quality of the statistical models that you are building and running if you do not, then how reliable are the trading systems that you will presumably place cash, and potentially leverage it, you could be in dustbin city far sooner than planned.

Therefore, utilizing medicine, and medical research, that utilize almost the same models, we can benchmark the methodological quality of the statistical tests, results, and conclusions.

To do so, we must start at the beginning, and build the concept of the model to be utilized, and look for potential faults, and potential remedies.

Now whether doing any or more of the above gives someone an 'edge' is simply up to one's own perception of what they call an edge so to speak.

Click to expand...

Which brings us directly into the psychological models that are being built within behavioral finance currently. However, that is somewhat a separate topic, and will muddy the water currently.

WayneL

Interesting, and I'm probably going to find myself agreeing with you here. Watching with interest....

Click to expand...

Oh dear, that will never do.

tech/a

I'm tied up with business issues today and will become "involved" when I can.

Suprising to some I'm with duc to a degree on some matters.

Click to expand...

Having an afternoon nap more likely.

jog on
d998

ducati916 said:

WayneL

Oh dear, that will never do.

Click to expand...

Ahhhhhh Ahhhlright!

We can argue about the expectancy equation (which I maintain is a misnomer)... if you insist

Just so that the initial questions that require consideration don't get too buried, here is a quick reminder.

Is this study larger, continued for longer, or otherwise more substantial than previous studies (studies equating to trading methodologies)

Is the methodology of this study any more rigorous (in particular, does it cover any specific methodological criticisms of previous studies)

Will the numerical results of this study add significantly to a meta-analysis of previous studies?

Is the population studied different in any way (has the study examined capitalization structures, duration of listing, economic environment, market environment, inflation, interest, alternate investment classes etc)

Click to expand...

These are the questions that need to be considered prior to moving forward.
So all you mechanical backtesting advocates, don't be shy.

jog on
d998

WayneL

Ahhhhhh Ahhhlright!

We can argue about the expectancy equation (which I maintain is a misnomer)... if you insist

Click to expand...

Indeed we can, but we are a long way from that point currently.
We still need to work our way through some of the assumptions that lie at the heart of the software designed to produce the statistics.

jog on
d998

ducati916 said:

Just so that the initial questions that require consideration don't get too buried, here is a quick reminder.



These are the questions that need to be considered prior to moving forward.
So all you mechanical backtesting advocates, don't be shy.

jog on
d998

Click to expand...

Duc,

So curve fitting and overoptimisation should be included in the discussion then.

Curve fitting: matching a system to the data. To make a system look great for the period and data given.

Overoptimisation: fudging the parameters of a sytem until it works. Much like changing the moving averages until you find one that gives the best exit entry etc. So as a result it only works on the data tested on and is no guarantee of future success.

How does one overcome these biases in the search for the edge ?:nono:

Hi ducati

ducati916 said:

bullmarket



Yes, this is true.
However you are running way ahead of the curve at the moment.
Implicit within your statement is that the trading plan built with a statistically backtested result is legitimate because it has been built mathematically or statistically, which assumes the statistical methodology is of a high methodological design.

Lets reverse this for a moment.
I as a new trader have an idea of a methodology that I wish to trade.
I formulate a trading plan.
I trade the plan.
I record the results.

Early returns are most unsatisfactory, I give up on this plan, and go to another plan and repeat.

This is your method, but you qualify it by saying paper-trade it, so that if its horrible, you don't lose too much cash.

jog on
d998

Click to expand...

If you paper trade your plan then you don't loose any cash at all so I'm not sure what you're getting at with "This is your method, but you qualify it by saying paper-trade it, so that if its horrible, you don't lose too much cash."

Also if the initial results are most unsatisfactory, as you put it, you don't have to give up on the plan at all as you suggest and move on to another.

I'd suggest looking into why the results are unsatisfactory and try to remedy as their are a multitude of possible reasons for the poor results during paper trading.

Some could be:

1) Poor selection criteria/parameters (fundamental and/or technical) when scanning the market for trading candidates either manually or preferably with market scanning software which should be included in any reputable charting software thus resulting in low grade trading opps.

2) Poor entry/exit criteria and parameters if using charts. Maybe the trader just needs to read up or whatever a bit more on charting techniques to hopefully improve their charting skills.

3) Maybe the trader needs to fine tune how he/she sets their position size

4) etc etc etc

I'd be surprised if there wasn't some modifications to the plan during paper trading. Basically, setting up and fine tuning a trading plan will require a lot of time and effort to get it to a stage where the trader will feel comfortable with starting to commit real funds. Now how long that takes will depend on the attitiude of the trader, their temperament how often they will likely trade etc etc.

But now after my having said the above, someone could also ask in reply to your "Early returns are most unsatisfactory, I give up on this plan, and go to another plan and repeat." - What if the initial results ARE satisfactory? I don't see anything wrong in the trader then commencing to commit real funds to their plan

But then, even after trading with real funds you might recall in other threads that I have suggested periodically reviewing one's trading plan to firstly monitor its performance against preset bench marks and then fine tuning it if required.

Don't get me wrong - the ideas/concepts I describe in this and earlier posts are by no means going to guarantee any significant profits in the long run, human nature and weaknesses being what they are, but there is no way you are going to convince me that following what I have posted earlier will not at least improve the probability of a trader being in the 10% or so of traders that end up being profitable in the long term.

How profitable they end up being in the long run will depend on things like their attitude, willingness to put in the large amount of time and effort required to be a successful professional trader and their skill in technical and/or fundamental analysis. Personally I don't have the temperament to be a full time trader.

Finally, now I'm sure others will have different views and methods on what they believe is the best way to approach trading - but imo the best approach is what the trader is most comfortable with. After all, if you're not happy and hence reluctant to put in the time and effort in whatever plan structure you choose there is a high probability that you will most likely simply loose interest and fail in the long run.

cheers

bullmarket

Snake

So curve fitting and overoptimisation should be included in the discussion then.

Curve fitting: matching a system to the data. To make a system look great for the period and data given.

Overoptimisation: fudging the parameters of a sytem until it works. Much like changing the moving averages until you find one that gives the best exit entry etc. So as a result it only works on the data tested on and is no guarantee of future success

Click to expand...

Absolutely.
Curve fitting and overoptimization sound very much the same as data mining.
These definitely need to be examined, and if present pretty much nullify any conclusions drawn from the data, and potentially open the trader/investor to serious losses.

How does one overcome these biases in the search for the edge

Click to expand...

By having a very high standard within the methodological design.
By starting from the bottom up, as opposed from the top down (seemingly the route taken by current software, although, currently this is simply my *feeling* ) we can examine the methodological quality, and decide as to the quality of the conclusions drawn

jog on
d998

Hello again Ducati

Why not go with the 'Best Practice' model and keep doing what you know works most of the time?

Sure it doesn't GUARANTEE perfection, but hey thats you, me, everyone else and life. Who really needs to be RIGHT all the time anyway to succeed?

We've all got to roll with the punches anyway, dust ourselves off and get up.
Anyone here know a trader that hasn't taken a decent hit? I don't.

Cheers
Happytrader

hi happytrader

happytrader said:

Hello again Ducati

Why not go with the 'Best Practice' model and keep doing what you know works most of the time?

Sure it doesn't GUARANTEE perfection, but hey thats you, me, everyone else and life. Who really needs to be RIGHT all the time anyway to succeed?

We've all got to roll with the punches anyway, dust ourselves off and get up.
Anyone here know a trader that hasn't taken a decent hit? I don't.

Cheers
Happytrader

Click to expand...

:iagree: with the jist of your post and its similar to the point I was making earlier re traders/investors should work with what works best for them according to their personalities, temperament, objectives etc etc.

I think it's a good idea to toss around different views and ideas in a forum like this but imo when people start insisting that their way is the best or whatever then from my point of view they leave themselves wide open to being laughed at by those who are much more successful than them and who didn't use their 'methodolgies' or whatever you want to call them

Imo choosing a trading/investment strategy and how you implement that is a case of horses for courses to some extent as there is no single 1 way of trading/investing which is the best because if there was then everybody would be using it

cheers

bullmarket

bullmarket

I can see that you have a very firm grip on the wrong end of the stick, and are not going to let go without a fight. Ok, here we go.

You can test a trading or investment methodology several different ways.
We shall call your outlined methodology method #1

In method #1
We manually test a trading plan by forward testing the plan.
Or, we backtest the plan using historical data.
In both instances, we do it ourselves.

Forward testing has one major flaw, and that is it is time consuming.
It cannot progress faster than one day at a time.
The second flaw (potential flaw) is the statistically insignificant sample size.
Therefore, statistically speaking, you are trading, and will trading for a very long period of time, a purely discretionary method

If you backtest, using historical data, then you must ensure that you have a methodological design quality that will deliver statistically significant outcomes.

This is what this thread is going to discuss.

When backtesting, you can go the slow, labour intensive method, that is time consuming, method #2 but has one major advantage in that you understand exactly the inputs into the methodological design, or, you go the automated route, that is fast, but, is not exactly transparent within the methodology and this can be method #3

Don't get me wrong - the ideas/concepts I describe in this and earlier posts are by no means going to guarantee any significant profits in the long run, human nature and weaknesses being what they are, but there is no way you are going to convince me that following what I have posted earlier will not at least improve the probability of a trader being in the 10% or so of traders that end up being profitable in the long term.

Click to expand...

Focus on the word probability
Probability has nothing whatsoever to do with the context in which you are using it. Probabilities within financial markets are argued over endlessly, and it really depends on how you view linear and non-linear theory, complexity theory, power laws, equilibrium, fractals, determinism and other quite fascinating theories

Probability does have relevance, to a degree, in that it has as a concept been so misused, that it has been accepted into trader lore, high probability set-up, high probability trade, etc, etc
Also, with mechanical systems, there has been the hint of an inference that the "numbers" portray a high probability of success..........
We'll see.

How profitable they end up being in the long run will depend on things like their attitude, willingness to put in the large amount of time and effort required to be a successful professional trader and their skill in technical and/or fundamental analysis. Personally I don't have the temperament to be a full time trader.

Click to expand...

I tend to agree with you on this one.
The mecchies, however will not, it will they argue take as long as it takes you to generate positive numbers.
Will it?

Finally, now I'm sure others will have different views and methods on what they believe is the best way to approach trading - but imo the best approach is what the trader is most comfortable with. After all, if you're not happy and hence reluctant to put in the time and effort in whatever plan structure you choose there is a high probability that you will most likely simply loose interest and fail in the long run.

Click to expand...

Comfort zone............
"Losers do what is comfortable, Winners do what is necessary"
Psychological compatibility, intellectual compatibility, emotional compatibility
All areas of comfort.
Comfort can be a mind-set. What starts out as "work" can, with the right attitude become pleasurable, or can it?
Interesting.

jog on
d998

happytrader

Why not go with the 'Best Practice' model and keep doing what you know works most of the time?

Click to expand...

Don't really follow your post here.
Best Practice model, refers to a benchmark, for comparative values.

bullmarket

I think it's a good idea to toss around different views and ideas in a forum like this but imo when people start insisting that their way is the best or whatever then from my point of view they leave themselves wide open to being laughed at by those who are much more successful than them and who didn't use their 'methodolgies' or whatever you want to call them

Click to expand...

Yes agree.
Does that mean that I will not be subjected to another post on your method?

Imo choosing a trading/investment strategy and how you implement that is a case of horses for courses to some extent as there is no single 1 way of trading/investing which is the best because if there was then everybody would be using it

Click to expand...

That is most likely true, but it still doesn't exclude the possibility that the vast majority do, and you're just the odd one out.

Intrestingly, when you start to examine the successes in the financial markets, you start to notice that there are far more similarities, than differences................but that really is off topic.

jog on
d998

no problem ducati

I just posted my views with supporting reasons in reply to your posts and that is all I can do.....if we disagree then that is fine - I don't have a problem with that

I'm not out to try to prove that you are wrong and I am right and I don't accept that you are right and I am wrong.

As I said before its horses for courses and I will simply stick with what works for me. Having said that, obviously what works for me will also work for some and not work for others....it's as simple as that I will just continue to call things as I see them with supporting reasons.

As I said before, those who try to insist they are right and everyone else is wrong (and I'm not saying you are doing that here) leave themselves wide open to being laughed at by those much more successful than they are and who also don't agree with them.

You have your views and supporting reasons and I have mine with my supporting reasons and that's all there is to it....both will have pros and cons

cheers

bullmarket

hi ducati, interesting thread you've started here.
quick question- Is it technical analysis or pure mechanical trading that you don't like?