Australian (ASX) Stock Market Forum

System optimisation: IS training vs. OOS testing, and Paper Trading (1 Viewer)

Joined
May 28, 2020
Posts
99
Reactions
150
Hi,

As I understand it...

In Sample (IS) training:
IS training is when you are optimizing your (hopefully!) frozen code. The object of the IS training is to solve for the best outcome of some objective function, say Net Profit, CAR/MDD, % System Drawdown, etc. Let's further say you might decide on a combination of objective functions, for example the combination of Net Profit, CAR/MDD, % System Drawdown, or whatever other combination makes you most "comfortable". So in this respect, the selection of "best" outcome is subjective - not very data science-y but perhaps acceptable? I know data science likes cold hard mathematics - perhaps someone smarter than me could codify some objective function (say Bandy's CAR25?) in a custom back test (CBT) that would, objectively and without any doubt, determine which set of optimization parameters are the best outcome.

As I understand it, in data science terms the sum total of your rules (code), optimization parameters, universe of instruments traded (your training/test/production data), analysis settings (eg. daily vs. weekly), and whatever else I'm forgetting comprises your model. (For the purposes of this discussion "model" means your trading system). ANY change to any of this collection of rules and parameters comprises a new model. In other words, if I have a system and change one parameter or one line of code, I've got a new model. An exception might be the universe of instruments traded, since that is data for the model and not the model itself. Still, if I changed the universe I would likely want to re-optimize.

When training your model, if you use all your data for training, you run the risk of overfitting. You can optimize your code, parameters, settings, or whatever else to the N-th degree to the point that your model is absolutely perfect for your training data, but it may then be "fragile" and not robust when given data with different characteristics to its test data. For this reason, you should not use the entire market data for your optimization. You need some out of sample data to test your model.

For the sake of argument let's say your optimization and therefore test data was from 2000 - 2014.

Out of Sample (OOS) testing:
OOS testing is when you run your model against data that was not used for training, with the goal of predicting how your model will solve an objective function, say Net Profit or CAR/MDD. The extent that your model predicted a favourable outcome (say max Net Profit) determines the "success" of your model. Otherwise you continue development until you get favourable results (or discard your model).

Amibroker's Walk Forward process also does OOS testing, in that it tests a set of data that was not used for training by "walking forward". However, it then re-optimizes, adding that OOS data to the previous IS data. Re-train, re-test, rinse and repeat, and display the aggregated results of the OOS testing.

To me, this implies that, if you conducted a Walk Forward test, got favourable end results, and expected your live system to mimic the Walk Forward, then you would need to re-optimize your model (system) at the Walk Forward time period, for example at the beginning of each year.

This would bring forward your training data across your entire set of production data, i.e. the market data up to "now". I wonder if this could result in overfitting? I suppose you could configure the Walk Forward to stop before consuming the entire data set, say end of 2018 if we're now in 2021. If that gave satisfactory results, you would then "hold back" three years when doing your annual re-optimization.

BTW, I know it's not as cut-and-dried as this. If your model isn't performing well in the current market (new data that doesn't fit the training data), you might re-optimize then instead of waiting for the beginning of the year.

For the sake of argument let's say your test data was from 2015 - today. As I understand it, that is OK - it is still just testing, as long as you are not using that data to optimize your model. If you get poor results from that more recent data, you continue development of your model, and re-optimize against the same test data as before. The minute you start including all your data in the training, you run the risk of overfitting, and have no data with which to test for that. But backtesting is fine.

Question: when do you re-optimize your model? On a regular basis, or only when it is broken ("if it ain't broke don't fix it"?)

Paper Trading:
I've read that, once you develop a system, you should paper trade it for some time before live trading. I guess I'd call this the "gold standard" of OOS testing, in that you are testing with the most current data, and of course it was OOS. And, of course, you're not losing money if current market conditions prove your model is out of sync with the current market data. I've read that you should allow a fair bit of time for that paper trading, say 3 months or more. But what if you're an impatient person like me?

Here's my main question...

Assuming historic market data is frozen, i.e. Norgate won't change data once it's been received, then wouldn't retrospective paper trading be the equivalent of live market paper trading?

For example, say I finish my system on 01Jan and consider it ready for trading and I want to trade NOW.

Scenario #1: In market paper trading
Each week I run an explore, record the results in some paper trading application (say Excel), and start live trading on 01Mar if the paper trading was successful.

Scenario #2: Retrospective paper trading
I run a weekly explore in Amibroker starting 01Sep, record the results in some paper trading application (say Excel), and start live trading on 01Jan if the paper trading was successful.

What's the difference between the two approaches?
 
Joined
May 28, 2020
Posts
99
Reactions
150
Scenario #1: In market paper trading
Each week I run an explore, record the results in some paper trading application (say Excel), and start live trading on 01Mar if the paper trading was successful.

Scenario #2: Retrospective paper trading
I run a weekly explore in Amibroker starting 01Sep, record the results in some paper trading application (say Excel), and start live trading on 01Jan if the paper trading was successful.
I realized this isn't clear - they read like the same thing.

Scenario #1 is "live paper trading" in the market, but you still have to record the results since you did not actually place trades. So paper buys, paper sells, paper profit, paper losses, etc.

Scenario #2 is using the past, unchanging (???) market data to paper trade against historic data. Same record keeping as in Scenario #1. But with an earlier start to live trading.
 
Joined
May 28, 2020
Posts
99
Reactions
150
Would the text book answer be;
When your model is not meeting expectancy?

That's not what Walk Forward does, at least the out of the box configurations. I haven't tried advanced mode, nor do I see how you would codify "not meeting expectations".

So, if you conducted a walk forward, let it re-optimize each year as per the default configuration (or every 6 months, whenever), and you supposedly give relevance to that, just as we give relevance to back testing (we expect the live results to jive/be similar to/sort of match the backtest), then should you not do what the walk forward did?

Can you please cite your textbook ;)
 

tech/a

No Ordinary Duck
Joined
Oct 14, 2004
Posts
19,709
Reactions
4,350
After 27 years and working for 5 years with quants I’ve come to the conclusion that you simply need to find momentum get on and stay on or get off if it stops or reverses.

Sure if you want a set and “ basically “ forget method you can design test and trade it .
But in efficient. The problem I find with most is they design a long system and expect it to work in all market conditions

if you want exceptional returns find filters to identify market conditions.
and trade futures long and short.
 
Joined
May 28, 2020
Posts
99
Reactions
150
if you want exceptional returns find filters to identify market conditions.
and trade futures long and short.
I've read a bit about futures but am still confused. I'm happy to learn more - I'm actually finding the learning process fun. Any books, websites, etc. to learn futures?

I'd also like to learn how to make money in a down market, since all the experts are saying we're headed for one. But who knows the timing? I've read about inverse ETF's but don't think they are the answer. I'm working on a shorting system but it's very much a work in progress right now (more like a jumbled mess).

I'm not asking you to reveal the secret sauce but pointers in the right direction are much appreciated.
 

tech/a

No Ordinary Duck
Joined
Oct 14, 2004
Posts
19,709
Reactions
4,350
Well done.

I lost patience with it and sold out at 0.19 yesterday.

gg

Credit where its due
I owe you a carton!

I've read a bit about futures but am still confused. I'm happy to learn more - I'm actually finding the learning process fun. Any books, websites, etc. to learn futures?

I'd also like to learn how to make money in a down market, since all the experts are saying we're headed for one. But who knows the timing? I've read about inverse ETF's but don't think they are the answer. I'm working on a shorting system but it's very much a work in progress right now (more like a jumbled mess).

I'm not asking you to reveal the secret sauce but pointers in the right direction are much appreciated.

A few of us here reveal the secret sauce all the time

The truth is
“You need to know what you don’t need to know before you can know what it is you need to know.”

The secret is —- there are no secrets.
One day it will hit you like a freight train

Sorry about the top quotes stuffed something up!.
 

Users who are viewing this thread

Top