While optimizing the results of my TREND_REV strategy on USDJPY, I was getting some very good results. However, when I backtest this configuration against the 2008-2009 period, the results are not that good. It’s a tough call, do I tell myself:
“It’s ok, the market is always changing, so just run the current configuration and make money!”
… or …
“You need to prepare yourself for the worst just in case”
Well, they say the difference between successful traders and losing traders are that
“losing traders think about how much money they can win, while winning traders think about how much money they can lose”
The process of tuning your strategy to overly match a known set of data is known as curve fitting. One of the ways to prevent/identify this are to do exactly what I have done and run the results against an unknown set of data and see if it still works. (I mention ways on improving backtesting/optimizing results in my July 14th post).
However, I must admit this happens almost every time. I optimize on period A, get the best strategies and run them on Period B and they don’t perform that great. So I optimize them on Period B, and test the best strategies on Period A and the same thing happens.
So what happens if we optimize across Period A and Period B? Well, of course there will be a “best” strategy for that period as well. However, it only produces mediocre results across the periods individually, at least compared to those that did best across the periods individually… and what about the unknown Period C…?
Well in the end, of course the optimizer will produce configuration better suited to the data we are optimizing against than an unknown set of data – if it didn’t do that then it wouldn’t be doing its job. If that is always the case, then what is the point of optimizing at all since we will probably never get the same results on future data?
There have been two approaches I’ve seen to dealing with this:
1) Don’t optimize at all.
If you know your strategy well, and what each parameter does, set the values to something realistic and just backtest across multiple periods it to make sure it ‘behaves’ in all those periods.
2) Optimize on recent short periods only.
The belief here is the market is always changing anyway, and thus your strategy needs to change with it. So, optimize for the last 3-6 months only and expect that to continue for a little bit. When your strategy starts weakening after a few months, re-optimize again. Here we are continually re-tuning our strategy to the market.
They both sound reasonable at a glance, but there are some difficulties I see as well.
The first just won’t make enough money. When you do this, you may have long periods where it ‘behaves’ but hasn’t made any money at all, which is a problem if you plan on living off your profits. You could choose to bet big so when it does ‘make money’ you’ll be fine for the dry periods, but of course, there could be periods where you lose big too – after all, you never can really predict the market.
The second is destined to make some serious losses eventually because according to the theory, the strategy will weaken and lose, and that is when you know you the market has changed and you need to re-tune. Also, I don’t believe this strategy will prepare you for extreme events, like a financial crisis/crash.
However, I think they both make their points:
- Your strategy should behave in all types of markets
- The markets are always changing, and the configuration must change too.
I will think more about this…