Backtesting: Why do it?
Question: Why we should think backtesting should work at all? If you take the simplest case of one asset price following a truly random walk, you can backtest the hell out of any strategy and you would still learn nothing because by virtue of it being a random walk there is no information you can gain out of its history.
Am I totally wrong here?
So when you are backtesting against entire portfolios consisting of assets that follow their own random walks, are you not just slipping one proverbial tortoise under the first, all the way down?
It seems that backtesting is more of a way to convince your superior or your client that your strategy is worth trying rather than a way of really predicting its efficacy.
I've always viewed backtesting as a means of seeing how screwed I would be in the worst case senarios. I never view it as a "backtest" but as a stress test. If I'm building a model portfolio whos goal is to minimize risk, I could either set it lose or I could see how it would have fucked up my returns during a major crisis. If I'm building a portfolio that is meant to take huge risks, I can see how that would pan out comparatively against some point in history to gauge performace at the worst of possible times. It's not supposed to be a perfect proof of anything, but a reference point and offer some guidance towards tweaking your model a bit.
This seems like the most reasonable justification for backtesting. But considering all financial models are underspecified (if someone has a financial model that leaves out no nontrivial variables, I'll eat my words), it seems that the value of modeling a strategy in a past situation gives very limited insight as to future performance at best -- and at worst, give very misleading conclusions (if the unaccounted for variables were driving the portfolio value a lot more than the accounted for ones). And even if we were right: say, it replicated the same results from X periods back -- that itself could lead to a false confidence in the strategy, if the agreement in results was purely spurious (i.e., the hidden variables just happened to be in alignment).
I'm sure this problem has been thought over extensively, so what I am really wondering is if practitioners themselves believe in it -- or if it is just a "best practice" that is the best of what we got and mainly performed to appease the higher-ups.
Dude, these are financial models we're talking about right? I know you come from a math/physics background, but financial models aren't supposed to have the predictive power of physical models. If you have a financial model that is correct 51% of the time, and you have strong risk management strategies in place for the 49% of the time it's wrong, I think the model is valuable.
Suppose you want to model a stock as a geometric brownian motion (which I am in no way endorsing), how would you calibrate the stochastic process without considering historical data? How would you estimate the volatility and drift?
Also, backtesting with just in-sample data is useless, but if you use a lot of data to build your model, and then test the model on out of sample data, you can get some insights about the validity of the model you are testing.
It's a matter of tolerances. As you said Guy 2 models cannot be made but almost everyone from the most sophisticated black box trader to the street gambler knows that as something occurs in a regular pattern more and more over time, the chances of it occurring again are increased. Like you said the woman brought her dog in 200 weeks in a row. On week 3 no one would take the bet that she would come in the next Monday but after 200 weeks, almost anyone would acknowledge the pattern and begin to incorporate it into their strategy. Similarly, if you can find patterns in the market that show when XYZ are in place (or whatever metric you want to use) then the price of ABC does N you can say with a decent amount of confidence that those patterns will repeat themselves. It's not perfect of course but its better than no testing prior to implementing a strategy. Just my .02
You're example is flawed. You have to realize when the market changes. Death in your scenario is a market changer.
and even saying Guy 1 gets smoked for a day realizes his model is broken and then has to determine some new way of saying if the ole lady will show up.
I feel backtesting is basically building the model itself. Are you going to tell me that you will take a conceptual trading idea and directly apply it to trading without determining if it can be profitable on some data set?
Let's say you got Guy 1A and Guy 1B. 1A backtests his stuff for 200 weeks and 1B only backtests for 2. 1A's got a LOT more sample points and he could definitely calculate some metric of certainty (take your pick) much greater than 1B's. Now the sad thing is that 1A and 1B are both going to miss the market-changer -- but 1A is way more confident and would have much more quantitatively-justified reason to, say, lever up huge and possibly blow up on week 201.
I realize that I am just rehashing Taleb here, but I've not found a good answer to his cantankerous objections about stats-driven financial modeling.
This is why most firms don't just do data-mining. There needs to be some a priori solid reason for strategy X to work (based on an economic model, for instance) before you bring it to the data. Otherwise, you run a very high risk of a strong in-sample fit with zero predictive power.
You are wrong because it's not a true random walk. I am from a Math background. So if you are from a math background too, then you should know time series data has a lot of autocorrelation. Hence never a true random walk. But I agree if a parameter is generated as a true random walk (as long as you know it's a valid assumption by backtesting), then backtesting for further insights is meaningless.
Aggred with Gekko. Also now you're arguing the quantity of backtesting. I'll take that as a victory saying that some backtesting is beneficial since you sided with 1B, who backtested.
Logger, the WSO debate medal is all yours, because I am sure you know a lot more about this than me.
But let's be clear on what I was saying. First: if backtesting is good, more backtesting is better, right? Then Guy 1A should be better off -- but he is not. More of a good thing should be better, shouldn't it?
Suppose instead we were backtesting betting strategies on a roulette wheel, for which we have very well defined statistics. More backtesting in THIS case would be a good thing because more sample points gets us closer to reality -- which is the house has a clear edge and any strategy against the house would fail in the long run.
Just because 1A is worse off because the backtesting has given him excessive confidence in his strategy, I'm not siding with 1B because he is wrong too. The difference is 1A is wrong and confident, and 1B is wrong and less confident.
I am saying that real life financial situations are more like the old lady situation, and not the roulette wheel situation. Backtesting the old lady problem isn't just not right -- it's not even wrong.
Ok, let me try again. So I agree that there is no point in backtesting to try to figure out where a stock is going. But if you have some sort of reasonable investment hypothesis, then backtesting is useful to check the validity of your assumptions. For example, my investment hypothesis could be that calendar spreads on natural gas futures are cointegrated. So then I would look at historical time series data, and see if I can find some spread of two nat gas futures contracts (maturities 1 year apart) that is mean reverting. If my statistical tests confirm stationarity of this spread, then I can build a trading strategy around this. If I find that the historical data doesn't confirm my assumptions, then I would discard the idea.
I think your whole argument here is that past isn't always prologue, but my point is that the past is all we have, and why shouldn't we use it to make the best decision we can?
Ok so your point is that backtesting is more useful as a means of ruling out bad ideas rather than validating goodones. And in the regime where statistics are not undergoing some wide heteroskedastic swings, this would work.
Haha thanks, I really enjoy medals. I think that ultimately backtesting has a limit to it's usefulness, you're not testing your theory on the open market and so I agree with you ivoteforthatguy. I think as manbearpig has stated that it is all that we have, and it's our duty to do some due dilligence on the ideas we try to profit from.
Aspernatur dolorem aliquid recusandae magni. Nihil repudiandae dolores est omnis debitis et et. Rerum quidem amet harum officiis nisi sit. Sequi dignissimos distinctio eaque iste temporibus. Et corporis ut sapiente non.
Impedit voluptas necessitatibus aspernatur recusandae nam aliquam dolor eligendi. Quis a omnis eum dolorem. Et alias facilis rerum quidem.
See All Comments - 100% Free
WSO depends on everyone being able to pitch in when they know something. Unlock with your email and get bonus: 6 financial modeling lessons free ($199 value)
or Unlock with your social account...