An offshoot from the threads about modelling.
Considering that there are so many scientists/engineers in the financial world, it's surprising how quickly the mind gets used to accepting crap modelling practice.
As an engineer, we use models to simulate a real-world situation. However, a model is worthless unless it conforms to accepted first principles (i.e. theory) and experimental results. Models are generally used because experimental simulation may be impossible, too expensive, too complex, etc. Computer modelling is reasonably quick and easy. However, it's always important to "validate" every model. Otherwise, you get garbage in, garbage out.
So, my question is:
Do bankers ever check the validity of their models after they have been used? How often do bankers look back in time to see how well their old model actually simulated the true outcome? How great would it be if (looking back 5 years), all the Large Cap Oil Exploration Company models (let's say a DCF model) of Banker A turned out to be within 2% of reality line for line, and the models of Banker B were off by 10-20% from revenue to CAPEX to free cash flow? You could then start to say that the for this type of situation, the assumptions and process undertaken by Banker A are more appropriate and (assuming another similar situation comes up) should be replicated.
Does this kind of thing ever happen? Is it studied at any business schools? Am I spouting rubbish?
I believe the best practice is always to go back and recheck how your assumptions played out. In addition, don't forget that one of the most important things in a model is the assumptions list: how conservative are they?
As far as models themselves go, not really. But there are plenty of studies on equity analyst accuracy over time, and how recieving awards like "Analyst of the year" can help or hinder, and why. It might be a stretch, but you could also think of it like this: we model the future, and the accuracy of that prediction plays a part in the modeller's/investor/manager/whoever's alpha.
So, (again, maybe a stretch) but you could say that managers that produce consistent alpha must have consistent and accurate modelling skills. Obviously in reality, some of them just have a great gut, or there are other things that they understand that the marginal investor can't.
But don't forget, every opportunity is unique. Just because research shows a trend to be true today doesn't mean it will exist tomorrow. In fact, the theory of an efficient market leads to the fact that these predictive abilities will be eroded away (the more efficient, the faster) just like any mispricing is quickly eroded away (though obviously, they must happen in the first place).
It's like: when papers came out from academia proving the excess returns potential of a momentum strategy, the returns from the strategy went away as more investors adopted the strategy. It's not like making a chemical, where the formula holds true forever.
I see what you're getting at, but accuracy really has nothing to do with modeling abilities and everything to do with a model's underlying assumptions. And of course, the accuracy of one's assumptions don't necessarily prove anything about the quality of their foresight.
In almost any case, the key assumptions driving a model come from senior folks who will never actually touch the model.
Thanks, guys. I think it's an interesting idea to compare two vastly different sciences and the differences in modelling. I guess a point that Alphaholic is making is that although parts of finance (and maybe more so, accounting) are governed by rules, there is a very social (behaviour) science aspect as well. And then of course the statistics are applied mathematics. I think it's interesting, anyway...
Smuguy, I think I was saying the same thing as you, in that the accuracy of a model depends on the assumptions (not the modelling skills), so I agree with you there. I was trying to say that someone who consistently makes the "correct" assumptions about the future is bound to understand something better than another analyst. Are you saying that foresight is different from making the correct assumptions about the future?
So, is there any way to identify which analyst/associate/VP/MD it is on a team that is consistently making these correct assumptions? Or is that exactly how you would actually define a good "banker", in addition to all their other skills, of course.
Sorry if I'm not making much sense. Trying to verbalise my philosophical views late at night...
"It is a fine thing to be out on the hills alone. A man can hardly be a beast or a fool alone on a great mountain." - Francis Kilvert (1840-1879)
"Ce serait bien plus beau si je pouvais le dire à quelqu'un." - Samivel
A lot of models are "standard" models used by every one else, this works to the advantage where if the model fails to take into account some events like (house prices do go down in value, people may decide to pay car loans and credit card yet default on mortgages) then the analyst can hind behind his "industry standard assumptions" to save his skin.
In Europe, it is considered blasphemy to default on your mortgage yet keep up repayments on car loans, yet we see the exact behaviour in the US.
To quote the top rating agencies, the assumption is that individuals will prefer to default in the following order, auto loan, credit card then mortgages.
But the reverse has been seen to occur (only us), an individual described this behaviour (seemed perfectly logical) in the following way
What a brilliant logical behaviour!!!
Molestiae perspiciatis ipsum voluptas atque eaque perferendis. Ut minima ut esse consectetur est dolorum. Consequuntur sunt omnis amet blanditiis nostrum nisi quia.
Sed cumque dicta quibusdam. Suscipit officiis occaecati et eaque libero nam. Totam nihil distinctio magni et corrupti facilis. Dolorem tenetur nulla veritatis quo modi.
Porro harum totam modi deleniti eaque officiis ducimus. Vel praesentium at eos nobis eius reiciendis enim. Non voluptatem aut libero adipisci officia suscipit.
Qui harum tempore inventore tenetur quis. Ut suscipit molestiae dolor incidunt voluptate provident nobis. Quaerat aut et quis id. Omnis aut est est qui quaerat cumque explicabo.
See All Comments - 100% Free
WSO depends on everyone being able to pitch in when they know something. Unlock with your email and get bonus: 6 financial modeling lessons free ($199 value)
or Unlock with your social account...