Machine Learning Taking over HF research analyst roles in near future?

therealgekko's picture
Rank: King Kong | banana points 1,382

I read an interesting article recently regarding machine learning taking over research analyst roles over the next 5 years or so (https://qz.com/1113633/wall-street-analyst-jobs-ar...).
I also spoke to the CFO of a multi-billion dollar pension fund recently, and he also supported this idea, that in the next 5 years, a majority of research analyst positions on Wall Street would simply be taken over by computers, putting them out of work (at least in public securities). What is your perspective on this? Machines can obviously crunch numbers faster and spot patterns/trends far more effectively.

Do you believe that the era of the research analyst will come to an end by 2030? Does it make any sense at all now to enter asset management or the hedge fund world (on the public securities side)? If so, where should someone interested in fundamental research look into for opportunity?

Comments (28)

Mar 10, 2018

I am highly skeptical that machine learning by itself (ignoring the influence of hype on fund flows) would make fundamental analysts irrelevant by 2030.

As long as human intelligence is able to process and synthesize information orthogonal to what can be done with ml algorithms, there will always be a demand for the best humans, especially in a large aum/headcount setting (where a marginal improvement in ability can translate into large economic gains). Finance is an especially challenging area for ml, due to noisy data and non-stationarity. However, it's also quite likely that things will continue to get more competitive.

Here's another thought, even the most quantitative firms still rely on humans to make hiring decisions (in-person interviews).

Finally, I would guess that private side investing (private equity) will have better prospects going forward than fundamental public side investing. There are much higher barriers to entry, more limited data, and soft-skills are more of a differentiator.

Learn More

814 questions across 165 hedge funds. 10+ Sample Pitches (Short and Long) with Template Files. The WSO Hedge Fund Interview Prep Course has everything you'll ever need to land the most coveted jobs on the buyside. Learn more.

Mar 11, 2018

How long do you feel before machine learning make the research analyst irrelevant?

Or, put another way, at what point do you think ML will reach a point to cull the analyst pool down by half or even a third?

Mar 14, 2018

Great post, I'm going to be honest though... I had to Google orthogonal

Mar 11, 2018

I guess I don't really see them as directly competing, and I've worked a decent amount with ML and NLP. There's a ton of hype in the media but it (including the article you linked) seems mostly unsubstantiated.

As another analogy, think about how irritating it can be trying to work with an automated voice system sometimes. Or using Siri/Alexa/Google Home. Ultimately these systems have nowhere near the flexibility and general reasoning ability (necessary for picking up and communicating certain types of information/context) as people.

Mar 11, 2018

I see this stuff as 99.9% breathless hype from people who know neither finance nor comp sci. "Analyst" isn't some single skill that can be brute forced. A computer that could perform the wide variety of tasks required would need to be near generalized AI, and that is nowhere close

Mar 12, 2018

+1SB. How do you feel about the neural network? Essentially, this is hundreds of layers of reasoning and connections in a machine that is designed to simulate the complexity of the human brain.

Do you think that new research analysts can fully make a career out of AM/HF in public equities for the next 30 years, at least until 2050? (I understand this question has some challenges to answer)

Mar 14, 2018

Neural networks are not "like the human brain". Would be hard to make this assertion b/c ... we really don't know how the human brain works.

I can't possibly answer that last question so wont even try

Learn More

814 questions across 165 hedge funds. 10+ Sample Pitches (Short and Long) with Template Files. The WSO Hedge Fund Interview Prep Course has everything you'll ever need to land the most coveted jobs on the buyside. Learn more.

Mar 12, 2018

No, it will not make them irrelevant. It may make certain types of strategies irrelevant (or at least less relevant) but I am fully confident that human investors will learn and adapt faster than any AI we will create by the year 2050.

Mar 13, 2018

+1 SB, thanks for responding. What kind of strategies do you feel will be less relevant? Do you think fundamental value, activism, or long-term GARP strategies will be diminished?

Mar 13, 2018

At most the roles will change. New technology will replace/augment certain aspects of the research analyst role. It'll just make researchers more efficient. I think AI could help with reducing the number of companies to investigate.

Mar 13, 2018

Got it. If you don't mind, can you maybe specify which aspects of the role will be taken over and which will still be left largely to the discretion of the analyst?

Best Response
Mar 14, 2018

I can only speculate but I think the amount of reading/research will be heavily reduced. natural language processing algorithms might be able to identify the important aspects of 10Ks and 10Qs. Maybe AI/machine learning will be able to reduce the number of companies that an analyst would want to consider based upon some predictive machine learning algorithm that isolates the top N companies in a particular industry that will do well (or poorly) over some time horizon.

Once the investing horizon becomes too long, I don't think ML will be of much help. Maybe there will be certain tools that will become helpful but I don't think any significant aspects of a long-term value investment approach would be replaced by machine learning. Perhaps there could be some sentiment analysis tools on earnings transcripts to see how 'risky' the language a CEO is using?

There's a lot that people could try doing but I don't think the primary investment decision making process could be replaced by an AI/machine learning algorithm in the near future for the strategies that you mentioned.

Mar 14, 2018

@DeepLearning: Do you have any experience in applying ML to biopharma investing? I've been working with some PhD friends (in ML and computational biology) on a drug development platform that we think may also have value in an investing context. We've built a series of models that can determine (with a fairly high degree of accuracy) whether a compound will pass/fail in clinical trials (based on tox and efficacy signals). No worries if this is outside your wheelhouse, was just curious.

For context, my background is in computational neuroscience and healthcare VC.

Mar 14, 2018

Nope, no experience in the area but sounds pretty interesting! Would love to learn more.

Mar 14, 2018

Great getting your insights, +1SB.

Mar 14, 2018

I'm a data scientist at a BB and I write algorithms . . . that's all I have to say about that . . .

I think that the general perceptions that are being expressed here are true. Algorithms may not be as advanced as people make them out to be and there's definitely all kinds of gaps (knowledge gaps, data gaps, hardware gaps, etc).

However, where all of you should be concerned is the speed at which ML can learn and adapt to new information. THAT is where algorithms will truly take off and make an impact in all kinds of industries. The key to any algorithm is the depth & variety of data that you can feed into it. Sparse data leads to weak learning, deep data leads to better learning. And when an algorithm has a lot of data, watch out, because it can learn faster than any human can hope to achieve.

If you really want to get your mind around what ML can do, go find the whitepaper written by the Google researchers that solved the game of Go last year. No, don't read the fucking CNN article on it, read the actual whitepaper. The researchers talk about how they gave their algorithm very little seed information and then set it loose on Google hardware to play millions & millions of games in very little time. This kind of ML development is where the world is headed - small algorithms, given enough time and data to learn, become big algorithms.

So, coming back to the OP, it may be easy to poo-poo algorithms in their current state, I get that. But once they have enough data, they will grow and become something huge almost literally overnight.

Mar 14, 2018

Ok yes Google creating an algorithm that beats the best Go players is nothing short of amazing. But there is a HUGE caveat here that you are aware of I'm sure and that's the fact that Go is a perfect information game. What is a perfect information game? It is a game where you have all of the relevant information directly in front of you that you need to make the optimal move or decision.

Unfortunately, real life isn't like board games. The information you have is imperfect and you don't have all the information relevant to you. This is where ML/AI truly falls apart in its current state. The human brain (and other animal brains too) are phenomenal at modeling unknown/hidden information as well as imperfect information and making accurate inferences nonetheless.

It is super cool that the researchers at Google gave their algorithm very little a priori information on how to play the game (unlike IBM's Deep Blue in the 90s). However, the fact that Go and most other board games are perfect information games makes the feat pretty much incomparable to the problems that humans solve on a daily basis at work.

Mar 14, 2018

That's only partially true. Other games are being beaten too, like poker. Heads-up no-limit poker is basically solved. Limit, multi-hand poker is also effectively solved. Next up will be no-limit multi-hand poker - I expect that to be solved in the next 2 years or so.

But really, its not a point of perfect/imperfect games. Go was solved because it could create its own database of games/knowledge, in literally a blink of an eye. Someone in this thread mentioned that humans are still used for quant interviews - a deliciously ironic situation, I get it. But that's a problem that just lacks data, the depth just isn't there. But when it is there? Humans will be eliminated from that process overnight. The same thing will happen across other industries.

Mar 15, 2018

I agree that the issue is not one of perfect information (as you pointed out w/ your poker examples).

However, in the case of poker, we have direct access to the underlying data generating process (i.e cards are drawn from a fixed deck in each round). Access to the DGP allows one to effectively generate infinite data to approximate nash equilibrium.

What's the DGP for say, a company's earnings? It is based on the aggregate behaviors of all individuals, whose decisions in turn are driven by some very complex biological process. So at the moment, it is hopeless to simulate possible worlds in a structural way, as can be done in poker.

Therefore, economists/investors try to understand earnings and economics in a more top-down way. Looking at statistics, valuation ratios, management, competitive landscape, etc. But once you do this, the problem of non-stationary immediately comes up. In a non-stationary environment, humans can avoid overfitting to the data by applying their intuitions about causality, for example.

That's not to say that a ML system in principle couldn't do the same thing. But it seems to me that it would have to be approaching general AI at that point.

Mar 18, 2018
multiplex3:

I agree that the issue is not one of perfect information (as you pointed out w/ your poker examples).

However, in the case of poker, we have direct access to the underlying data generating process (i.e cards are drawn from a fixed deck in each round). Access to the DGP allows one to effectively generate infinite data to approximate nash equilibrium.

What's the DGP for say, a company's earnings? It is based on the aggregate behaviors of all individuals, whose decisions in turn are driven by some very complex biological process. So at the moment, it is hopeless to simulate possible worlds in a structural way, as can be done in poker.

Therefore, economists/investors try to understand earnings and economics in a more top-down way. Looking at statistics, valuation ratios, management, competitive landscape, etc. But once you do this, the problem of non-stationary immediately comes up. In a non-stationary environment, humans can avoid overfitting to the data by applying their intuitions about causality, for example.

That's not to say that a ML system in principle couldn't do the same thing. But it seems to me that it would have to be approaching general AI at that point.

Right. The DGP basically makes poker an unrealistic comparison. You can simulate everything. Excellent point on non-stationarity

Mar 14, 2018

I find it quite amazing how some journalist with near no experience in Finance / CompSci is so confident in himself that he can say that ML will make research analyst roles obsolete. Come on, there are industry professionals, that have far more inside information than you could possibly imagine, who take less binary stances on topics such as these. (I understand a lot of it has to due with "clickbaiting" titles of articles to generate site traffic and whatnot)

Will ML change / effect the role of an analyst in the next few years / decades? Perhaps

Could it make an analysts role obsolete by 2030? Again, possibly. But is that really going to make you completely reconsider your career trajectory? Doing so every time someone wrote an article like this would be crazy.

On a less "complainy" note I strongly agree with dazedmonk in that what the author keeps referring to sounds less like ML and more like AI. Plus I couldn't imagine anyone would feel comfortable with a computer making investment decisions (sort of hard to assign blame / responsibility).

Mar 15, 2018

It will be a tool. But a tool used for decision making I think. It removes cognitive biases which leads to poor decision making by humans

A.I. already showing promise at diagnosing disease better than "experts." Probably because decisions are based on objective stats rather than some convoluted human mind filled with past experiences.

https://qz.com/1130687/stanford-trained-ai-to-diag...
Same can apply to investment decisions. Will need a human touch for some data points that aren't as "hard" though.

Mar 15, 2018

Yeah I definitely agree that the most likely outcome is that ML becomes a useful tool to blaze through the more mechanistic analyses.

That article is really interesting. I do believe that diagnosing disease is far different than actually treating the problem / doing surgery / acting on it. Comparatively we could consider the "disease" in banking to be the research. Sure the ML / AI can do the research but at the end of the day I believe we will need humans to actually "do the surgery".

And yeah, as you and others have pointed out above there are definitely a lot of soft skills necessary in this line of work as well.

Mar 16, 2018

The big difference between Go or poker and the markets is that in the markets the rules change based on how the participants act. If there was some master AI strategy that always produced alpha every year AND could have massive capacity, it would take over the market and by definition stop working once it gets too big that it has no more dumb money to trade against. If everyone in the market is quant/AI that still doesnt change that its a zero sum game and after fees most of these players will underperform the index. There could be a scenario where quant strategies consistently beat humans, in this world they will take fund flows until a point where they are too big and can no longer outperform and the pendulum will swing back towards human fundamental strategies (again because no one can run a capacity unconstrained strategy that consistently outperforms in any market). To say that 100% of the market goes quant means nothing because it doesnt change the amount of alpha out there, there will always be losers to offset winners.

Mar 18, 2018

I generally agree with what has been posted here. There's a ton of hype about machine learning, but I believe it's highly unlikely algos will "take over" investing.

The issue is the overwhelming scarcity of useful data for a computer to use. The reason machine learning dominates short-term stock-market trading (HFT, intra-day, etc.), is because the relevant data for figuring out what moves stocks over short time horizons are things like stock momentum, correlations across stocks, order book data, trade sizes, etc. There are billions (trillions?) of relevant datapoints available, in well-formatted data-sets, for quants to use to develop trading strategies around. It's a near-ideal environment for machine learning approaches to investment. But it does not generalize at all to longer time horizons. And even in short-term trading, approaches tend to be about gaining "51/49-style" edges on the next move in a stock, not predicting big winners/losers.

For longer-term investments, there are definitely useful quant signals, arbitrages, etc. (such as the classic "value" and "momentum" strategies), but most information that is actually relevant for how a stock should be valued/should perform over time is sparse and poorly structured. Think about what tends to matter for stock market investors (and what moves stocks): financial releases, earnings calls, meetings with management, management presentations at conferences, diligence calls with industry experts, etc. None of this data is well set up for machine learning. Firms release 4 (badly formatted, inconsistently presented) financial statements each year. Huge amounts of relevant data is not included in the actual financials, but in commentary on the earnings call, non-GAAP company-specific operating measures in the MD&A of the 10-Q/K or the press release. And then there is the way management's tone/attitude changes from meeting to meeting, the industry color you get from your diligence calls, things like that. And there are also only a couple thousand public companies releasing such data points, which isn't really all that much from a machine learning perspective.

One can imagine a computer synthesizing such information, yes, but you're getting into the realm of science fiction and true AI, not machine learning algorithms as they're seen today. That being said, there's lots of room for AI to simplify things for analysts: generating financial models from reported financials, highlighting changes to risk factor in the 10-K, creating a readable summary of the key portions of a 10-K, identifying "risk phrases" in earnings calls. A lot of this stuff already exists, though I can easily imagine improvements. But this isn't the sort of thing that replaces analysts in the foreseeable future.

Taking a step back here, there are near-constant articles in the press about how machine learning is going to take over everything. I have a framework I like to fall back on to quickly assess the plausibility of such claims.

  1. Does the relevant (key word: RELEVANT) data exist at all to predict/accomplish a task systematically?
  2. Is there a vast (and I do mean: VAST) amount of this data, and is it available in some sort of structured/digitize-able format? You'd be amazed at how much data is needed for machine learning algorithms to work.
  3. Can more relevant data be generated and processed with relative ease? In board games, this is easy: a computer can just keep "playing" the game with itself over time until there's a huge database of games and strategies can "emerge," if you will. And even for self-driving cars, by putting more cars with sensors on the road, and prototype vehicles into play, it's possible to generate very rich data over time.
  4. Does the data have a relatively manageable number of dimensions of relevant data? The more types of data that are likely to be highly relevant/necessary for analysis, the tougher it is to build a model.

If the answer to these 4 questions is yes, then there is reasonable likelihood that the task can be automated. But always keep in mind that what we call "machine learning" isn't at all like "AI" in the popular imagination. The algorithms tend to do something akin to optimized trial and error. They throw variations of predictive models at a huge data set until they come up with something that looks like it "works." It's "dumb learning," in some sense. To get any sort of result requires incredible amounts of data, and as the complexity of the problems rise, the amount of (relevant) data required for a useful model rises in a non-linear fashion (read: rises massively).

In college I did a fair amount of machine learning coursework, though by now I've forgotten pretty much all of it. However, I do remember one thing that struck me was that human intuitions of what is "hard" or "easy" to do don't line up well with what is "hard" or "easy" for a computer to do. And I've noticed that this tends to lead journalists, commentators, and hypesters to point toward some success of a computer at one task to suggest that another success is right around the corner, even if the logistical issues involved in gathering and using the relevant data are radically different.

Put another way: imagine if you gave a digital calculator from the 1970s to a man from the 18th century, and explained to him that it was a form of machine intelligence. Imagine his shock as it performed calculations instantly that would take him hours, and performed them flawlessly. What would his understanding of machine intelligence be? He might well assume that machines in our era could easily perform any tasks that humans could do, or that such a development must be right around the corner, given the impressive power of this machine. He would have no intuition at all about its limitations, or the limitations of the era's technology. Today's public commentary on machine learning feels a bit like this to me.

Mar 18, 2018

I think the consensus is that Machine Learning and other high level computation techniques will greatly aid in certain tasks. However, any claims that such technologies will totally overtake technical jobs are usually made by those with little to no understanding of what the underlying techniques are...

Mar 18, 2018

1-Click to Unlock All Comments - 100% FREE

Why do I need to be signed in?
WSO is a knowledge-sharing community that depends on everyone being able to pitch in when they know something.
+ Bonus: 6 Free Financial Modeling Lessons with 1-Click Signup ($199 value)