Who are the investors in hedge funds and why do they do it?

The goal of some hedge funds isn't to beat the market. Furthermore, higher returns doesn't mean a better hedge fund! The goal of a hedge fund is to hedge against market downturns to ultimately protect capital and generate returns that are uncorrelated to the market whatever the market condition. For example, fund ΑΒC might have a mandate that is 'we are going to generate sp500 like returns ie. roughly 10% however with half the volatility and be totally uncorrelated to the market'. That to an LP is attractive because whilst the market might beat the HF over a long duration of time, there are periods where the market could have significant drawdown whilst fund ΑΒC is still outperforming. Over a short time horizon it is possible that an investment in a HF would be much wiser than the market itself as whilst the market could extremely choppy, HF's can still generate returns as per their mandate. On the flip side, there are funds that do return much higher than the market (RenTech's medallion fund being one of them) and so if LP's wanted a more aggressive strategy then they could opt for that sort of fund. 

Archegos was a family office which is totally different to a hedge fund - that fund was essentially a one man band who over leveraged in naked exposures and ignored risk limits. This sort of greedy behaviour isn't indicative of the whole industry however there are certainly others who are probably too highly leveraged for their own good. With regards to WSB retail trader trend and GameStop, there were definitely HF's on the other side of the trade (ie driving the price higher - https://www.wsj.com/articles/this-hedge-fund-made-700-million-on-gamest…). I personally don't see retail to pose any threat to HF's, I mean the algorithmic HF's already front run their trades and make a killing of them anyway but even moving forward, in coming years I think the retail euphoria in the market now will die down after we get a sizeable enough correction in the market. Unfortunately high retail engagement and euphoria is a good contrarian indicator of the market topping off like the dot com bubble when everyday people were loading up on tech and even the years leading up to the Great Depression where newspapers where putting out adds to encouraging people to buy stocks. 

I am currently writing my final thesis on HF activism, so I can point you to a few papers that give a great intro to HFs in general. The opinions from professionals here are absolutely great and important, but there is some empirical work that gives you a wider sense of why/which investors allocate to HFs and how the return characteristics are in the cross-section (despite all data issues).

Agarwal, Naik, Fung & Hsieh, and Ackermann have done some good work that you can look up on Google Scholar if you want to invest the time to get deeper into it. If you can't get the pdfs off of Google Scholar, shoot me a dm I have them downloaded. 

Bit odd, anyone who wants to work at a HF down the line would think this way/ask this. Warren example was over 20 years not 1 year outlook that was his point. Warren is basically saying "ignore the vol short-term" long-term the market as a whole creates value. Everyone in finance is going to say this and truly we should otherwise we all got no jobs.

Now, why hedge funds exist. Cause within that twenty year span this is a wide range of moves/changes which are hard to keep track of or hard to invest your time to specialize in a lot of them. That is how Alpha is created. Today we have more funds than 10 years ago which means we will have more losers and more winners, again definition of Alpha.

Not sure why we keep picking on Archegos. Sure the guy is a massive shady mofo and should possibly be in jail (but blame the law and government). But if you search twitter his actual strategy was not that wild, I bet over the 3 year span hes not net down a shit ton and the banks allowed him the insane leverage remember they "margin called him" otherwise I am sure Hwang would have doubled down again. His actual strategy was not that bad truly just execution here. Ohh lastly as mentioned was mainly his own money...guy had enough yachts wanted to chase Elon and Bezos.

Marks. 

The reason is that Warren Buffet, you and I are targeting a total return i.e., what is the best way to get the maximum amount of money at the end of 20 years if I just put it into a single fund and leave it. The clear answer to that has been to put it into an S&P500 index fund because despite the large volatility, you will earn a higher return in the long run over any other type of fund. However, pension funds, insurance companies and possibly sovereign wealth funds depending on their mandate, do not target a total return without worrying about volatility, as they are focused on their liabilities that they have to pay out, and so the sequencing of the returns actually does matter. By way of example, say a hedge fund manager returns 5% every year, while the S&P returns 10% every year but drops by 50% once every 20 years:

1. You invest $100 with both the hedge fund manager and the S&P500 index fund. At the end of 20 years, the hedge fund investment is worth $265, while the S&P investment is worth $306. Whether the S&P500 drops 50% in the first year, the last year, or somewhere in the middle, it doesn't matter. You can check this yourself using Excel and actually encourage you to do so.

2. Now say you are a pension fund that has $100bn in assets and needs to pay $5bn in pensions every year into perpetuity. With the hedge fund, you are covered - every year you get $5bn, every year you pay out $5bn. On average, the S&P will give you a higher return each year, however, if the S&P drops 50% in the first year, you are now left with $50bn of assets, and still have to pay $5bn out, and therefore left with $45bn. A 10% return on this will not even cover the $5bn you have to pay out every year, and so you will slowly erode your asset base over the next 20 years.