Any downsides of joining a family office

This is anecdotal - I'm at a top 3 BB bank and have seen hires from people who went from AWM into a Family Office and now they're back. I don't think you'll ever really completely shut doors as long as you continue to be involved in interesting opportunities and have grown in value of what you bring to the table.

I would also take the opportunity to really dig into the people at the family office you're thinking of joining. The family office space is a spectrum of knowledge and complexity. You want to ensure that you're going into an office with a team that has a lot of experience so that it guarantees you'll continue to learn, not just from the work but also from them. 

Location could matter as well depending on state tax regulations. Which state / region is it located in?

In a family office seat right now, one thing I overlooked during my recruitment that I wish I would have known more about was the level of involvement of the family. I spent a lot of time vetting the team I would be working with and making sure they were up to par, but it turned out that the family likes to get involved in the investment process and they are lackluster at best when it comes to their level of sophistication. I’m looking at going back to a larger fund but have found it to be slightly more difficult than when I was at my previous company because the FO I’m at is relatively unknown.

I’d make sure you think about their involvement and if their potential lack of sophistication can add any headaches to your daily life. Hope this doesn’t scare you away from a good gig, there are also a lot of cool perks to a FO as well.

I was in LO AM and moved to a small FO. For me the job lacked structure and I moved back to LO AM, so you can definitely move back.

As already said here, every FO is different and it is important to do your research. On one hand by talking with other people who may know the outfit, on the other hand by talking clearly and honestly about your long term function and long term development (or at least plan). It is important to find out whether the family has a say in investments (if yes, do they actually add value to the investment process or do they just stick their nose into everything). If it isn't the family running the show, who is and what makes him/her good? How much can you learn from him/her. I have seen FOs with top people and very well organized teams. On the other hand, I have seen some that were run horribly by completely unqualified people. I'd also add that the infrastructure at some places is quite bad, meaning you may be the guy fixing/maintaining printers, computers, Excel sheets and databases.

Finally, I can tell you that even when working with well known and respected (from the outside) principals, you can actually be quite surprised how these people interact and run their FOs. Some are great, funny and offer huge learning experiences, while others are super erratic and disorganized.