Getting the VP Promote at the "Wrong" Firm

For context, I'm a sr. associate at the VC/Growth shop. I'm well liked at my group and my MD just gave me the nod that I'm looking at a VP promotion at the turn of the year. I'm grateful to be in my seat, but I worry that as I take the promotion and spend more time here that I will get stuck and have little ability to pivot if I want. I will give some of the pros and cons, but I would love insight from anyone that has been in the VC space a bit longer. 

Pros: ~$320K TC at reasonable hours (40-60). We get to invest in some of the top vc-backed names given the influence of some of our connections, so I have some "hot" AI names on my resume that bode well for recruiting. Everyone on my team is a genuinely good person that respects me and there is very little toxic behavior. I have a very reasonable path in the next 5+ years to stay stable and make solid money while starting a family and focusing on other elements of life etc.

Cons: Narrow scope of investment opportunities, which makes the job feel less like investing and more like a corporate processor. I don't want to get into details, but we are very constrained on what we can look at. My group isn't well known and doesn't have the best reputation, so associating myself with them for the long term could hurt my potential. Although my teammates are great kind people, I don't feel like I am learning what I need here to be a great investor – most don't have prior investing experience. I do genuinely love investing, but we only get to do 1-3 deals per year, and I feel like a high % of my time is spent doing admin work and non-sourcing/execution. I also feel like my talent/excitement is being underused here.

If I stay and get the VP promote and put my head down for another year or two, will that severely limit my optionality to leave? I've heard the VC hiring scene is pretty brutal right now.