Career bankers, upgrading banks worth it?

I recently made the switch back to banking as an Associate from a small buyside firm. I finally broke into a large balance sheet bank in Europe (think Santander / Nomura / RBC) after previously only working at tiny LMM banks.

I’ve pretty much decided that I want to be a career banker. Hence my question:

Is it worth lateraling upstream from a solid — but not elite — bank? Especially if the current bank is known for promoting from within and offers good internal mobility?
 

I feel like I’ve finally found what I want to do long term and I don’t care about exit opportunities anymore. At the same time, I’d love to maximize my long-term earnings potential.

My concern: if I stay here for ~2 years, build a good reputation, and make VP, would it still make sense to try upgrading to a BB? My current desk doesn’t really exist at boutiques, and if I move later I might have to start over with no goodwill in a stronger, more competitive team.

Another factor: my current bank’s location is ideal for me, and every other city I could move to would be a bit of a lifestyle downgrade — though I’d be willing to move again if it made a real difference.