Turning Down $500k+ VP Offer in Niche Field With No Other Options?

I'm in my late 20s / early 30s and have 7 - 10 years experience investing in a niche real asset class (think equity investments in the assets that would be securitized into esoteric ABS), most recently as a VP at PE-backed specialty finance platform.The sponsor has decided to cut exposure to the strategy and I've been told I will be let go in the next 90 days. 

I'm at a turning point and would prefer to pivot towards broader investment seats if possible. The main problems are that I'm most marketable in this industry (despite what I believe are transferable investing skills and strong credentials) and most of the roles in this field are in NYC (vs. I'm currently based in a T2 city on the West Coast with a long-term partner who would be unlikely to fully move to NYC).

I am in advanced conversations with a Japanese investment bank that reached out to me for a VP role originating structured financing in this sector and expect to receive an offer for ~$500k total compensation. The catch is I'm not passionate about the industry, prefer actual investing work vs.coverage / lending work, and the role would be 5 days per week in office in a city I'm not thrilled about and 6+ hour flight from my partner. I worry that this role would be well paying, but would remove any flexibility from my life and only open up doors in this particular industry. If I were going to move to NYC, I feel like I would prefer something on the buyside or with higher upside / future optionality. 

Has anyone been in a similar situation or have thoughts on this? This process has felt rushed because the bank reached out to me first... I have a high-paying offer in essentially in hand in a field / location I dislike and little insight into my ability to find a similar role (or even Associate / Senior Associate) elsewhere.