IB SA --> REPE, Brokerage, Secondaries FT

I recently accepted a SA job at a BB. It is bittersweet: I love the people and the opportunity (I won't be the guy to be ungrateful). With that all said, I really am passionate about RE, and, during this recruiting cycle, I have tried to target RE-focused firms, but this opportunity came up and I couldn't pass it up. I do intend on recruiting again FT for REPE, brokerage shops, or secondaries (think Park Hill, Evercore, etc.). What should I do? Is this possible? If so, how difficult is it? I appreciate any info you guys can provide me.