Accepted FT AM offer, but want to pursue IB recruitment

In a dilemma. Recently finished my junior internship at a large AM firm and accepted the FT offer. However, I spent much of the summer recruiting for FT IB roles and still really want to break into the industry. Ultimately, I decided it was better to accept the FT position to make sure I have a fail-safe, as NYC's job market seems brutal right now and I believe that I will be able to lateral into the industry given enough time. 

This is all to say that while I fully plan on continuing to network throughout the year, I'm unsure about actually applying and interviewing for IBD positions. For one, the firm I'm at took me on with no finance experience (go to a target school, but was underqualified relative to my peers), and it feels shitty to reneg on them when I really liked everyone I worked with. Additionally, wouldn't want the bank to call my company and find out that I'm doing this, and then end up completely screwed. 

Many of my friends in the industry have encouraged applying and reneging, as IB is worth it. I'm wondering if it would really make that much of a difference to wait a year before applying with more experience under my belt, or if lateraling would be even harder in the future compared to still being in college. Would appreciate any advice on handling this and how you guys would perceive the risk/reward of it.