What is the career risk of entrepreneurship after 1 year of work experience?

I've come to the conclusion that the IB / PE route is not for me. I think even if I ended up at a megafund I would hate my life with the long hours and work that I didn't find fulfilling. I think ideally, starting and running my own business would bring me the most fulfillment and be the most exciting, even if im making a little less than if I stayed in IB. Say I take 2 years to try to start a business and fail. How much career damage would be done if I then tried to go back and get a job? I've also considered working at 40-50hr a week job and getting a business off the ground on the side before quitting to mitigate the risk.