Worried I get fucked over

I recently quit my summer internship after signing for it last month. My boss didn't take it too well and usually holds grudges. Almost everyone I talked to told me that I made the right decision, but now I'm worried that he contacts all of his connections to screw me over. I had a legitimate reason. Should I be worried? What's the worse that can happen? He is decently well connected I believe.

Edit: I'm a freshman. I'm worried about junior summer recruiting next spring.

Thanks