Is switching groups after an internship too risky?

I’ve seen a lot of posts about people saying that they want to lateral to a more prestigious group or try to use the summer to network and go to a higher tier bank, and I always wonder, couldn’t that axe your possible return offer? This past summer I did an internship at a BB (not top 3 but decent for sure) in a Middle office role that works with IB, and the advice I always received was get the return offer and then lateral once you hit full time after an appropriate amount of time in that role. I could just be dumb but I’m worried about young bright college students screwing themselves out of a good job trying to outdo their current spot. Idk I could be wrong but it seems incredibly risky to me. Any thoughts? I may also be lucky because I loved my job/group and I understand sometimes you need to GTFO

Note: I still networked at my bank and my group was chill about it but I never came off as wanting to be out of my group or trying to jump banks