Any way to navigate negative informal reference checks?

Going through the buyside recruiting process after some time away from banking as an analyst (did something else in between), and am running into situations where I suspect networking calls or interviews don’t go my way because of informal reference calls to my old staffer. To be clear, I was consistently middle / upper middle middle bucket in banking but was on a deal or two with this staffer where I didn’t perform my best and is likely coloring his / her perception of me. Is it worth reaching out to this staffer to avoid getting burned in the future? What do I say? Would really appreciate any insight