What is the case against "real" ethnicity-based diversity programs?

Earlier in my career, I was told that a director in my team was against me being hired due to the fact that I am black. He believed that it suggested that I was lazy and less intelligent. The CEO of the place ended up making the executive decision to hire me. I performed quite well at the place and consistently received solid feedback. I later ended up moving from the "no name" boutique to a stronger firm (PJT/EVR/LZR). At my initial employer, an associate also made jokes (?) to me about not hiring specific applicants due to their heritage/surname. Apparently, I was one of the "good ones".

There are more stories I have experienced that highlight that people from certain backgrounds have an objectively harder time getting into IB irrespective of qualifications.

Given that there are people with a racially-influenced recruiting process, does it not make sense for there to be programs in place that make it more likely for people with higher probabilities of discrimination and negative stereotypes to get a fair shot? What is the argument against having quotas that make this possible? Alternatively, what would be a better solution?

FYI: I have never knowingly benefitted from diversity programs.