I think the trick here is to remove all identity from applications - legacy placements shouldn't be accepted any more than Affirmative Action should be. Submissions to university should be entirely blind, with places awarded on academic merit and sorted through a complex double-blind algorithm based on geographic averages. You seem to be simultaneously arguing that removing AA is a bad thing, but also that it won't be having much of an effect if its removed... so I'm confused about the argument you're making here.

Social tinkering with selected equality of outcomes has always been the problem. Society should always strive to sort equity over equality. If you cancel out all the variables (through blind selection and algorithmic interpretation the university is unaware of) you'd be left with the right students in the right places. The equity principle needs to be put in place - if you're a rich white kid from LA and you get a high GPA, or you're a Black kid from the mean streets of Detroit and your GPA is a little lower, but still way above the average for your area, you get the place. This is not the same as saying 'you must take 25% Black students'.

Universities should find out the ethnicity and gender makeup of its student body on day one. I suspect, if done properly and AA + legacy placements are revoked, you'd get a decent spread of human beings across the race and gender spectrum, on account of neither melanin nor genitals being related to intelligence.