I agree. And if skin colour were the only determiner of privilege then you'd be correct... but it isn't. A white homeless person might have more white privilege than a black lawyer, but trying to convince the world that the black lawyer is the one being disadvantaged by society is a tall order.
You cannot separate all facets of identity out and make any one of them the single determiner of privilege. Doing this is basically a way to get disadvantaged people to fight amongst themselves and progress nowhere.
I make the argument against identity politics in this article. I think it's a the worst thing to happen for BIPOC people in a long time and will lead pretty much to failure and more racism at every level. https://medium.com/lucid-nightmare/what-we-all-lose-when-society-plays-identity-politics-397dcb896d5b