You have mistaken considering what other people say as something important to consider. How YOU feel about something and how OTHERS feel about you are two different things. In a liberal society we regularly try and balance the rights of individuals to co-exist, but co-existence does not mean utopia. As a general rule (and I appreciate it is a general rule) most people can integrate into a pluralistic liberal society - others might not like it, but that in and of itself does not require a response. Many groups have a tendency to rile themselves up in little pockets of activism which is tilting at windmills.
I would further argue that in a capitalist society such at this one, the only real facet of privilege with any degree of real relevance is money. Throw enough money at a problem and it tends to not be a problem anymore. :o)