The problem with that is the definition of 'others' and 'harm' argument. There are a great many people who would be devastated to see me eat a ham sandwich because it causes them distress. I could eat beef in India, but should I? To what extent should I bow down to the culturally accepted standards of others? Or is it a negotiation. And by 'others' do we mean other humans, or other entities? Nature is a battle of animals doing harm to each other across the course of survival. Does my right to harm a pig outweigh the desire of the pig to live a free and happy life. Some might argue yes, some might argue no.
And then we come to harm... do we mean harm to an individual? It might cause the world a huge economic shock if the West were all to suddenly go to Vegetarian. Entire livelihoods would be destroyed, industries would collapse. Then, without a need for them, we might slowly wipe out entire species of domesticated animals. Or if not wipe them out, certainly endanger them. Some species of animal have done very well (evolutionarily speaking) by being our food. Could the domesticated cow become endangered? And if so, isn't that harmful?