Paedophiles are protected by this principle because being a paedophile is not an act of harm in and of itself. The harm only comes when someone acts on paedophilic intentions — you could defend paedophiles under this argument if they were to accept their sexuality was only a state of being and never a state of action.
There’s a great play called ‘The Nether’ which goes into a lot of the morality around this subject matter. I should state for clarity that I’m not the author — though I wish I was, the play is excellent and I would imagine right up your street. :o)