Paedophilia doesn’t violate the rights of the child, acting on paedophilic intentions does. The distinction is important to the argument. No harm is caused by a sexual preference. Only any actions that stem from it. That’s why you (theoretically) could argue for paedophilia whilst also condemning child abuse. It’s a valid position to take – but not a popular one.