No one is bothered by people who have SSA but never act upon that attraction, never advertise that attraction, never publicly discuss that attraction. The idea that anyone holds anything against anyone else who has SSA is a fig leaf, red herring and strawman.
The issue, and only issue, is those who any upon said attraction. Or parade it. Or try to normalize it.