Does intent matter?

It was recently inferred by someone that because I gave a man a compliment on his clothes, he and I were having an affair. What didn’t matter in this situation was my intent. I would never have an affair with anyone except Tim (who I’ve been with forever and is my other half), and why should they care anyway? Don’t they have their own lives to focus on? So, now I wonder: “did they say this because they were bored? Because they’re not happy?”