I actually get more down votes on comments I make rather than stories. Is this the place for that too?

Yes there is. It's called Downvoted With No Comment.

In all seriousness, it happens all the time. Some people like to troll and others just don't like certain users.

5476504 How do I know I'm not being stalked? It seems that every comment I make is down voted, no matter what I talk about.