It's because they don't deliver on their promise.
I've read horrible clop, and I've read Fall of Equestria stuff, and I've read and reviewed things that were barely readable because of of horrid grammar, and as a rule I don't downvote them. To each his own, it takes all kinds and all that. Mostly I don't feel myself good enough to judge that the fic is objectively bad, merely that I don't like it.
Today I downvoted three fics. All of them not the worst on the website, perhaps even in the upper half.
I downvoted them because they don't deliver on their promise. Because they started well, and then they went on and crushed me with a horrid disappointment.
Is it a fair thing to do? I do not know.
But I would much rather have a fic which is crap and I know it when I begin reading it, than have a decently written turn Jar-Jar Binks on me when I've already invested into the story.