What would happen if a villain called Fluttershy's bluff? You know, she says something like “Why don’t you pick on someone your own size?” or “You do not hurt my friends!” and they say “Oh yeah? And what are YOU going to do about it?!” or just pull a Moe from Calvin and Hobbes and say something like “They’d hit back” or “You ARE my size” or “But it’s fun.”

Well, Fluttershy's Stare is something that only a few (namely Discord) can resist.

7865184
Has she ever used it on someone who isn’t seen as an animal or scary in the show?

7865461
Technically (based on what you define as animal or scary in MLP) theirs Razor(the dragon from dragon-shy episode) who got both stair and yelled at by shy to the point he cried. (Being a dragon that kinda awesome for shy). No other instance comes to mind apart from discord.