There are villains I praise as excellent writing because they are threatening and evil without ever being too over-the-top to write off as just a story. In this category, I'd include Lord of the Flies' Jack and 1984's O'Brien. Any comeuppances they may get are double-edged and unsatisfying by design, but I end up devouring the stories anyway just to see how the heroes cope with and survive them... if they do.
However, the villains I really love- the ones I read again and again, that I take notes on, that I might have a shelf in my house dedicated to, trend towards being morally gray, if not straight-up morally blue-and-orange. They have motives I can understand even when they take them in the strangest of directions, unique methods, and a whole lot of presence. I know I'll forget somebody, but a few of these guys include...
The Terrible Trivium, The Phantom Tollbooth. A soft-spoken monster who confronts bypassers on the road, solicits their help with a mix of flattery and affected vulnerability, and then literally bores them to death. Aside from the way he uses manipulation and flattery rather than brute force, dude weaponized boredom and made intense reading out of it. How cool is that?
Sir Francis Varney, Varney the Vampyre: An undead robber and con-artist, charming without being scrubbed or idealized, Varney is a villain... but because of what he does with his vampirism rather than the vampirism itself being shorthand for evil. He's not hard to sympathize with, as he genuniely regrets it when he goes too far, but he's never romanticized.
Erik, The Phantom of the Opera. The threat of losing his favorite diva drives Erik to extremes that all but ensure the very outcome he intends to prevent. He's not really cruel, just so obsessed with getting Christine to stay he doesn't think two minutes beyond that goal (hence the ending where he realizes what he's doing and stops pursuing her of his own volition).
Xayide, The Neverending Story. Tries to exploit Fantastica's rules to replace the Childlike Empress, using outsiders to gain entry to the Ivory Tower, even though the things they have to do to get there will erode their minds and trap them in Fantastica forever. She's such a legitimate threat the heroes are still suffering the effects of the plan she's put into motion after she dies.
Non-book:
Kefka Palazzo, Final Fantasy VI. For all his ostentation, the thing that really set Kefka apart for me is his progression. He starts out a funny goldfish poop character with brief moments of horrible cruelty, but as the game goes on, those moments of cruelty get longer and closer together, the jokes that punctuate them growing increasingly mean-spirited.
Cobra Commander: He started out wanting revenge for the death of his brother, and it spiralled so far out of control, he couldn't stop it if he wanted to. Admitting that would be tantamount to defeat, so he goes with it, and he brings such a heaping helping of eccentricity watching his plots unfold is equal parts watching a genius at work and watching a train wreck.
