Sentiment Analysis Still Can't Reliably Catch Sarcasm, We Tested It Again

Every year or two I run the same informal test: take a batch of genuinely sarcastic customer reviews — 'oh great, ANOTHER outage, love that for us' — and run them through whatever sentiment tooling is currently fashionable. This year it was an LLM-based classifier instead of the usual fine-tuned BERT variant. The result was the same as every previous year: roughly a third of clearly sarcastic negative feedback got scored as positive.

Why this keeps not getting fixed

Sarcasm isn't a vocabulary problem, which is why throwing a bigger model at it helps less than you'd expect. It's a pragmatics problem — the words are positive, the intended meaning is the opposite, and resolving that requires context the model usually doesn't have: tone, prior conversation history, sometimes just shared cultural knowledge about what counts as a believable compliment in context. A bigger LLM is better at a lot of things. It is not automatically better at inferring that someone doesn't mean what they literally said.

Sarcasm isn't a scaling problem. Bigger models get better at language, not automatically better at knowing when someone means the opposite of what they said.

What actually helped, in a limited pilot for one client, wasn't a better sentiment model at all — it was giving up on pure sentiment scoring for ambiguous cases and routing anything with high linguistic markers of sarcasm (exaggeration, excessive punctuation, certain idioms) to human review instead of trusting the automated score.

• Don't trust an aggregate sentiment accuracy number without specifically testing on a sarcasm-heavy subset — it's rarely reported separately.
• LLM-based sentiment tools are meaningfully better than older approaches on plain language, but the sarcasm gap hasn't closed proportionally.
• Routing ambiguous cases to human review is still more reliable than trusting a model to resolve pragmatic ambiguity at scale.
• Be skeptical of any vendor sentiment demo that doesn't include a sarcastic example — it's the easiest failure mode to hide.

If a client's use case depends on correctly scoring sarcastic feedback at scale — and plenty do, since sarcasm is where a lot of the most urgent complaints live — the honest answer right now is still: budget for human review on the ambiguous tail. The model will handle the easy 80% fine. It's the loud, sarcastic, high-signal 20% where it keeps failing in the same way it always has.