Contents
In brief
ROUGE and BERTScore remain common for LLM evaluation, yet this work shows they often rate mutually contradictory texts almost the same as faithful answers. MATCHA rewards proximity to a gold reference while penalizing an adversarial “opposite” rewrite—and aligns better with humans on eight public benchmarks.
What they studied
The team compared token-overlap and embedding metrics against MATCHA’s dual view: (1) closeness to the reference, (2) distance from a generated counterfactual contradiction. Tasks span QA, summarization, NLI, captioning, STS, plus TruthfulQA without local metric training.
Key findings
- Mainstream metrics inflate scores for answers that negate the reference.
- On TruthfulQA, human agreement improves by 18.38% over ROUGE-L and 20.82% over BERTScore.
- Against 23 embedding models used BERTScore-style, MATCHA best separates correct vs incorrect statements from a reference alone.
- Open source: github.com/Siran-Li/MATCHA.
What this means for developers
- RAG/chat CI: don’t rely on ROUGE alone—add contradiction-aware checks vs gold snippets.
- Prompt regressions: track counterfactual distance, not just average similarity.
- Eval harness: use MATCHA beside LLM-as-judge for cheaper pre-release gates.
- Customer reports: high BERTScore ≠ factual alignment—say so explicitly.
Limitations
Needs a gold reference; open-ended chats still need judges or task-specific rubrics. Counterfactuals are synthetic—critical domains still need human spot checks.