mlsanityData to 7 Oct 2026

Towards Understanding the Robustness of LLM-based Evaluations under Perturbations

arXiv 2412.09269 ↗First posted 12 Dec 2024cs.CL cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

Traditional evaluation metrics like BLEU and ROUGE fall short when capturing the nuanced qualities of generated text, particularly when there is no single ground truth. In this paper, we explore the potential of Large Language Models (LLMs), specifically Google Gemini 1, to serve as automatic evaluators for non-standardized metrics in summarization and dialog-based tasks. We conduct experiments across multiple prompting strategies to examine how LLMs fare as quality evaluators when compared with human judgments on the SummEval and USR datasets, asking the model to generate both a score as well as a justification for the score. Furthermore, we explore the robustness of the LLM evaluator by using perturbed inputs. Our findings suggest that while LLMs show promise, their alignment with human evaluators is limited, they are not robust against perturbations and significant improvements are required for their standalone use as reliable evaluators for subjective metrics.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 16 of its 26 references that are in the map. First posted on arXiv 12 Dec 2024.

Builds on

16 of 26 references.

References · 16

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  8. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  9. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  10. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  11. 2021Generated Knowledge Prompting for Commonsense Reasoningcited by 213
  12. 2020Re-evaluating Evaluation in Text Summarizationcited by 144
4 more
  1. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  2. 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
  3. 2024Towards Understanding Jailbreak Attacks in LLMs: A Representation Space Analysiscited by 61
  4. 2024Scaling Trends in Language Model Robustnesscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 7.

10 papers.

  1. 2026The Fragility Of Moral Judgment In Large Language Modelscited by 8
  2. 2025Framing the Game: How Context Shapes LLM Decision-Makingcited by 4
  3. 2026Evaluating Robustness of Large Language Models in Enterprise Applications: Benchmarks for Perturbation Consistency Across Formats and Languagescited by 4
  4. 2025DialogGuard: Multi-Agent Psychosocial Safety Evaluation of Sensitive LLM Responsescited by 3
  5. 2025I Spy With My Model's Eye: Visual Search as a Behavioural Test for MLLMscited by 1
  6. 2026Evaluating LLM-Based Grant Proposal Review via Structured Perturbationscited by 1
  7. 2026From Performance to Purpose: A Sociotechnical Taxonomy for Evaluating Large Language Model Utilitycited by 0
  8. 2026Learning When to Trust in Contextual Social Banditscited by 0
  9. 2026Prompt Perturbation for Reliable LLM Evaluation over Comparison Graphscited by 0
  10. 2026Noisy Test-Time Reinforcement Learning for Code LLMscited by 0