mlsanityData to 7 Oct 2026

A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarios

arXiv 2408.01963 ↗First posted 4 Aug 2024cs.CL stat.APcited by 8

Not on the map. Why?

Abstract (arXiv)

We evaluate the robustness of several large language models on multiple datasets. Robustness here refers to the relative insensitivity of the model's answers to meaning-preserving variants of their input. Benchmark datasets are constructed by introducing naturally-occurring, non-malicious perturbations, or by generating semantically equivalent paraphrases of input questions or statements. We further propose a novel metric for assessing a model robustness, and demonstrate its benefits in the non-adversarial scenario by empirical evaluation of several models on the created datasets.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 14 of its 39 references that are in the map. First posted on arXiv 4 Aug 2024.

Builds on

14 of 39 references.

References · 14

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  4. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  5. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  6. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  7. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 299
  8. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  9. 2024Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvementscited by 82
  10. 2022Prompt Consistency for Zero-Shot Task Generalizationcited by 66
  11. 2022Measuring Reliability of Large Language Models through Semantic Consistencycited by 38
  12. 2024Improving the Robustness of Large Language Models via Consistency Alignmentcited by 36
2 more
  1. 2023Predicting Question-Answering Performance of Large Language Models through Semantic Consistencycited by 28
  2. 2024Unitxt: Flexible, Shareable and Reusable Data Preparation and Evaluation for Generative AIcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 4.

8 papers.

  1. 2025On the Robustness of Agentic Function Callingcited by 17
  2. 2026Same Meaning, Different Scores: Lexical and Syntactic Sensitivity in LLM Evaluationcited by 4
  3. 2025The Non-Determinism of Small LLMs: Evidence of Low Answer Consistency in Repetition Trials of Standard Multiple-Choice Benchmarkscited by 3
  4. 2026Robustness as an Emergent Property of Task Performancecited by 2
  5. 2025Improving Score Reliability of Multiple Choice Benchmarks with Consistency Evaluation and Altered Answer Choicescited by 1
  6. 2026Robust Explanations for User Trust in Enterprise NLP Systemscited by 1
  7. 2025Evaluating perturbation robustness of generative systems that use COBOL code inputscited by 0
  8. 2026Reliability Scaling Laws for Quantized Large Language Modelscited by 0