mlsanityData to 7 Oct 2026

Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metrics

arXiv 2505.18658 ↗First posted 24 May 2025cs.CL cs.AI cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have emerged as a promising cornerstone for the development of natural language processing (NLP) and artificial intelligence (AI). However, ensuring the robustness of LLMs remains a critical challenge. To address these challenges and advance the field, this survey provides a comprehensive overview of current studies in this area. First, we systematically examine the nature of robustness in LLMs, including its conceptual foundations, the importance of consistent performance across diverse inputs, and the implications of failure modes in real-world applications. Next, we analyze the sources of non-robustness, categorizing intrinsic model limitations, data-driven vulnerabilities, and external adversarial factors that compromise reliability. Following this, we review state-of-the-art mitigation strategies, and then we discuss widely adopted benchmarks, emerging metrics, and persistent gaps in assessing real-world reliability. Finally, we synthesize findings from existing surveys and interdisciplinary studies to highlight trends, unresolved issues, and pathways for future research.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 156 of its 184 references that are in the map. First posted on arXiv 24 May 2025.

Builds on

156 of 184 references.

References · 156

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2013Intriguing properties of neural networkscited by 5,114
  5. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  6. 2019Language Models as Knowledge Bases?cited by 1,922
  7. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  8. 2024A Survey on LLM-as-a-Judgecited by 1,384
  9. 2021Generalized Out-of-Distribution Detection: A Surveycited by 796
  10. 2023Large Language Modelscited by 774
  11. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  12. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
38 more
  1. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  2. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  3. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  4. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
  5. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  6. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 329
  7. 2021Recent Advances in Adversarial Training for Adversarial Robustnesscited by 276
  8. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  9. 2023Certifying LLM Safety against Adversarial Promptingcited by 248
  10. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  11. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
  12. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  13. 2023Large Language Models in Finance: A Surveycited by 223
  14. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 208
  15. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  16. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 200
  17. 2023Mitigating the Alignment Tax of RLHFcited by 181
  18. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  19. 2022A Review on Language Models as Knowledge Basescited by 156
  20. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solverscited by 155
  21. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 150
  22. 2020Understanding the Failure Modes of Out-of-Distribution Generalizationcited by 146
  23. 2020Robust Pre-Training by Adversarial Contrastive Learningcited by 144
  24. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  25. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  26. 2019End-to-End Bias Mitigation by Modelling Biases in Corporacited by 127
  27. 2025Why do LLMs attend to the first token?cited by 122
  28. 2020Maximum-Entropy Adversarial Data Augmentation for Improved Generalization and Robustnesscited by 121
  29. 2025Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonablecited by 120
  30. 2023Chatlaw: A Multi-Agent Legal Assistant based on a Role-Aligned Mixture-of-Experts Architecturecited by 119
  31. 2023Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Modelscited by 117
  32. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  33. 2024Granite Code Models: A Family of Open Foundation Models for Code Intelligencecited by 116
  34. 2024A Thorough Examination of Decoding Methods in the Era of LLMscited by 112
  35. 2024Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Codecited by 106
  36. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 104
  37. 2023Generating Benchmarks for Factuality Evaluation of Language Modelscited by 103
  38. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 96

50 most cited of 156 All 156 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 11.

12 papers.

  1. 2026SEE: Signal Embedding Energy for Quantifying Noise Interference in Large Audio Language Modelscited by 3
  2. 2026LPDS: Evaluating LLM Robustness Through Logic-Preserving Difficulty Scalingcited by 2
  3. 2026Measuring Reasoning Quality in LLMs: A Multi-Dimensional Behavioral Frameworkcited by 2
  4. 2026Do LLMs Truly Generalize in the Molecular Domain? A Perturbation-Based Analysiscited by 1
  5. 2026GSM-Plus-BN: A Perturbation-Based Benchmark for Bangla Mathematical Reasoning in Large Language Modelscited by 1
  6. 2025Sentra-Guard: A Real-Time Multilingual Defense Against Adversarial LLM Promptscited by 0
  7. 2026Semantic Content Determines Algorithmic Performancecited by 0
  8. 2026Robust Biomedical Publication Type and Study Design Classification with Knowledge-Guided Perturbationscited by 0
  9. 2026EchoDistill: Robust Large Audio Language Models via Noisy-to-Clean Self-Distillationcited by 0
  10. 2026Harnessing non-adversarial robustness in large language modelscited by 0
  11. 2026Cultural Bias Without a Cultural Self:A Disassociation Study of LLM's Persona and Biascited by 0
  12. 2026Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboardcited by 0