mlsanityData to 14 Oct 2026

Consistency in Language Models: Current Landscape, Challenges, and Future Directions

arXiv 2505.00268 ↗First posted 1 May 2025cs.CL cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

The hallmark of effective language use lies in consistency: expressing similar meanings in similar contexts and avoiding contradictions. While human communication naturally demonstrates this principle, state-of-the-art language models (LMs) struggle to maintain reliable consistency across task- and domain-specific applications. Here we examine the landscape of consistency research in LMs, analyze current approaches to measure aspects of consistency, and identify critical research gaps. Our findings point to an urgent need for quality benchmarks to measure and interdisciplinary approaches to ensure consistency while preserving utility.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 47 of its 59 references that are in the map. First posted on arXiv 1 May 2025.

Builds on

47 of 59 references.

References · 47

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  3. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  4. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  5. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  6. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 407
  7. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 378
  8. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  9. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  10. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 232
  11. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 206
  12. 2023Implicit Chain of Thought Reasoning via Knowledge Distillationcited by 176
35 more
  1. 2023Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigationcited by 124
  2. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contextscited by 94
  3. 2022Evaluating the Factual Consistency of Large Language Models Through News Summarizationcited by 92
  4. 2023Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queriescited by 82
  5. 2020Logic-Guided Data Augmentation and Regularization for Consistent Question Answeringcited by 78
  6. 2023The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AIcited by 73
  7. 2023On Measuring Faithfulness or Self-consistency of Natural Language Explanationscited by 73
  8. 2019Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanationscited by 71
  9. 2024Self-Contrast: Better Reflection Through Inconsistent Solving Perspectivescited by 71
  10. 2020I like fish, especially dolphins: Addressing Contradictions in Dialogue Modelingcited by 68
  11. 2019A Logic-Driven Framework for Consistency of Neural Modelscited by 64
  12. 2019Evaluating Coherence in Dialogue Systems using Entailmentcited by 53
  13. 2023Benchmarking and Improving Generator-Validator Consistency of Language Modelscited by 44
  14. 2022Enhancing Self-Consistency and Performance of Pre-Trained Language Models through Natural Language Inferencecited by 40
  15. 2025Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistenciescited by 39
  16. 2022Measuring Reliability of Large Language Models through Semantic Consistencycited by 38
  17. 2023Consistency Analysis of ChatGPTcited by 37
  18. 2023Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMscited by 36
  19. 2024Improving the Robustness of Large Language Models via Consistency Alignmentcited by 36
  20. 2023Predicting Question-Answering Performance of Large Language Models through Semantic Consistencycited by 28
  21. 2024Consistency Checks for Language Model Forecasterscited by 27
  22. 2023RELIC: Investigating Large Language Model Responses using Self-Consistencycited by 23
  23. 2021Don't be Contradicted with Anything! CI-ToD: Towards Benchmarking Consistency for Task-oriented Dialogue Systemcited by 18
  24. 2022Factual Consistency of Multilingual Pretrained Language Modelscited by 15
  25. 2023Reproducibility in Machine Learning-Driven Researchcited by 15
  26. 2024LLM Whisperer: An Inconspicuous Attack to Bias LLM Responsescited by 15
  27. 2025Improving Consistency in Large Language Models through Chain of Guidancecited by 13
  28. 2024Aligning with Logic: Measuring, Evaluating and Improving Logical Preference Consistency in Large Language Modelscited by 12
  29. 2024From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistencycited by 7
  30. 2024DCR-Consistency: Divide-Conquer-Reasoning for Consistency Evaluation and Improvement of Large Language Modelscited by 4
  31. 2024Evaluating Knowledge-based Cross-lingual Inconsistency in Large Language Modelscited by 4
  32. 2023Measuring Consistency in Text-based Financial Forecasting Modelscited by 3
  33. 2024Inconsistent dialogue responses and how to recover from themcited by 3
  34. 2022SSD: Towards Better Text-Image Consistency Metric in Text-to-Image Generationcited by 2
  35. 2025Consistency Evaluation of News Article Summaries Generated by Large (and Small) Language Modelscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 9.

14 papers.

Sort:

By area:

  1. 2025Optimal Self-Consistency for Efficient Reasoning with Large Language Modelscited by 9
  2. 2025Estimating LLM Consistency: A User Baseline vs Surrogate Metricscited by 8
  3. 2025Found in Translation: Measuring Multilingual LLM Consistency as Simple as Translate then Evaluatecited by 3
  4. 2026STAGE: A Full-Screenplay Benchmark for Reasoning over Evolving Storiescited by 3
  5. 2026Human Simulation Computation: A Human-Inspired Framework for Adaptive AI Systemscited by 3
  6. 2026Consistency as a Testable Property: Statistical Methods to Evaluate AI Agent Reliabilitycited by 3
  7. 2025Improving Consistency in Retrieval-Augmented Systems with Group Similarity Rewardscited by 2
  8. 2026Communication Enhances LLMs' Stability in Strategic Thinkingcited by 1
  9. 2026Unlearners Can Lie: Evaluating and Improving Honesty in LLM Unlearningcited by 1
  10. 2026From Token Probabilities to Semantic Constraints: Towards Declarative Probabilistic Evaluation of Language Modelscited by 1
  11. 2025Graph RAG as Human Choice Model: Building a Data-Driven Mobility Agent with Preference Chaincited by 0
  12. 2026From Instruction to Output: The Role of Prompting in Modern NLGcited by 0
2 more
  1. 2026Nsanku: Evaluating Zero-Shot Translation Performance of LLMs for Ghanaian Languagescited by 0
  2. 2026TriAlign: Towards Universal Truth Consistency in Personalized LLM Alignmentcited by 0