# Consistency in Language Models: Current Landscape, Challenges, and Future Directions

arXiv: https://arxiv.org/abs/2505.00268 · HTML: https://mlsanity.com/paper/2505.00268/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 59 references that are in the map.
- First posted on arXiv 1 May 2025.

## Builds on (most cited first, 30 of 47)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://mlsanity.com/paper/2302.09664/) (2023)
- [The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"](https://mlsanity.com/paper/2309.12288/) (2023)
- [Asking and Answering Questions to Evaluate the Factual Consistency of Summaries](https://mlsanity.com/paper/2004.04228/) (2020)
- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024)
- [Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models](https://mlsanity.com/paper/2111.02840/) (2021)
- [ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning](https://mlsanity.com/paper/2304.05613/) (2023)
- [Implicit Chain of Thought Reasoning via Knowledge Distillation](https://mlsanity.com/paper/2311.01460/) (2023)
- [Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigation](https://mlsanity.com/paper/2305.15852/) (2023)
- [The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts](https://mlsanity.com/paper/2401.13136/) (2024)
- [Evaluating the Factual Consistency of Large Language Models Through News Summarization](https://mlsanity.com/paper/2211.08412/) (2022)
- [Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries](https://mlsanity.com/paper/2310.13132/) (2023)
- [Logic-Guided Data Augmentation and Regularization for Consistent Question Answering](https://mlsanity.com/paper/2004.10157/) (2020)
- [The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI](https://mlsanity.com/paper/2310.16787/) (2023)
- [On Measuring Faithfulness or Self-consistency of Natural Language Explanations](https://mlsanity.com/paper/2311.07466/) (2023)
- [Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanations](https://mlsanity.com/paper/1910.03065/) (2019)
- [Self-Contrast: Better Reflection Through Inconsistent Solving Perspectives](https://mlsanity.com/paper/2401.02009/) (2024)
- [I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling](https://mlsanity.com/paper/2012.13391/) (2020)
- [A Logic-Driven Framework for Consistency of Neural Models](https://mlsanity.com/paper/1909.00126/) (2019)
- [Evaluating Coherence in Dialogue Systems using Entailment](https://mlsanity.com/paper/1904.03371/) (2019)
- [Benchmarking and Improving Generator-Validator Consistency of Language Models](https://mlsanity.com/paper/2310.01846/) (2023)
- [Enhancing Self-Consistency and Performance of Pre-Trained Language Models through Natural Language Inference](https://mlsanity.com/paper/2211.11875/) (2022)
- [Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies](https://mlsanity.com/paper/2502.08554/) (2025)
- [Measuring Reliability of Large Language Models through Semantic Consistency](https://mlsanity.com/paper/2211.05853/) (2022)
- [Consistency Analysis of ChatGPT](https://mlsanity.com/paper/2303.06273/) (2023)
- [Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMs](https://mlsanity.com/paper/2305.14279/) (2023)

## Built on by (most cited first, 14 of 14)

- [Optimal Self-Consistency for Efficient Reasoning with Large Language Models](https://mlsanity.com/paper/2511.12309/) (2025)
- [Estimating LLM Consistency: A User Baseline vs Surrogate Metrics](https://mlsanity.com/paper/2505.23799/) (2025)
- [Found in Translation: Measuring Multilingual LLM Consistency as Simple as Translate then Evaluate](https://mlsanity.com/paper/2505.21999/) (2025)
- [STAGE: A Full-Screenplay Benchmark for Reasoning over Evolving Stories](https://mlsanity.com/paper/2601.08510/) (2026)
- [Human Simulation Computation: A Human-Inspired Framework for Adaptive AI Systems](https://mlsanity.com/paper/2601.13887/) (2026)
- [Consistency as a Testable Property: Statistical Methods to Evaluate AI Agent Reliability](https://mlsanity.com/paper/2605.10516/) (2026)
- [Improving Consistency in Retrieval-Augmented Systems with Group Similarity Rewards](https://mlsanity.com/paper/2510.04392/) (2025)
- [Communication Enhances LLMs' Stability in Strategic Thinking](https://mlsanity.com/paper/2602.06081/) (2026)
- [Unlearners Can Lie: Evaluating and Improving Honesty in LLM Unlearning](https://mlsanity.com/paper/2605.08765/) (2026)
- [From Token Probabilities to Semantic Constraints: Towards Declarative Probabilistic Evaluation of Language Models](https://mlsanity.com/paper/2609.13520/) (2026)
- [Graph RAG as Human Choice Model: Building a Data-Driven Mobility Agent with Preference Chain](https://mlsanity.com/paper/2508.16172/) (2025)
- [From Instruction to Output: The Role of Prompting in Modern NLG](https://mlsanity.com/paper/2602.11179/) (2026)
- [Nsanku: Evaluating Zero-Shot Translation Performance of LLMs for Ghanaian Languages](https://mlsanity.com/paper/2605.04208/) (2026)
- [TriAlign: Towards Universal Truth Consistency in Personalized LLM Alignment](https://mlsanity.com/paper/2606.01755/) (2026)

## Abstract (arXiv)

The hallmark of effective language use lies in consistency: expressing similar meanings in similar contexts and avoiding contradictions. While human communication naturally demonstrates this principle, state-of-the-art language models (LMs) struggle to maintain reliable consistency across task- and domain-specific applications. Here we examine the landscape of consistency research in LMs, analyze current approaches to measure aspects of consistency, and identify critical research gaps. Our findings point to an urgent need for quality benchmarks to measure and interdisciplinary approaches to ensure consistency while preserving utility.