mlsanityData to 14 Oct 2026

Enhancing Self-Consistency and Performance of Pre-Trained Language Models through Natural Language Inference

arXiv 2211.11875 ↗First posted 21 Nov 2022cs.CL cs.AIcited by 40

Not on the map. Why?

Abstract (arXiv)

While large pre-trained language models are powerful, their predictions often lack logical consistency across test inputs. For example, a state-of-the-art Macaw question-answering (QA) model answers 'Yes' to 'Is a sparrow a bird?' and 'Does a bird have feet?' but answers 'No' to 'Does a sparrow have feet?'. To address this failure mode, we propose a framework, Consistency Correction through Relation Detection, or ConCoRD, for boosting the consistency and accuracy of pre-trained NLP models using pre-trained natural language inference (NLI) models without fine-tuning or re-training. Given a batch of test inputs, ConCoRD samples several candidate outputs for each input and instantiates a factor graph that accounts for both the model's belief about the likelihood of each answer choice in isolation and the NLI model's beliefs about pair-wise answer choice compatibility. We show that a weighted MaxSAT solver can efficiently compute high-quality answer choices under this factor graph, improving over the raw model's predictions. Our experiments demonstrate that ConCoRD consistently boosts accuracy and consistency of off-the-shelf closed-book QA and VQA models using off-the-shelf NLI models, notably increasing accuracy of LXMERT on ConVQA by 5% absolute. See https://ericmitchell.ai/emnlp-2022-concord/ for code and data.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Builds on 30 of its 42 references that are in the map. First posted on arXiv 21 Nov 2022.

Builds on

30 of 42 references.

References · 30

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  7. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  9. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  10. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,067
  11. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  12. 2021Fast Model Editing at Scalecited by 498
18 more
  1. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 407
  2. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  3. 2019Differentiation of Blackbox Combinatorial Solverscited by 226
  4. 2020Unsupervised Commonsense Question Answering with Self-Talkcited by 161
  5. 2022Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanationscited by 159
  6. 2020Editable Neural Networkscited by 155
  7. 2019Synthetic QA Corpora Generation with Roundtrip Consistencycited by 143
  8. 2020Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generationcited by 141
  9. 2018Dialogue Natural Language Inferencecited by 135
  10. 2018Transforming Question Answering Datasets Into Natural Language Inference Datasetscited by 99
  11. 2020Logic-Guided Data Augmentation and Regularization for Consistent Question Answeringcited by 78
  12. 2019A Logic-Driven Framework for Consistency of Neural Modelscited by 64
  13. 2021BeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Beliefcited by 57
  14. 2019Evaluating Coherence in Dialogue Systems using Entailmentcited by 53
  15. 2021Can NLI Models Verify QA Systems' Predictions?cited by 46
  16. 2021General-Purpose Question-Answering with Macawcited by 40
  17. 2019Sunny and Dark Outside?! Improving Answer Consistency in VQA through Entailed Question Generationcited by 36
  18. 2019Generating Persona Consistent Dialogues by Exploiting Natural Language Inferencecited by 34

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 13; 2024: 4; 2025: 13; 2026: 9.

40 papers.

Sort:

By area:

  1. 2023Conformal Language Modelingcited by 170
  2. 2025Empowering LLMs with Logical Reasoning: A Comprehensive Surveycited by 51
  3. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 50
  4. 2023Benchmarking and Improving Generator-Validator Consistency of Language Modelscited by 44
  5. 2023The Consensus Game: Language Model Generation via Equilibrium Searchcited by 36
  6. 2023Interactive Natural Language Processingcited by 31
  7. 2024Logically Consistent Language Models via Neuro-Symbolic Integrationcited by 30
  8. 2023The Trickle-down Impact of Reward (In-)consistency on RLHFcited by 29
  9. 2023Human-like Few-Shot Learning via Bayesian Reasoning over Natural Languagecited by 22
  10. 2024A Probabilistic Framework for LLM Hallucination Detection via Belief Tree Propagationcited by 21
  11. 2023Language Models with Rationalitycited by 15
  12. 2023Semantic Consistency for Assuring Reliability of Large Language Modelscited by 15
28 more
  1. 2025Consistency in Language Models: Current Landscape, Challenges, and Future Directionscited by 14
  2. 2025Improving Consistency in Large Language Models through Chain of Guidancecited by 13
  3. 2025MMRAG: Multi-Mode Retrieval-Augmented Generation with Large Language Models for Biomedical In-Context Learningcited by 13
  4. 2022Improved Beam Search for Hallucination Mitigation in Abstractive Summarizationcited by 10
  5. 2025Incentivizing Truthful Language Models via Peer Elicitation Gamescited by 9
  6. 2024From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistencycited by 7
  7. 2023Prompt to be Consistent is Better than Self-Consistent? Few-Shot and Zero-Shot Fact Verification with Pre-trained Language Modelscited by 6
  8. 2026Beyond Calibration: Do a Typed-Decision Model's Probabilities Obey the Probability Axioms?cited by 6
  9. 2023Zero-shot Learning with Minimum Instruction to Extract Social Determinants and Family History from Clinical Notes using GPT Modelcited by 5
  10. 2025Beyond the Singular: Revealing the Value of Multiple Generations in Benchmark Evaluationcited by 5
  11. 2026Where Knowledge Collides: A Mechanistic Study of Intra-Memory Knowledge Conflict in Language Modelscited by 5
  12. 2026Pixels Don't Lie (But Your Detector Might): Bootstrapping MLLM-as-a-Judge for Trustworthy Deepfake Detection and Reasoning Supervisioncited by 5
  13. 2023KNOW How to Make Up Your Mind! Adversarially Detecting and Alleviating Inconsistencies in Natural Language Explanationscited by 4
  14. 2023No Strong Feelings One Way or Another: Re-operationalizing Neutrality in Natural Language Inferencecited by 4
  15. 2025VideoJudge: Bootstrapping Enables Scalable Supervision of MLLM-as-a-Judge for Video Understandingcited by 4
  16. 2026Online Reasoning Calibration: Test-Time Training Enables Generalizable Conformal LLM Reasoningcited by 4
  17. 2026Dutch Books for Language Modelscited by 3
  18. 2025Improving Consistency in Retrieval-Augmented Systems with Group Similarity Rewardscited by 2
  19. 2025SIMBA UQ: Similarity-Based Aggregation for Uncertainty Quantification in Large Language Modelscited by 2
  20. 2026Same Patient, Different Words, Different Diagnosis? Evaluating Semantic Stability in Clinical LLMscited by 2
  21. 2025Legal Requirements Translation from Lawcited by 1
  22. 2026Hierarchical Alignment: Enforcing Hierarchical Instruction-Following in LLMs through Logical Consistencycited by 1
  23. 2026Empirical Bayes Conformal Prediction for Vision and Language Modelscited by 1
  24. 2023A deep Natural Language Inference predictor without language-specific training datacited by 0
  25. 2025Waste Not, Want Not; Recycled Gumbel Noise Improves Consistency in Natural Language Generationcited by 0
  26. 2025Evaluating Large Language Models in Code Generation: INFINITE Methodology for Defining the Inference Indexcited by 0
  27. 2025OR-VSKC: Resolving Visual-Semantic Knowledge Conflicts in Operating Rooms with Synthetic Data-Guided Alignmentcited by 0
  28. 2026Towards Annotation-Free Validation of MLLMs: A Vision-Language Logical Consistency Metriccited by 0