mlsanityData to 7 Oct 2026

Accurate, yet inconsistent? Consistency Analysis on Language Understanding Models

arXiv 2108.06665 ↗First posted 15 Aug 2021cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

Consistency, which refers to the capability of generating the same predictions for semantically similar contexts, is a highly desirable property for a sound language understanding model. Although recent pretrained language models (PLMs) deliver outstanding performance in various downstream tasks, they should exhibit consistent behaviour provided the models truly understand language. In this paper, we propose a simple framework named consistency analysis on language understanding models (CALUM)} to evaluate the model's lower-bound consistency ability. Through experiments, we confirmed that current PLMs are prone to generate inconsistent predictions even for semantically identical inputs. We also observed that multi-task training with paraphrase identification tasks is of benefit to improve consistency, increasing the consistency by 13% on average.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 35 of its 48 references that are in the map. First posted on arXiv 15 Aug 2021.

Builds on

35 of 48 references.

References · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2017Decoupled Weight Decay Regularizationcited by 1,722
  9. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  10. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  11. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  12. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
23 more
  1. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  2. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  3. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  4. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  5. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  6. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  7. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  8. 2020BAE: BERT-based Adversarial Examples for Text Classificationcited by 316
  9. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 240
  10. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  11. 2019Probing Neural Network Comprehension of Natural Language Argumentscited by 212
  12. 2018Stress Test Evaluation for Natural Language Inferencecited by 211
  13. 2020Contextualized Perturbation for Textual Adversarial Attackcited by 136
  14. 2021KLUE: Korean Language Understanding Evaluationcited by 113
  15. 2019Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Trainingcited by 110
  16. 2020Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Modelscited by 110
  17. 2020Logic-Guided Data Augmentation and Regularization for Consistent Question Answeringcited by 77
  18. 2021Understanding by Understanding Not: Modeling Negation in Language Modelscited by 73
  19. 2019Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanationscited by 71
  20. 2017The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrantscited by 55
  21. 2020KorNLI and KorSTS: New Benchmark Datasets for Korean Natural Language Understandingcited by 37
  22. 2019A Simple but Effective Method to Incorporate Multi-turn Context with BERT for Conversational Machine Comprehensioncited by 18
  23. 2021Does He Wink or Does He Nod? A Challenging Benchmark for Evaluating Word Understanding of Language Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 3; 2023: 4; 2024: 2; 2025: 2; 2026: 1.

13 papers.

Sort:

By area:

  1. 2024Looking Inward: Language Models Can Learn About Themselves by Introspectioncited by 105
  2. 2023Evaluating Superhuman Models with Consistency Checkscited by 47
  3. 2022Measuring Reliability of Large Language Models through Semantic Consistencycited by 38
  4. 2023Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMscited by 36
  5. 2023Predicting Question-Answering Performance of Large Language Models through Semantic Consistencycited by 28
  6. 2022Unpacking Large Language Models with Conceptual Consistencycited by 18
  7. 2022Factual Consistency of Multilingual Pretrained Language Modelscited by 15
  8. 2023Semantic Consistency for Assuring Reliability of Large Language Modelscited by 15
  9. 2025Potemkin Understanding in Large Language Modelscited by 14
  10. 2024One vs. Many: Comprehending Accurate Information from Multiple Erroneous and Inconsistent AI Generationscited by 13
  11. 2025Improving Consistency in Large Language Models through Chain of Guidancecited by 13
  12. 2021A Survey on Awesome Korean NLP Datasetscited by 3
1 more
  1. 2026TriAlign: Towards Universal Truth Consistency in Personalized LLM Alignmentcited by 0