mlsanityData to 7 Oct 2026

Uncovering More Shallow Heuristics: Probing the Natural Language Inference Capacities of Transformer-Based Pre-Trained Language Models Using Syllogistic Patterns

arXiv 2201.07614 ↗First posted 19 Jan 2022cs.CL cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

In this article, we explore the shallow heuristics used by transformer-based pre-trained language models (PLMs) that are fine-tuned for natural language inference (NLI). To do so, we construct or own dataset based on syllogistic, and we evaluate a number of models' performance on our dataset. We find evidence that the models rely heavily on certain shallow heuristics, picking up on symmetries and asymmetries between premise and hypothesis. We suggest that the lack of generalization observable in our study, which is becoming a topic of lively debate in the field, means that the PLMs are currently not learning NLI, but rather spurious heuristics.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 32 of its 44 references that are in the map. First posted on arXiv 19 Jan 2022.

Builds on

32 of 44 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  10. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  12. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
20 more
  1. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  2. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  3. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  4. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  5. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  6. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  7. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  8. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  9. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  10. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  11. 2019Probing Neural Network Comprehension of Natural Language Argumentscited by 212
  12. 2020Adversarial Filters of Dataset Biasescited by 159
  13. 2019End-to-End Bias Mitigation by Modelling Biases in Corporacited by 127
  14. 2019Unlearn Dataset Bias in Natural Language Inference by Fitting the Residualcited by 119
  15. 2020Towards Debiasing NLU Models from Unknown Biasescited by 116
  16. 2019Probing Natural Language Inference Models through Semantic Fragmentscited by 64
  17. 2020Towards Robustifying NLI Models Against Lexical Dataset Biasescited by 34
  18. 2021A Generative Approach for Mitigating Structural Biases in Natural Language Inferencecited by 3
  19. 2019Adversarial Analysis of Natural Language Inference Systemscited by 2
  20. 2016Natural Language Semantics and Computabilitycited by 1

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1; 2024: 1.

3 papers.

  1. 2022FOLIO: Natural Language Reasoning with First-Order Logiccited by 233
  2. 2024Graph Transformers: A Surveycited by 32
  3. 2023Keeping Up with the Language Models: Systematic Benchmark Extension for Bias Auditingcited by 0