Testing Pre-trained Language Models' Understanding of Distributivity via Causal Mediation Analysis
Not on the map. Why?
Abstract (arXiv)
To what extent do pre-trained language models grasp semantic knowledge regarding the phenomenon of distributivity? In this paper, we introduce DistNLI, a new diagnostic dataset for natural language inference that targets the semantic difference arising from distributivity, and employ the causal mediation analysis framework to quantify the model behavior and explore the underlying mechanism in this semantically-related task. We find that the extent of models' understanding is associated with model size and vocabulary size. We also provide insights into how models encode such high-level semantic knowledge.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 20 of its 41 references that are in the map. First posted on arXiv 11 Sep 2022.
Builds on
20 of 41 references.
References · 20
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
- 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
- 2015A large annotated corpus for learning natural language inferencecited by 1,847
- 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
- 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
- 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
- 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
- 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 320
- 2013Direct and Indirect Effectscited by 280
8 more
- 2016Enhanced LSTM for Natural Language Inferencecited by 200
- 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 191
- 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
- 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 131
- 2021Language Models Use Monotonicity to Assess NPI Licensingcited by 31
- 2020ConjNLI: Natural Language Inference Over Conjunctive Sentencescited by 27
- 2022What Changed? Investigating Debiasing Methods using Causal Mediation Analysiscited by 7
- 2018DR-BiLSTM: Dependent Reading Bidirectional LSTM for Natural Language Inferencecited by 5
Built on by
Citing papers per year: 2022: 1; 2023: 2.
3 papers.