Towards Equipping Transformer with the Ability of Systematic Compositionality
Not on the map. Why?
Abstract (arXiv)
One of the key factors in language productivity and human cognition is the ability of systematic compositionality, which refers to understanding composed unseen examples of seen primitives. However, recent evidence reveals that the Transformers have difficulty generalizing the composed context based on the seen primitives. To this end, we take the first step to propose a compositionality-aware Transformer called CAT and two novel pre-training tasks to facilitate systematic compositionality. We tentatively provide a successful implementation of a multi-layer CAT on the basis of the especially popular BERT. The experimental results demonstrate that CAT outperforms baselines on compositionality-aware tasks with minimal impact on the effectiveness on standardized language understanding tasks.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 38 of its 53 references that are in the map. First posted on arXiv 12 Dec 2023.
Builds on
38 of 53 references.
References · 38
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
- 2017Neural Discrete Representation Learningcited by 4,650
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
- 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
- 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
- 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
- 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
- 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
26 more
- 2016Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filteringcited by 517
- 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
- 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
- 2018Towards a Definition of Disentangled Representationscited by 294
- 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
- 2019Understanding Straight-Through Estimator in Training Activation Quantized Neural Netscited by 225
- 2019Recurrent Independent Mechanismscited by 219
- 2019Good-Enough Compositional Data Augmentationcited by 158
- 2017The Consciousness Priorcited by 105
- 2020Compositionality and Generalization in Emergent Languagescited by 103
- 2019Linguistic generalization and compositionality in modern artificial neural networkscited by 58
- 2019Compositional generalization in a deep seq2seq model by separating syntax and semanticscited by 56
- 2023How Do In-Context Examples Affect Compositional Generalization?cited by 51
- 2020Assessing Phrasal Representation and Composition in Transformerscited by 41
- 2021Sparse Attention with Linear Unitscited by 41
- 2021Discrete-Valued Neural Communicationcited by 39
- 2019OpenHowNet: An Open Sememe-based Lexical Knowledge Basecited by 21
- 2020Latent Compositional Representations Improve Systematic Generalization in Grounded Question Answeringcited by 16
- 2018Compositional Representation of Morphologically-Rich Input for Neural Machine Translationcited by 12
- 2022Discrete and continuous representations and processing in deep learning: Looking forwardcited by 12
- 2022Adaptive Discrete Communication Bottlenecks with Dynamic Vector Quantizationcited by 11
- 2019Modeling Semantic Compositionality with Sememe Knowledgecited by 8
- 2021On the Interplay Between Fine-tuning and Composition in Transformerscited by 6
- 2022Pathologies of Pre-trained Language Models in Few-shot Fine-tuningcited by 3
- 2021Sparse Communication via Mixed Distributionscited by 2
- 2023Efficient Learning of Discrete-Continuous Computation Graphscited by 2
Built on by
Citing papers per year: 2025: 2.
2 papers.