Layer-wise Representation Fusion for Compositional Generalization
Not on the map. Why?
Abstract (arXiv)
Existing neural models are demonstrated to struggle with compositional generalization (CG), i.e., the ability to systematically generalize to unseen compositions of seen components. A key reason for failure on CG is that the syntactic and semantic representations of sequences in both the uppermost layer of the encoder and decoder are entangled. However, previous work concentrates on separating the learning of syntax and semantics instead of exploring the reasons behind the representation entanglement (RE) problem to solve it. We explain why it exists by analyzing the representation evolving mechanism from the bottom to the top of the Transformer layers. We find that the ``shallow'' residual connections within each layer fail to fuse previous layers' information effectively, leading to information forgetting between layers and further the RE problems. Inspired by this, we propose LRF, a novel Layer-wise Representation Fusion framework for CG, which learns to fuse previous layers' information back into the encoding and decoding process effectively through introducing a fuse-attention module at each encoder and decoder layer. LRF achieves promising results on two realistic benchmarks, empirically demonstrating the effectiveness of our proposal.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 71 of its 89 references that are in the map. First posted on arXiv 20 Jul 2023.
Builds on
71 of 89 references.
References · 71
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
- 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
- 2018Deep contextualized word representationscited by 2,285
- 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
- 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
- 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
38 more
- 2018Universal Transformerscited by 613
- 2017Deep Layer Aggregationcited by 436
- 2019Learning Deep Transformer Models for Machine Translationcited by 309
- 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 280
- 2017Beam Search Strategies for Neural Machine Translationcited by 232
- 2019The Evolved Transformercited by 180
- 2016Language to Logical Form with Neural Attentioncited by 158
- 2019Good-Enough Compositional Data Augmentationcited by 158
- 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
- 2016Data Recombination for Neural Semantic Parsingcited by 111
- 2020Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translationcited by 104
- 2019Compositional generalization through meta sequence-to-sequence learningcited by 98
- 2020Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architecturescited by 95
- 2022Compositional Semantic Parsing with Large Language Modelscited by 95
- 2020Span-based Semantic Parsing for Compositional Generalizationcited by 77
- 2020Learning Compositional Rules via Neural Program Synthesiscited by 74
- 2020Sequence-Level Mixed Sample Data Augmentationcited by 71
- 2020AdvAug: Robust Adversarial Augmentation for Neural Machine Translationcited by 66
- 2020Learning to Recombine and Resample Data for Compositional Generalizationcited by 65
- 2021Meta-Learning to Compositionally Generalizecited by 62
- 2021The paradox of the compositionality of natural language: a neural machine translation case studycited by 58
- 2019Compositional generalization in a deep seq2seq model by separating syntax and semanticscited by 56
- 2021Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representationscited by 55
- 2020Compositional Generalization by Learning Analytical Expressionscited by 53
- 2019Compositional Generalization for Primitive Substitutionscited by 48
- 2021Improving Compositional Generalization with Latent Structure and Data Augmentationcited by 46
- 2018Training Deeper Neural Machine Translation Models with Transparent Attentioncited by 37
- 2020Shallow-to-Deep Training for Neural Machine Translationcited by 36
- 2018Towards Robust Neural Machine Translationcited by 35
- 2019On the Linguistic Representational Power of Neural Machine Translation Modelscited by 34
- 2021On Compositional Generalization of Neural Machine Translationcited by 34
- 2016Incorporating Discrete Translation Lexicons into Neural Machine Translationcited by 33
- 2020Compositional Generalization via Semantic Taggingcited by 32
- 2021Learning Algebraic Recombination for Compositional Generalizationcited by 31
- 2021Disentangled Sequence to Sequence Learning for Compositional Generalizationcited by 30
- 2021Lexicon Learning for Few-Shot Neural Sequence Modelingcited by 28
- 2021Finding needles in a haystack: Sampling Structurally-diverse Training Sets from Synthetic Data for Compositional Generalizationcited by 27
- 2020Hierarchical Poset Decoding for Compositional Generalization in Languagecited by 26
50 most cited of 71 All 71 (JSON)
Built on by
Citing papers per year: 2025: 2.
2 papers.