# Lexicon Learning for Few-Shot Neural Sequence Modeling

arXiv: https://arxiv.org/abs/2106.03993 · HTML: https://mlsanity.com/paper/2106.03993/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 42 references that are in the map.
- First posted on arXiv 7 Jun 2021.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Measuring Compositional Generalization: A Comprehensive Method on Realistic Data](https://mlsanity.com/paper/1912.09713/) (2019)
- [COGS: A Compositional Generalization Challenge Based on Semantic Interpretation](https://mlsanity.com/paper/2010.05465/) (2020)
- [Deep Learning: A Critical Appraisal](https://mlsanity.com/paper/1801.00631/) (2018)
- [Good-Enough Compositional Data Augmentation](https://mlsanity.com/paper/1904.09545/) (2019)
- [Data Recombination for Neural Semantic Parsing](https://mlsanity.com/paper/1606.03622/) (2016)
- [Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures](https://mlsanity.com/paper/2007.08970/) (2020)
- [Learning Compositional Rules via Neural Program Synthesis](https://mlsanity.com/paper/2003.05562/) (2020)
- [Learning to Recombine and Resample Data for Compositional Generalization](https://mlsanity.com/paper/2010.03706/) (2020)
- [Compositional generalization in a deep seq2seq model by separating syntax and semantics](https://mlsanity.com/paper/1904.09708/) (2019)
- [Learning to Transduce with Unbounded Memory](https://mlsanity.com/paper/1506.02516/) (2015)
- [Compositional Generalization by Learning Analytical Expressions](https://mlsanity.com/paper/2006.10627/) (2020)
- [Human few-shot learning of compositional instructions](https://mlsanity.com/paper/1901.04587/) (2019)
- [Incorporating Discrete Translation Lexicons into Neural Machine Translation](https://mlsanity.com/paper/1606.02006/) (2016)
- [Improving Lexical Choice in Neural Machine Translation](https://mlsanity.com/paper/1710.01329/) (2017)
- [Towards one-shot learning for rare-word translation with external experts](https://mlsanity.com/paper/1809.03182/) (2018)
- [Pointer-based Fusion of Bilingual Lexicons into Neural Machine Translation](https://mlsanity.com/paper/1909.07907/) (2019)
- [Compositional Neural Machine Translation by Removing the Lexicon from Syntax](https://mlsanity.com/paper/2002.08899/) (2020)

## Built on by (most cited first, 28 of 28)

- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Compositional Semantic Parsing with Large Language Models](https://mlsanity.com/paper/2209.15003/) (2022)
- [Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement](https://mlsanity.com/paper/2310.08559/) (2023)
- [Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation](https://mlsanity.com/paper/2203.09435/) (2022)
- [Disentangled Sequence to Sequence Learning for Compositional Generalization](https://mlsanity.com/paper/2110.04655/) (2021)
- [Uncontrolled Lexical Exposure Leads to Overestimation of Compositional Generalization in Pretrained Models](https://mlsanity.com/paper/2212.10769/) (2022)
- [Structural generalization is hard for sequence-to-sequence models](https://mlsanity.com/paper/2210.13050/) (2022)
- [ReCOGS: How Incidental Details of a Logical Form Overshadow an Evaluation of Semantic Interpretation](https://mlsanity.com/paper/2303.13716/) (2023)
- [SLOG: A Structural Generalization Benchmark for Semantic Parsing](https://mlsanity.com/paper/2310.15040/) (2023)
- [Revisiting the Compositional Generalization Abilities of Neural Sequence Models](https://mlsanity.com/paper/2203.07402/) (2022)
- [Improving Word Translation via Two-Stage Contrastive Learning](https://mlsanity.com/paper/2203.08307/) (2022)
- [Compositionality as Lexical Symmetry](https://mlsanity.com/paper/2201.12926/) (2022)
- [Mutual Exclusivity Training and Primitive Augmentation to Induce Compositionality](https://mlsanity.com/paper/2211.15578/) (2022)
- [Learning to Substitute Spans towards Improving Compositional Generalization](https://mlsanity.com/paper/2306.02840/) (2023)
- [MAGNIFICo: Evaluating the In-Context Learning Ability of Large Language Models to Generalize to Novel Interpretations](https://mlsanity.com/paper/2310.11634/) (2023)
- [Compositional Generalisation with Structured Reordering and Fertility Layers](https://mlsanity.com/paper/2210.03183/) (2022)
- [Compositional Generalization without Trees using Multiset Tagging and Latent Permutations](https://mlsanity.com/paper/2305.16954/) (2023)
- [Equivariant Transduction through Invariant Alignment](https://mlsanity.com/paper/2209.10926/) (2022)
- [Real-World Compositional Generalization with Disentangled Sequence-to-Sequence Learning](https://mlsanity.com/paper/2212.05982/) (2022)
- [Learning to Compose Representations of Different Encoder Layers towards Improving Compositional Generalization](https://mlsanity.com/paper/2305.12169/) (2023)
- [Data-driven Model Generalizability in Crosslinguistic Low-resource Morphological Segmentation](https://mlsanity.com/paper/2201.01845/) (2022)
- [Compositional Generalization Requires Compositional Parsers](https://mlsanity.com/paper/2202.11937/) (2022)
- [Layer-wise Representation Fusion for Compositional Generalization](https://mlsanity.com/paper/2307.10799/) (2023)
- [Harnessing Dataset Cartography for Improved Compositional Generalization in Transformers](https://mlsanity.com/paper/2310.12118/) (2023)
- [Efficient Model-Agnostic Multi-Group Equivariant Networks](https://mlsanity.com/paper/2310.09675/) (2023)
- [Inducing Systematicity in Transformers by Attending to Structurally Quantized Embeddings](https://mlsanity.com/paper/2402.06492/) (2024)
- [Sequential Compositional Generalization in Multimodal Models](https://mlsanity.com/paper/2404.12013/) (2024)
- [Learning to Substitute Components for Compositional Generalization](https://mlsanity.com/paper/2502.20834/) (2025)

## Abstract (arXiv)

Sequence-to-sequence transduction is the core problem in language processing applications as diverse as semantic parsing, machine translation, and instruction following. The neural network models that provide the dominant solution to these problems are brittle, especially in low-resource settings: they fail to generalize correctly or systematically from small datasets. Past work has shown that many failures of systematic generalization arise from neural models' inability to disentangle lexical phenomena from syntactic ones. To address this, we augment neural decoders with a lexical translation mechanism that generalizes existing copy mechanisms to incorporate learned, decontextualized, token-level translation rules. We describe how to initialize this mechanism using a variety of lexicon learning algorithms, and show that it improves systematic generalization on a diverse set of sequence modeling tasks drawn from cognitive science, formal semantics, and machine translation.