mlsanityData to 14 Oct 2026

The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization

arXiv 2110.07732 ↗First posted 14 Oct 2021cs.LG cs.AI cs.NEcited by 62

Not on the map. Why?

Abstract (arXiv)

Despite progress across a broad range of applications, Transformers have limited success in systematic generalization. The situation is especially frustrating in the case of algorithmic tasks, where they often fail to find intuitive solutions that route relevant information to the right node/operation at the right time in the grid represented by Transformer columns. To facilitate the learning of useful control flow, we propose two modifications to the Transformer architecture, copy gate and geometric attention. Our novel Neural Data Router (NDR) achieves 100% length generalization accuracy on the classic compositional table lookup task, as well as near-perfect accuracy on the simple arithmetic task and a new variant of ListOps testing for generalization across computational depths. NDR's attention and gating patterns tend to be interpretable as an intuitive form of neural routing. Our code is public.

Key facts. Cited by 62 papers in the map (data through 14 Oct 2026). Builds on 45 of its 61 references that are in the map. First posted on arXiv 14 Oct 2021.

Builds on

45 of 61 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  7. 2020GLU Variants Improve Transformercited by 1,804
  8. 2018Self-Attention with Relative Position Representationscited by 1,120
  9. 2016Language Modeling with Gated Convolutional Networkscited by 702
  10. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  11. 2018Universal Transformerscited by 647
  12. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 506
33 more
  1. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 471
  2. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 284
  3. 2019Stabilizing Transformers for Reinforcement Learningcited by 270
  4. 2019Compositionality decomposed: how do neural networks generalise?cited by 265
  5. 2020On the Binding Problem in Artificial Neural Networkscited by 233
  6. 2021Thinking Like Transformerscited by 184
  7. 2015Training Very Deep Networkscited by 173
  8. 2021PonderNet: Learning to Pondercited by 168
  9. 2020Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?cited by 147
  10. 2015Character-Aware Neural Language Modelscited by 132
  11. 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 129
  12. 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformerscited by 117
  13. 2019Compositional generalization through meta sequence-to-sequence learningcited by 99
  14. 2019Language Modeling with Deep Transformerscited by 96
  15. 2020Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architecturescited by 95
  16. 2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 89
  17. 2020Compositional Generalization via Neural-Symbolic Stack Machinescited by 80
  18. 2021Making Transformers Solve Compositional Taskscited by 72
  19. 2019CLOSURE: Assessing Systematic Generalization of CLEVR Modelscited by 63
  20. 2019Compositional generalization in a deep seq2seq model by separating syntax and semanticscited by 56
  21. 2020Compositional Generalization by Learning Analytical Expressionscited by 53
  22. 2019Compositional Generalization for Primitive Substitutionscited by 48
  23. 2018Memorize or generalize? Searching for a compositional RNN in a haystackcited by 41
  24. 2019Location Attention for Extrapolation to Longer Sequencescited by 33
  25. 2019Cooperative Learning of Disjoint Syntax and Semanticscited by 27
  26. 2019CNNs found to jump around more skillfully than RNNs: Compositional generalization in seq2seq convolutional networkscited by 22
  27. 2019Ordered Memorycited by 20
  28. 2021Modeling Hierarchical Structures with Continuous Recursive Neural Networkscited by 15
  29. 2019Transcoding compositionally: using attention to find more generalizable solutionscited by 14
  30. 2021Can Transformers Jump Around Right in Natural Language? Assessing Performance Transfer from SCANcited by 14
  31. 2018Learning compositionally through attentive guidancecited by 11
  32. 2019Improving Differentiable Neural Computers Through Memory Masking, De-allocation, and Link Distribution Sharpness Controlcited by 9
  33. 2021Reinforcement Learning of Implicit and Explicit Control Flow in Instructionscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 7; 2023: 18; 2024: 17; 2025: 12; 2026: 6.

62 papers.

Sort:

By area:

  1. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 117
  2. 2022Block-Recurrent Transformerscited by 110
  3. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 106
  4. 2021Linear algebra with transformerscited by 63
  5. 2022A Generalist Neural Algorithmic Learnercited by 59
  6. 2024MoEUT: Mixture-of-Experts Universal Transformerscited by 59
  7. 2023Positional Description Matters for Transformers Arithmeticcited by 56
  8. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 53
  9. 2022A Modern Self-Referential Weight Matrix That Learns to Modify Itselfcited by 41
  10. 2024How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpadcited by 36
  11. 2023SwitchHead: Accelerating Transformers with Mixture-of-Experts Attentioncited by 34
  12. 2025PaTH Attention: Position Encoding via Accumulating Householder Transformationscited by 34
38 more
  1. 2022SALSA: Attacking Lattice Cryptography with Transformerscited by 28
  2. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encodingcited by 28
  3. 2023Approximating Two-Layer Feedforward Networks for Efficient Transformerscited by 27
  4. 2023When can transformers reason with abstract symbols?cited by 24
  5. 2023Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Taskscited by 22
  6. 2025Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformerscited by 20
  7. 2022Systematic Generalization and Emergent Structures in Transformers Trained on Structured Taskscited by 19
  8. 2023ModuleFormer: Modularity Emerges from Mixture-of-Expertscited by 17
  9. 2024Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizingcited by 17
  10. 2023Transformer Working Memory Enables Regular Language Reasoning and Natural Language Length Extrapolationcited by 13
  11. 2024Scaling Stick-Breaking Attention: An Efficient Implementation and In-depth Studycited by 13
  12. 2025An Analysis for Reasoning Bias of Language Models with Small Initializationcited by 13
  13. 2022CTL++: Evaluating Generalization on Never-Seen Compositional Patterns of Known Functions, and Compatibility of Neural Representationscited by 12
  14. 2026Hyperloop Transformerscited by 12
  15. 2023SALSA PICANTE: a machine learning attack on LWE with binary secretscited by 11
  16. 2023From Interpolation to Extrapolation: Complete Length Generalization for Arithmetic Transformerscited by 10
  17. 2025Learning to Skip the Middle Layers of Transformerscited by 10
  18. 2025Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddingscited by 10
  19. 2023Monotonic Location Attention for Length Generalizationcited by 9
  20. 2024Benchmarking GPT-4 on Algorithmic Problems: A Systematic Evaluation of Prompting Strategiescited by 9
  21. 2025The Role of Sparsity for Length Generalization in Transformerscited by 8
  22. 2023Beam Tree Recursive Cellscited by 6
  23. 2024Transformer Normalisation Layers and the Independence of Semantic Subspacescited by 6
  24. 2024Recurrent Aggregators in Neural Algorithmic Reasoningcited by 6
  25. 2024Why Are Positional Encodings Nonessential for Deep Autoregressive Transformers? Revisiting a Petroglyphcited by 6
  26. 2023Compositional Generalization without Trees using Multiset Tagging and Latent Permutationscited by 5
  27. 2024Investigating Recurrent Transformers with Dynamic Haltcited by 5
  28. 2023Efficient Beam Tree Recursioncited by 4
  29. 2026Linear Probe Accuracy Scales with Model Size and Benefits from Multi-Layer Ensemblingcited by 4
  30. 2023Approximating CKY with Transformerscited by 3
  31. 2023Attribute Diversity Determines the Systematicity Gap in VQAcited by 3
  32. 2024Assessing the Emergent Symbolic Reasoning Abilities of Llama Large Language Modelscited by 3
  33. 2024Overcoming classic challenges for artificial neural networks by providing incentives and practicecited by 3
  34. 2024A Neural Rewriting System to Solve Algorithmic Problemscited by 2
  35. 2024Attention-based Iterative Decomposition for Tensor Product Representationcited by 2
  36. 2021State-Space Constraints Can Improve the Generalisation of the Differentiable Neural Computer to Input Sequences With Unseen Lengthcited by 1
  37. 2023The Construction of Reality in an AI: A Reviewcited by 1
  38. 2023A Hybrid System for Systematic Generalization in Simple Arithmetic Problemscited by 1

50 most cited of 62 All 62 (JSON)