mlsanityData to 14 Oct 2026

Systematic Generalization with Edge Transformers

arXiv 2112.00578 ↗First posted 1 Dec 2021cs.CL cs.LGcited by 38

Not on the map. Why?

Abstract (arXiv)

Recent research suggests that systematic generalization in natural language understanding remains a challenge for state-of-the-art neural models such as Transformers and Graph Neural Networks. To tackle this challenge, we propose Edge Transformer, a new model that combines inspiration from Transformers and rule-based symbolic AI. The first key idea in Edge Transformers is to associate vector states with every edge, that is, with every pair of input nodes -- as opposed to just every node, as it is done in the Transformer model. The second major innovation is a triangular attention mechanism that updates edge representations in a way that is inspired by unification from logic programming. We evaluate Edge Transformer on compositional generalization benchmarks in relational reasoning, semantic parsing, and dependency parsing. In all three settings, the Edge Transformer outperforms Relation-aware, Universal and classical Transformer baselines.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Builds on 35 of its 44 references that are in the map. First posted on arXiv 1 Dec 2021.

Builds on

35 of 44 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  5. 2017Graph Attention Networkscited by 8,923
  6. 2020End-to-End Object Detection with Transformerscited by 7,414
  7. 2016Layer Normalizationcited by 5,669
  8. 2017Neural Message Passing for Quantum Chemistrycited by 3,487
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  10. 2020Rethinking Attention with Performerscited by 1,601
  11. 2018Self-Attention with Relative Position Representationscited by 1,120
  12. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 723
23 more
  1. 2018Universal Transformerscited by 647
  2. 2019The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervisioncited by 372
  3. 2016Deep Biaffine Attention for Neural Dependency Parsingcited by 334
  4. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 284
  5. 2019Compositionality decomposed: how do neural networks generalise?cited by 265
  6. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 263
  7. 2017Learning Explanatory Rules from Noisy Datacited by 211
  8. 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 162
  9. 2017End-to-End Differentiable Provingcited by 161
  10. 2019Good-Enough Compositional Data Augmentationcited by 158
  11. 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformerscited by 117
  12. 2019Compositional generalization through meta sequence-to-sequence learningcited by 99
  13. 2020Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architecturescited by 95
  14. 2020Learning Compositional Rules via Neural Program Synthesiscited by 74
  15. 2021Making Transformers Solve Compositional Taskscited by 72
  16. 2020Learning to Recombine and Resample Data for Compositional Generalizationcited by 65
  17. 2020Learning Reasoning Strategies in End-to-End Differentiable Provingcited by 60
  18. 2020Measuring Systematic Generalization in Neural Proof Generation with Transformerscited by 43
  19. 2020Hierarchical Poset Decoding for Compositional Generalization in Languagecited by 26
  20. 2017Universal Semantic Parsingcited by 21
  21. 2020*-CFQ: Analyzing the Scalability of Machine Learning on a Compositional Taskcited by 16
  22. 2019Utilizing Edge Features in Graph Neural Networks via Variational Information Maximizationcited by 10
  23. 2021Compositional Generalization in Dependency Parsingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 14; 2024: 7; 2025: 6; 2026: 6.

38 papers.

Sort:

By area:

  1. 2023Large Language Models can Learn Rulescited by 69
  2. 2023Coupling Large Language Models with Logic Programming for Robust and General Reasoning from Textcited by 59
  3. 2024MoEUT: Mixture-of-Experts Universal Transformerscited by 59
  4. 2022Relational Attention: Generalizing Transformers for Graph-Structured Taskscited by 41
  5. 2023Smooth, exact rotational symmetrization for deep learning on point cloudscited by 39
  6. 2023Sparse Universal Transformercited by 32
  7. 2024Graph Transformers: A Surveycited by 32
  8. 2022Uncontrolled Lexical Exposure Leads to Overestimation of Compositional Generalization in Pretrained Modelscited by 26
  9. 2023Neural-Logic Human-Object Interaction Detectioncited by 25
  10. 2023ReCOGS: How Incidental Details of a Logical Form Overshadow an Evaluation of Semantic Interpretationcited by 24
  11. 2023Systematic Visual Reasoning through Object-Centric Relational Abstractioncited by 20
  12. 2025Strassen Attention, Split VC Dimension and Compositionality in Transformerscited by 12
26 more
  1. 2024Towards Principled Graph Transformerscited by 11
  2. 2025How simple can you go? An off-the-shelf transformer approach to molecular dynamicscited by 10
  3. 2025Fast and Simplex: 2-Simplicial Attention in Tritoncited by 8
  4. 2025Analyzing the Inner Workings of Transformers in Compositional Generalizationcited by 6
  5. 2022Transformer Module Networks for Systematic Generalization in Visual Question Answeringcited by 5
  6. 2023Learning Language Representations with Logical Inductive Biascited by 5
  7. 2024Towards Understanding the Relationship between In-context Learning and Compositional Generalizationcited by 5
  8. 2024Systematic Relational Reasoning With Epistemic Graph Neural Networkscited by 4
  9. 2024Automatic Screening for Children with Speech Disorder using Automatic Speech Recognition: Opportunities and Challengescited by 3
  10. 2023Laziness Is a Virtue When It Comes to Compositionality in Neural Semantic Parsingcited by 2
  11. 2023D3: Data Diversity Design for Systematic Generalization in Visual Question Answeringcited by 2
  12. 2023MetaReVision: Meta-Learning with Retrieval for Visually Grounded Compositional Concept Acquisitioncited by 2
  13. 2024SHAKTI: A 2.5 Billion Parameter Small Language Model Optimized for Edge AI and Low-Resource Environmentscited by 2
  14. 2025When No Paths Lead to Rome: Benchmarking Systematic Neural Relational Reasoningcited by 2
  15. 2022Embeddings as Epistemic States: Limitations on the Use of Pooling Operators for Accumulating Knowledgecited by 1
  16. 2023HICO-DET-SG and V-COCO-SG: New Data Splits for Evaluating the Systematic Generalization Performance of Human-Object Interaction Detection Modelscited by 1
  17. 2023A Data Source for Reasoning Embodied Agentscited by 1
  18. 2026FloydNet: A Learning Paradigm for Global Relational Reasoningcited by 1
  19. 2026Graph Machine: Exploring Edge Mechanisms as an Inductive Biascited by 1
  20. 2022A Short Survey of Systematic Generalizationcited by 0
  21. 2023Friend Ranking in Online Games via Pre-training Edge Transformerscited by 0
  22. 2025Scaling Graph-Based Dependency Parsing with Arc Vectorization and Attention-Based Refinementcited by 0
  23. 2026Efficient Lookahead Encoding and Abstracted Width for Learning General Policies in Classical Planningcited by 0
  24. 2026Project Auto-World: Towards Automated Benchmarking of Neural Relational Reasonerscited by 0
  25. 2026Higher-Order Fourier Neural Operator: Explicit Mode Mixer for Nonlinear PDEscited by 0
  26. 2026Linearized 2-Simplicial Attentioncited by 0