mlsanityData to 7 Oct 2026

Differentiable Tree Operations Promote Compositional Generalization

arXiv 2306.00751 ↗First posted 1 Jun 2023cs.CL cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

In the context of structure-to-structure transformation tasks, learning sequences of discrete symbolic operations poses significant challenges due to their non-differentiability. To facilitate the learning of these symbolic sequences, we introduce a differentiable tree interpreter that compiles high-level symbolic tree operations into subsymbolic matrix operations on tensors. We present a novel Differentiable Tree Machine (DTM) architecture that integrates our interpreter with an external memory and an agent that learns to sequentially select tree operations to execute the target transformation in an end-to-end manner. With respect to out-of-distribution compositional generalization on synthetic semantic parsing and language generation tasks, DTM achieves 100% while existing baselines such as Transformer, Tree Transformer, LSTM, and Tree2Tree LSTM achieve less than 30%. DTM remains highly interpretable in addition to its perfect performance.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 29 of its 43 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

29 of 43 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  3. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  4. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  5. 2014Neural Turing Machinescited by 630
  6. 2015Neural Module Networkscited by 375
  7. 2014Memory Networkscited by 354
  8. 2015Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networkscited by 305
  9. 2019Compositionality decomposed: how do neural networks generalise?cited by 264
  10. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 260
  11. 2016Language to Logical Form with Neural Attentioncited by 158
  12. 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformerscited by 117
17 more
  1. 2021A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part I: Models and Data Transformationscited by 98
  2. 2015Neural Programmer-Interpreterscited by 96
  3. 2018Tree-to-tree Neural Networks for Program Translationcited by 93
  4. 2015Inferring Algorithmic Patterns with Stack-Augmented Recurrent Netscited by 76
  5. 2019Tree Transformer: Integrating Tree Structures into Self-Attentioncited by 68
  6. 2015Learning to Transduce with Unbounded Memorycited by 54
  7. 2022Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scalecited by 40
  8. 2019Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solvingcited by 34
  9. 2016Programming with a Differentiable Forth Interpretercited by 33
  10. 2019Discovering the Compositional Structure of Vector Representations with Role Learning Networkscited by 33
  11. 2015Neural Random-Access Machinescited by 32
  12. 2018Learning to Reason with Third-Order Tensor Productscited by 27
  13. 2022Uncontrolled Lexical Exposure Leads to Overestimation of Compositional Generalization in Pretrained Modelscited by 26
  14. 2018RNNs Implicitly Implement Tensor Product Representationscited by 25
  15. 2021Enriching Transformers with Structured Tensor-Product Representations for Abstractive Summarizationcited by 11
  16. 2017Question-Answering with Grammatically-Interpretable Representationscited by 8
  17. 2022Structural Biases for Improving Transformers on Translation into Morphologically Rich Languagescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 0; 2026: 1.

5 papers.

  1. 2024From Frege to chatGPT: Compositionality in language, cognition, and deep neural networkscited by 7
  2. 2024Compositional Generalization Across Distributional Shifts with Sparse Tree Operationscited by 5
  3. 2024Terminating Differentiable Tree Expertscited by 1
  4. 2024Discrete Dictionary-based Decomposition Layer for Structured Representation Learningcited by 0
  5. 2026Recursive Binding on a Budget: Subspace Carving in Order-p Tensor Memoriescited by 0