mlsanityData to 7 Oct 2026

Terminating Differentiable Tree Experts

arXiv 2407.02060 ↗First posted 2 Jul 2024cs.LG cs.AI cs.SCcited by 1

Not on the map. Why?

Abstract (arXiv)

We advance the recently proposed neuro-symbolic Differentiable Tree Machine, which learns tree operations using a combination of transformers and Tensor Product Representations. We investigate the architecture and propose two key components. We first remove a series of different transformer layers that are used in every step by introducing a mixture of experts. This results in a Differentiable Tree Experts model with a constant number of parameters for any arbitrary number of steps in the computation, compared to the previous method in the Differentiable Tree Machine with a linear growth. Given this flexibility in the number of steps, we additionally propose a new termination algorithm to provide the model the power to choose how many steps to make automatically. The resulting Terminating Differentiable Tree Experts model sluggishly learns to predict the number of steps without an oracle. It can do so while maintaining the learning capabilities of the model, converging to the optimal amount of steps.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 18 of its 29 references that are in the map. First posted on arXiv 2 Jul 2024.

Builds on

18 of 29 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2024Mixtral of Expertscited by 1,785
  5. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  6. 2019Neural Probabilistic Logic Programming in DeepProbLogcited by 389
  7. 2019The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervisioncited by 369
  8. 2017A Semantic Loss Function for Deep Learning with Symbolic Knowledgecited by 267
  9. 2020DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learningcited by 159
  10. 2021PonderNet: Learning to Pondercited by 156
  11. 2020Compositional Generalization via Neural-Symbolic Stack Machinescited by 80
  12. 2022A Neuro-vector-symbolic Architecture for Solving Raven's Progressive Matricescited by 64
6 more
  1. 2022Neurocompositional computing: From the Central Paradox of Cognition to a new generation of AI systemscited by 44
  2. 2021Abstract Spatial-Temporal Reasoning via Probabilistic Abduction and Executioncited by 43
  3. 2019Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solvingcited by 34
  4. 2021Enriching Transformers with Structured Tensor-Product Representations for Abstractive Summarizationcited by 11
  5. 2017Question-Answering with Grammatically-Interpretable Representationscited by 8
  6. 2023Differentiable Tree Operations Promote Compositional Generalizationcited by 5

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Can Large Reasoning Models do Analogical Reasoning under Perceptual Uncertainty?cited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2026Differentiable Symbolic Planning: A Neural Architecture for Constraint Reasoning with Learned Feasibility
  2. 2025Neurosymbolic Decision Trees
  3. 2026Differentiable Logic Programming to Mitigate Reasoning Shortcuts in Neurosymbolic Systems
  4. 2022Differentiable Fuzzy ALC: A Neural-Symbolic Representation Language for Symbol Grounding
  5. 2025Multi-head Transformers Provably Learn Symbolic Multi-step Reasoning via Gradient Descent
  6. 2026DPNL: A DPLL-based Algorithm for Probabilistic Neurosymbolic Learning
  7. 2025TreeGPT: Pure TreeFFN Encoder-Decoder Architecture for Structured Reasoning Without Attention Mechanisms
  8. 2026Form Follows Function: Recursive Stem Model