mlsanityData to 7 Oct 2026

A Neural Tangent Kernel Perspective of Infinite Tree Ensembles

arXiv 2109.04983 ↗First posted 10 Sep 2021cs.LG stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

In practical situations, the tree ensemble is one of the most popular models along with neural networks. A soft tree is a variant of a decision tree. Instead of using a greedy method for searching splitting rules, the soft tree is trained using a gradient method in which the entire splitting operation is formulated in a differentiable form. Although ensembles of such soft trees have been used increasingly in recent years, little theoretical work has been done to understand their behavior. By considering an ensemble of infinite soft trees, this paper introduces and studies the Tree Neural Tangent Kernel (TNTK), which provides new insights into the behavior of the infinite ensemble of soft trees. Using the TNTK, we theoretically identify several non-trivial properties, such as global convergence of the training, the equivalence of the oblivious tree structure, and the degeneracy of the TNTK induced by the deepening of the trees.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 24 of its 45 references that are in the map. First posted on arXiv 10 Sep 2021.

Builds on

24 of 45 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  5. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  6. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  7. 2018On Lazy Training in Differentiable Programmingcited by 671
  8. 2019TabNet: Attentive Interpretable Tabular Learningcited by 640
  9. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  10. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  11. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 337
  12. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
12 more
  1. 2017Distilling a Neural Network Into a Soft Decision Treecited by 230
  2. 2019Neural Oblivious Decision Ensembles for Deep Learning on Tabular Datacited by 200
  3. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 168
  4. 2019Sparse Sequence-to-Sequence Modelscited by 167
  5. 2019Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernelscited by 160
  6. 2019Harnessing the Power of Infinitely Wide Deep Nets on Small-data Taskscited by 87
  7. 2019Enhanced Convolutional Neural Tangent Kernelscited by 76
  8. 2020The Recurrent Neural Tangent Kernelcited by 67
  9. 2018Adaptive Neural Treescited by 64
  10. 2020Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspectivecited by 60
  11. 2020The Tree Ensemble Layer: Differentiability meets Conditional Computationcited by 57
  12. 2021Learning Accurate Decision Trees with Bandit Feedback via Quantized Gradient Descentcited by 14

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 1; 2024: 1.

4 papers.

  1. 2023Random Smoothing Regularization in Kernel Gradient Descent Learningcited by 6
  2. 2022Gradient Boosting Performs Gaussian Process Inferencecited by 4
  3. 2022Analyzing Tree Architectures in Ensembles via Neural Tangent Kernelcited by 2
  4. 2024Linear Mode Connectivity in Differentiable Tree Ensemblescited by 1