mlsanityData to 7 Oct 2026

Learning a Decision Tree Algorithm with Transformers

arXiv 2402.03774 ↗First posted 6 Feb 2024cs.LG cs.AI cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Decision trees are renowned for their ability to achieve high predictive performance while remaining interpretable, especially on tabular data. Traditionally, they are constructed through recursive algorithms, where they partition the data at every node in a tree. However, identifying a good partition is challenging, as decision trees optimized for local segments may not yield global generalization. To address this, we introduce MetaTree, a transformer-based model trained via meta-learning to directly produce strong decision trees. Specifically, we fit both greedy decision trees and globally optimized decision trees on a large number of datasets, and train MetaTree to produce only the trees that achieve strong generalization performance. This training enables MetaTree to emulate these algorithms and intelligently adapt its strategy according to the context, thereby achieving superior generalization performance.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 35 of its 69 references that are in the map. First posted on arXiv 6 Feb 2024.

Builds on

35 of 69 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  6. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  7. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  8. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  9. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
  10. 2014OpenML: networked science in machine learningcited by 523
  11. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  12. 2019Interpretable machine learning: definitions, methods, and applicationscited by 303
23 more
  1. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 273
  2. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  3. 2017Distilling a Neural Network Into a Soft Decision Treecited by 230
  4. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  5. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 198
  6. 2019Optimal Sparse Decision Treescited by 104
  7. 2020Generalized and Scalable Optimal Sparse Decision Treescited by 98
  8. 2023XTab: Cross-table Pretraining for Tabular Transformerscited by 81
  9. 2018Adaptive Neural Treescited by 64
  10. 2020PMLB v1.0: An open source dataset collection for benchmarking machine learning methodscited by 58
  11. 2018Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Insteadcited by 40
  12. 2022Augmenting Interpretable Models with LLMs during Trainingcited by 36
  13. 2016Neural Random Forestscited by 25
  14. 2023Language models are weak learnerscited by 21
  15. 2022Hierarchical Shrinkage: improving the accuracy and interpretability of tree-based methodscited by 18
  16. 2023Scaling TabPFN: Sketching and Feature Selection for Tabular Prior-Data Fitted Networkscited by 18
  17. 2023Tree Prompting: Efficient Task Adaptation without Fine-Tuningcited by 17
  18. 2022Learning Optimal Fair Classification Trees: Trade-offs Between Interpretability, Fairness, and Accuracycited by 15
  19. 2019Oblique Decision Trees from Derivatives of ReLU Networkscited by 14
  20. 2023TabRet: Pre-training Transformer-based Tabular Models for Unseen Columnscited by 14
  21. 2020Learning Binary Decision Trees by Argmin Differentiationcited by 11
  22. 2020Random Planted Forest: a directly interpretable tree ensemblecited by 5
  23. 2021Sparse Oblique Decision Trees: A Tool to Understand and Manipulate Neural Net Featurescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 2.

6 papers.

  1. 2025Zero-Shot Decision Tree Construction via Large Language Modelscited by 3
  2. 2024Through the Thicket: A Study of Number-Oriented LLMs derived from Random Forest Modelscited by 2
  3. 2024AutoSpec: Automated Generation of Neural Network Specificationscited by 2
  4. 2026Selecting Feature Interactions for Generalized Additive Models by Distilling Foundation Modelscited by 1
  5. 2026Agentic-imodels: Evolving agentic interpretability tools via autoresearchcited by 1
  6. 2025Towards Scalable Meta-Learning of near-optimal Interpretable Models via Synthetic Model Generationscited by 0