mlsanityData to 14 Oct 2026

Agentic Search Spaces for Tabular Machine Learning

arXiv 2609.16309 ↗First posted 14 Sep 2026cs.LGcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Despite the rapid progress of LLM-based agents for planning, code generation, and debugging, their practical value for tabular machine learning remains underexplored. In this paper, we investigate a concrete use case: whether state-of-the-art agentic AI systems can design extended HPO search spaces for established tabular models that outperform the standard search spaces provided by the model authors. Specifically, we represent each tabular model as a modular pipeline covering preprocessing, embeddings, architecture, training, and inference. We then task the agent to propose candidate code implementations for each module and use a classical HPO algorithm to jointly optimize over these candidates and the model's default hyperparameters. Compared with the base HPO spaces, the expanded search spaces improve the performance of nearly every model family across a suite of 45 datasets, with average relative gains of 0.6%, rising to 2.0% on small-to-medium regression datasets. Notably, these gains come at no extra tuning cost: the enlarged spaces outperform the base under the same tuning and ensembling budgets. The gains transfer to the recent TabArena benchmark, where the agentic spaces improve the official Elo scores of four of the five model families and the two strongest agentic ensembles surpass the best AutoGluon ensemble of conventional models. Overall, our study suggests that LLM agents can provide practical value for tabular ML by expanding the design space.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 55 of its 82 references that are in the map. First posted on arXiv 14 Sep 2026.

Builds on

55 of 82 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026No Time Like the Present: Agentic Test-Time Training for LLM Agents3 Jul
  2. 2026Can LLMs Beat Classical Hyperparameter Optimization Algorithms? A Study on autoresearch25 Mar

Other references · 53

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  6. 2017On Calibration of Modern Neural Networkscited by 4,960
  7. 2017Squeeze-and-Excitation Networkscited by 4,789
  8. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  9. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  10. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  11. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
  12. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
38 more
  1. 2020GLU Variants Improve Transformercited by 1,804
  2. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  3. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  4. 2019Root Mean Square Layer Normalizationcited by 1,480
  5. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  6. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  7. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  8. 2021Revisiting Deep Learning Models for Tabular Datacited by 740
  9. 2016Language Modeling with Gated Convolutional Networkscited by 702
  10. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
  11. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  12. 2020AutoGluon-Tabular: Robust and Accurate AutoML for Structured Datacited by 383
  13. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 371
  14. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 336
  15. 2025TabICL: A Tabular Foundation Model for In-Context Learning on Large Datacited by 289
  16. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 277
  17. 2023From Sparse to Soft Mixtures of Expertscited by 248
  18. 2022On Embeddings for Numerical Features in Tabular Deep Learningcited by 186
  19. 2025TabArena: A Living Benchmark for Machine Learning on Tabular Datacited by 176
  20. 2018Auto-Keras: An Efficient Neural Architecture Search Systemcited by 171
  21. 2026TabICLv2: A better, faster, scalable, and open tabular foundation modelcited by 144
  22. 2024Large Language Models to Enhance Bayesian Optimizationcited by 135
  23. 2023Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineeringcited by 127
  24. 2024TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensemblingcited by 113
  25. 2024Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Datacited by 92
  26. 2016Entity Embeddings of Categorical Variablescited by 87
  27. 2022Towards Learning Universal Hyperparameter Optimizers with Transformerscited by 81
  28. 2024Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learningcited by 81
  29. 2025LimiX: Unleashing Structured-Data Modeling Capability for Generalist Intelligencecited by 79
  30. 2024Can large language models explore in-context?cited by 76
  31. 2024Large Scale Transfer Learning for Tabular Data via Language Modelingcited by 76
  32. 2023TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023cited by 74
  33. 2022AMLB: an AutoML Benchmarkcited by 55
  34. 2023TabRepo: A Large Scale Repository of Tabular Model Evaluations and its AutoML Applicationscited by 33
  35. 2024TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarkscited by 33
  36. 2024Revisiting Nearest Neighbor for Tabular Data: A Deep Tabular Baseline Two Decades Latercited by 30
  37. 2025Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddingscited by 7
  38. 2025Thompson Sampling via Fine-Tuning of LLMscited by 6

50 most cited of 53 All 53 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026TabAgent: A Framework for Replacing Agentic Generative Components with Tabular-Textual Classifiers
  2. 2025OPT-BENCH: Evaluating LLM Agent on Large-Scale Search Spaces Optimization Problems
  3. 2025TAGAL: Tabular Data Generation using Agentic LLM Methods
  4. 2024Large Language Model Agent for Hyper-Parameter Optimization
  5. 2026AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers
  6. 2024SELA: Tree-Search Enhanced LLM Agents for Automated Machine Learning
  7. 2025Can Agentic AI Match the Performance of Human Data Scientists?
  8. 2026ASAP: Agent-System Co-Design for Wall-Clock-Centered Auto HPO Research for ML Experiments