# Large Language Models for Automated Cross-Domain Machine Learning Task Type Identification: A Benchmark Dataset and Evaluation

arXiv: https://arxiv.org/abs/2609.35335 · HTML: https://mlsanity.com/paper/2609.35335/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 64 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Builds on (most cited first, 28 of 28)

- [Deep learning for time series classification: a review](https://mlsanity.com/paper/1809.04356/) (2018)
- [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://mlsanity.com/paper/2004.02349/) (2020)
- [AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data](https://mlsanity.com/paper/2003.06505/) (2020)
- [MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation](https://mlsanity.com/paper/2310.03302/) (2023)
- [TabLLM: Few-shot Classification of Tabular Data with Large Language Models](https://mlsanity.com/paper/2210.10723/) (2022)
- [TAPEX: Table Pre-training via Learning a Neural SQL Executor](https://mlsanity.com/paper/2107.07653/) (2021)
- [Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs](https://mlsanity.com/paper/2402.03927/) (2024)
- [Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning](https://mlsanity.com/paper/2007.04074/) (2020)
- [Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering](https://mlsanity.com/paper/2305.03403/) (2023)
- [Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study](https://mlsanity.com/paper/2305.13062/) (2023)
- [FLAML: A Fast and Lightweight AutoML Library](https://mlsanity.com/paper/1911.04706/) (2019)
- [Bake off redux: a review and experimental evaluation of recent time series classification algorithms](https://mlsanity.com/paper/2304.13029/) (2023)
- [TUTA: Tree-based Transformers for Generally Structured Table Pre-training](https://mlsanity.com/paper/2010.12537/) (2020)
- [TableFormer: Table Structure Understanding with Transformers](https://mlsanity.com/paper/2203.01017/) (2022)
- [Investigating Data Contamination for Pre-training Language Models](https://mlsanity.com/paper/2401.06059/) (2024)
- [AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML](https://mlsanity.com/paper/2410.02958/) (2024)
- [AutoML-GPT: Automatic Machine Learning with GPT](https://mlsanity.com/paper/2305.02499/) (2023)
- [MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Tasks](https://mlsanity.com/paper/2304.14979/) (2023)
- [AutoML to Date and Beyond: Challenges and Opportunities](https://mlsanity.com/paper/2010.10777/) (2020)
- [HTLM: Hyper-Text Pre-Training and Prompting of Language Models](https://mlsanity.com/paper/2107.06955/) (2021)
- [MATE: Multi-view Attention for Table Transformer Efficiency](https://mlsanity.com/paper/2109.04312/) (2021)
- [A Zero-shot and Few-shot Study of Instruction-Finetuned Large Language Models Applied to Clinical and Biomedical Tasks](https://mlsanity.com/paper/2307.12114/) (2023)
- [Characterizing and Detecting Mismatch in Machine-Learning-Enabled Systems](https://mlsanity.com/paper/2103.14101/) (2021)
- [Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Survey](https://mlsanity.com/paper/2411.10478/) (2024)
- [GAMA: a General Automated Machine learning Assistant](https://mlsanity.com/paper/2007.04911/) (2020)
- [Naive Automated Machine Learning](https://mlsanity.com/paper/2111.14514/) (2021)
- [Adaptable and Reliable Text Classification using Large Language Models](https://mlsanity.com/paper/2405.10523/) (2024)
- [Detection of Personal Data in Structured Datasets Using a Large Language Model](https://mlsanity.com/paper/2506.22305/) (2025)

## Abstract (arXiv)

Machine learning task type identification is essential for constructing valid ML pipelines, yet in practice it is typically specified manually. We investigate whether large language models (LLMs) can infer both the data domain and the downstream prediction task directly from dataset-level information when only the target feature is provided by the user. Together with our LLM-based system we also release an annotated benchmark comprising 625 public tabular and time series datasets. We evaluate the proposed approach in three settings: (i) tabular datasets in comparison with established AutoML heuristics, (ii) cross-domain evaluation across tabular and time series datasets, and (iii) a practical deployment scenario using smaller local models. The results show consistent advantages for LLM-based task type identification, with increasing difficulty in heterogeneous and resource-constrained settings. LLM-based approaches outperform AutoGluon in the tabular setting, reaching 0.98 F1 macro compared to 0.93. In the cross-domain setting, the best model achieves 0.90 F1 macro, while smaller locally deployable models reach 0.75, indicating a trade-off between deployment feasibility and accuracy.