mlsanityData to 14 Oct 2026

AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Risks

arXiv 2306.08107 ↗First posted 13 Jun 2023cs.LG cs.CLcited by 35

Not on the map. Why?

Abstract (arXiv)

The fields of both Natural Language Processing (NLP) and Automated Machine Learning (AutoML) have achieved remarkable results over the past years. In NLP, especially Large Language Models (LLMs) have experienced a rapid series of breakthroughs very recently. We envision that the two fields can radically push the boundaries of each other through tight integration. To showcase this vision, we explore the potential of a symbiotic relationship between AutoML and LLMs, shedding light on how they can benefit each other. In particular, we investigate both the opportunities to enhance AutoML approaches with LLMs from different perspectives and the challenges of leveraging AutoML to further improve LLMs. To this end, we survey existing work, and we critically assess risks. We strongly believe that the integration of the two fields has the potential to disrupt both fields, NLP and AutoML. By highlighting conceivable synergies, but also risks, we aim to foster further exploration at the intersection of AutoML and LLMs.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 130 of its 210 references that are in the map. First posted on arXiv 13 Jun 2023.

Builds on

130 of 210 references.

References · 130

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  10. 2020Scaling Laws for Neural Language Modelscited by 6,586
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
38 more
  1. 2016Layer Normalizationcited by 5,669
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  10. 2023A Survey of Large Language Modelscited by 2,576
  11. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
  12. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  13. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  14. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  15. 2017Decoupled Weight Decay Regularizationcited by 1,831
  16. 2020GLU Variants Improve Transformercited by 1,804
  17. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  18. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  19. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  20. 2020On Layer Normalization in the Transformer Architecturecited by 922
  21. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  22. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  23. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  24. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  25. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  26. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 699
  27. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
  28. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  29. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  30. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  31. 2020Knowledge Graphscited by 621
  32. 2016Learning to learn by gradient descent by gradient descentcited by 620
  33. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  34. 2018Explaining Explanations: An Overview of Interpretability of Machine Learningcited by 508
  35. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 416
  36. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  37. 2020AutoGluon-Tabular: Robust and Accurate AutoML for Structured Datacited by 383
  38. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 360

50 most cited of 130 All 130 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 15; 2025: 5; 2026: 10.

35 papers.

Sort:

By area:

  1. 2023Challenges and Applications of Large Language Modelscited by 254
  2. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 186
  3. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 175
  4. 2025MLGym: A New Framework and Benchmark for Advancing AI Research Agentscited by 124
  5. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  6. 2024AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMLcited by 87
  7. 2025AgentRxiv: Towards Collaborative Autonomous Researchcited by 72
  8. 2024Large Language Model Agent for Hyper-Parameter Optimizationcited by 64
  9. 2023AutoGluon-TimeSeries: AutoML for Probabilistic Time Series Forecastingcited by 36
  10. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  11. 2023Large Language Model-Enhanced Algorithm Selection: Towards Comprehensive Algorithm Representationcited by 21
  12. 2024LLM4PLC: Harnessing Large Language Models for Verifiable Programming of PLCs in Industrial Control Systemscited by 21
23 more
  1. 2024Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Surveycited by 19
  2. 2024Position: A Call to Action for a Human-Centered AutoML Paradigmcited by 14
  3. 2023QualEval: Qualitative Evaluation for Model Improvementcited by 11
  4. 2024A Review on Generative AI Models for Synthetic Medical Text, Time Series, and Longitudinal Datacited by 10
  5. 2024Large Language Models Synergize with Automated Machine Learningcited by 9
  6. 2024Bag of Tricks for Multimodal AutoML with Image, Text, and Tabular Datacited by 7
  7. 2026Agentic Bayesian Optimization through Surrogate-Augmented Autoresearchcited by 6
  8. 2026TabFM: A Zero-Shot Foundation Model for Tabular Datacited by 6
  9. 2024CliMB: An AI-enabled Partner for Clinical Predictive Modelingcited by 2
  10. 2026Accelerating Scientific Research with Gemini in the Real-Worldcited by 2
  11. 2024Data augmentation with automated machine learning: approaches and performance comparison with classical data augmentation methodscited by 1
  12. 2024Proficient Graph Neural Network Design by Accumulating Knowledge on Large Language Modelscited by 1
  13. 2026Data-Local Autonomous LLM-Guided Neural Architecture Search for Multiclass Multimodal Time-Series Classificationcited by 1
  14. 2024Capturing and Anticipating User Intents in Data Analytics via Knowledge Graphscited by 0
  15. 2025Large Language Models in Cybersecurity: Applications, Vulnerabilities, and Defense Techniquescited by 0
  16. 2025Pre-Hoc Predictions in AutoML: Leveraging LLMs to Enhance Model Selection and Benchmarking for Tabular datasetscited by 0
  17. 2025Are We Aligned? A Preliminary Investigation of the Alignment of Responsible AI Values between LLMs and Human Judgmentcited by 0
  18. 2026Learning When to Trust LLM Priors: A Validated Framework for Semantic Prior Integrationcited by 0
  19. 2026Algorithmic Blindness in Large Language Models: A Calibration Study of Performance Predictioncited by 0
  20. 2026When Prompts Become Payloads: A Framework for Mitigating SQL Injection Attacks in Large Language Model-Driven Applicationscited by 0
  21. 2026SAGE-Loop: Reliable Closed-Loop LLM-Driven AutoML with Trial-and-Correction and Adaptive Ensemblingcited by 0
  22. 2026An LLM-Assisted AutoML Framework for Intrusion Detection in IoT Networkscited by 0
  23. 2026TabFM-Auto: Self-Evolving Pipelines for Tabular Foundation Modelscited by 0