mlsanityData to 14 Oct 2026

ATTUNE: Agentic Throughput Tuning for LLM Training

arXiv 2610.17528 ↗First posted 13 Oct 2026cs.DC cs.LGcited by 0

New in LLM agents and tool use; no direction yet. LLM agents and tool use

Abstract (arXiv)

Training recipes are tuned by hand for one model, cluster and workload, and every new model, accelerator, framework version or dataset means tuning again. Automated tuners search a fixed space of knobs and check only that a job runs, yet a configuration can run to completion while it silently changes training. ATTUNE is a single coding agent that tunes any working recipe for the workload and stack at hand, unattended, within a GPU-hour budget the user sets. It diagnoses each run and changes one lever at a time, guided by a catalog of lessons from past campaigns, and ships a recipe only if it passes a correctness gate: a configuration check against objective-changing settings and known stack defects, and a convergence check of its loss curve against the baseline's. On four held-out models at 64-128 GPUs, ATTUNE raises mean throughput over the strongest public recipe run on our stack by 6.7% on Llama-3.1-405B (best campaign 11.9%) and 33.9% on GLM-4.5 (54.7%), whose public recipe was written for H100. It keeps Mixtral-8x22B's recipe, where nothing it finds clears the noise band. Under a common GPU-hour budget it beats OptiCo, Bayesian optimization and random search on GLM-4.5 and MiniMax-M2; on GLM-4.5 its mean gain is more than four times the best alternative's at less than half the GPU-hours, and on both models its tuning pays for itself within a day of training. The gate rejects no legitimate run and catches 16 of 24 faulty runs, missing only ones that barely move the 20-step loss. Ablations tie its best campaigns to one of its search rules: retrying an out-of-memory layout with recomputation. We release ATTUNE's code and TrainingThroughputBench, a leaderboard of the fastest known recipes for MoE models up to 744B parameters on up to 256 GPUs, each with its throughput, cost per million tokens and a passed correctness check, at https://github.com/amazon-science/attune.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 36 of its 59 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

36 of 59 references.

References · 36

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  4. 2024Mixtral of Expertscited by 1,823
  5. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  6. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  7. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  8. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  9. 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 867
  10. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  11. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  12. 2025Why Do Multi-Agent LLM Systems Fail?cited by 604
24 more
  1. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  2. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  3. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 179
  4. 2024Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?cited by 159
  5. 2019MLPerf Training Benchmarkcited by 115
  6. 2023Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMscited by 114
  7. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 112
  8. 2025Barbarians at the Gate: How AI is Upending Systems Researchcited by 54
  9. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 53
  10. 2023GPTuner: A Manual-Reading Database Tuning System via GPT-Guided Bayesian Optimizationcited by 29
  11. 2023vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal Large Language Model Trainingcited by 23
  12. 2025MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Corecited by 17
  13. 2025Sailor: Automating Distributed Training over Dynamic, Heterogeneous, and Geo-distributed Clusterscited by 13
  14. 2026TREX: Automating LLM Fine-tuning via Agent-Driven Tree-based Explorationcited by 8
  15. 2025Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimizationcited by 7
  16. 2024Rubick: Exploiting Job Reconfigurability for Deep Learning Cluster Schedulingcited by 4
  17. 2025ASAP: an Agentic Solution to Auto-optimize Performance of Large-Scale LLM Trainingcited by 3
  18. 2026CUCo: An Agentic Framework for Compute and Communication Co-designcited by 2
  19. 2026CCL-Bench 1.0: A Trace-Based Benchmark for LLM Infrastructurecited by 2
  20. 2024CubicML: Automated ML for Large ML Systems Co-design with ML Prediction of Performancecited by 1
  21. 2026STELLAR: Storage Tuning Engine Leveraging LLM Autonomous Reasoning for High Performance Parallel File Systemscited by 1
  22. 2026Rudder: Steering Prefetching in Distributed GNN Training using LLM Agentscited by 1
  23. 2026AutoScout: Structured Optimization for Automating ML System Configurationcited by 1
  24. 2026Searching for Plans You Can Actually Build: A Realizability-Aware Full-Space Optimizer for MoE Training and Servingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026AutoTuneBench: Trustworthy Measurement for Agent Auto-Tuning of LLM Serving Engines
  2. 2024Star-Agents: Automatic Data Optimization with LLM Agents for Instruction Tuning
  3. 2026Atropos: Improving Cost-Benefit Trade-off of LLM-based Agents under Self-Consistency with Early Termination and Model Hotswap
  4. 2026PostTrainBench: Can LLM Agents Automate LLM Post-Training?
  5. 2025Balanced and Elastic End-to-end Training of Dynamic LLMs
  6. 2026ATTest: Agent-Driven Tensor Testing for Deep Learning Library Modules
  7. 2026Fast and Accurate Probing of In-Training LLMs' Downstream Performances
  8. 2025Scaling Laws for Upcycling Mixture-of-Experts Language Models