mlsanityData to 7 Oct 2026

Optimizing ML Training with Metagradient Descent

arXiv 2503.13751 ↗First posted 17 Mar 2025stat.ML cs.AI cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

A major challenge in training large-scale machine learning models is configuring the training process to maximize model performance, i.e., finding the best training setup from a vast design space. In this work, we unlock a gradient-based approach to this problem. We first introduce an algorithm for efficiently calculating metagradients -- gradients through model training -- at scale. We then introduce a "smooth model training" framework that enables effective optimization using metagradients. With metagradient descent (MGD), we greatly improve on existing dataset selection methods, outperform accuracy-degrading data poisoning attacks by an order of magnitude, and automatically find competitive learning rate schedules.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Mar 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 9.

22 papers.

Sort:

By area:

  1. 2025End-to-End Test-Time Training for Long Contextcited by 53
  2. 2025DataMIL: Selecting Data for Robot Imitation Learning with Datamodelscited by 25
  3. 2025CUPID: Curating Data your Robot Loves with Influence Functionscited by 21
  4. 2025MAGIC: Near-Optimal Data Attribution for Deep Learningcited by 20
  5. 2025Ambient Diffusion Omni: Training Good Models with Bad Datacited by 18
  6. 2025DataRater: Meta-Learned Dataset Curationcited by 13
  7. 2025How Reliable is Language Model Micro-Benchmarking?cited by 8
  8. 2025Optimizing Canaries for Privacy Auditing with Metagradient Descentcited by 6
  9. 2026Efficient Estimation of Kernel Surrogate Models for Task Attributioncited by 4
  10. 2026Infusion: Shaping Model Behavior by Editing Training Data via Influence Functionscited by 4
  11. 2025Rescaled Influence Functions: Accurate Data Attribution in High Dimensioncited by 3
  12. 2025Train on Validation (ToV): Fast data selection with applications to fine-tuningcited by 3
10 more
  1. 2025Robust and Diverse Multi-Agent Learning via Rational Policy Gradientcited by 2
  2. 2026How to sketch a learning algorithmcited by 2
  3. 2026Synthetic Data for any Differentiable Targetcited by 2
  4. 2026Bergson: An Open Source Library for Data Attributioncited by 2
  5. 2026Generalization Guarantees on Data-Driven Tuning of Gradient Descent with Langevin Updatescited by 1
  6. 2026NoiseRater: Meta-Learned Noise Valuation for Diffusion Model Trainingcited by 1
  7. 2025On the Accuracy of Newton Step and Influence Function Data Attributionscited by 0
  8. 2025Understanding the Gain from Data Filtering in Multimodal Contrastive Learningcited by 0
  9. 2026TADS: Task-Aware Data Selection for Multi-Task Multimodal Pre-Trainingcited by 0
  10. 2026Data Attribution via Sketched Metadifferentiationcited by 0