mlsanityData to 14 Oct 2026

Commute Your Domains: Trajectory Optimality Criterion for Multi-Domain Learning

arXiv 2501.15556 ↗First posted 26 Jan 2025cs.LG cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

In multi-domain learning, a single model is trained on diverse data domains to leverage shared knowledge and improve generalization. The order in which the data from these domains is used for training can significantly affect the model's performance on each domain. However, this dependence is under-studied. In this paper, we investigate the influence of training order (or data mixing) in multi-domain learning using the concept of Lie bracket of gradient vector fields. By analyzing the infinitesimal effects of changing the training order, we identify regions in the parameter space where altering the order between two training domains can benefit the target loss. We validate the predictions of our theoretical framework on the influence of training order (or data mixing) both on a toy example and bilingual LLM pre-training.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 29 of its 39 references that are in the map. First posted on arXiv 26 Jan 2025.

Builds on

29 of 39 references.

References · 29

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Qwen Technical Reportcited by 3,344
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  5. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  6. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  7. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
  8. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  9. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 405
  10. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 365
  11. 2019Pareto Multi-Task Learningcited by 224
  12. 2022Multi-Task Learning as a Bargaining Gamecited by 215
17 more
  1. 2020Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropoutcited by 201
  2. 2016UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memorycited by 200
  3. 2015The Benefit of Multitask Representation Learningcited by 195
  4. 2020Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Modelscited by 169
  5. 2020Understanding Self-Training for Gradual Domain Adaptationcited by 149
  6. 2021Scaling Up Influence Functionscited by 132
  7. 2021RotoGrad: Gradient Homogenization in Multitask Learningcited by 83
  8. 2023Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flowscited by 45
  9. 2018The loss landscape of overparameterized neural networkscited by 39
  10. 2022AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task Learningcited by 31
  11. 2022Gradient-based Bi-level Optimization for Deep Learning: A Surveycited by 26
  12. 2022How Does Adaptive Optimization Impact Local Neural Network Geometry?cited by 24
  13. 2023Scalarization for Multi-Task and Multi-Domain Learning at Scalecited by 19
  14. 2021Sequential Reptile: Inter-Task Gradient Alignment for Multilingual Learningcited by 16
  15. 2022On the Power-Law Hessian Spectrums in Deep Learningcited by 16
  16. 2023Order Matters in the Presence of Dataset Imbalance for Multilingual Learningcited by 7
  17. 2023Implicit biases in multitask and continual learning from a backward error analysis perspectivecited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 7.

9 papers.

  1. 2026The Geometry of Sequential Learning: Lie-Bracket Prediction of Transfer Ordercited by 9
  2. 2026Optimizer Memory Makes Shuffle Order a First-Order Source of Fine-Tuning Noisecited by 7
  3. 2025Data Mixing Agent: Learning to Re-weight Domains for Continual Pre-trainingcited by 5
  4. 2026First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformerscited by 4
  5. 2026Order Matters: Improving Domain Adaptation by Reordering Datacited by 2
  6. 2026Fiber Fingerprints of Hidden Learning-State Dynamicscited by 2
  7. 2025Gamayun's Path to Multilingual Mastery: Cost-Efficient Training of a 1.5B-Parameter LLMcited by 0
  8. 2026Dimensionless Controls of Plasticity Under Alternating Tasks: From Evolutionary Biology to Continual Learningcited by 0
  9. 2026Commutator Memory: Sparse, Path-Local Reading and Steering in Language Modelscited by 0