mlsanityData to 14 Oct 2026

FAMO: Fast Adaptive Multitask Optimization

arXiv 2306.03792 ↗First posted 6 Jun 2023cs.LGcited by 78

Not on the map. Why?

Abstract (arXiv)

One of the grand enduring goals of AI is to create generalist agents that can learn multiple different tasks from diverse data via multitask learning (MTL). However, in practice, applying gradient descent (GD) on the average loss across all tasks may yield poor multitask performance due to severe under-optimization of certain tasks. Previous approaches that manipulate task gradients for a more balanced loss decrease require storing and computing all task gradients (O(k) space and time where k is the number of tasks), limiting their use in large-scale scenarios. In this work, we introduce Fast Adaptive Multitask Optimization FAMO, a dynamic weighting method that decreases task losses in a balanced way using O(1) space and time. We conduct an extensive set of experiments covering multi-task supervised and reinforcement learning problems. Our results indicate that FAMO achieves comparable or superior performance to state-of-the-art gradient manipulation techniques while offering significant improvements in space and computational efficiency. Code is available at urlhttps://github.com/Cranial-XIX/FAMO.

Key facts. Cited by 78 papers in the map (data through 14 Oct 2026). Builds on 33 of its 48 references that are in the map. First posted on arXiv 6 Jun 2023.

Builds on

33 of 48 references.

References · 33

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2023Segment Anythingcited by 8,234
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  5. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  7. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,434
  8. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
  9. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  10. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  11. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
  12. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
21 more
  1. 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
  2. 2018End-to-End Multi-Task Learning with Attentioncited by 534
  3. 2016Cross-stitch Networks for Multi-task Learningcited by 467
  4. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 459
  5. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 375
  6. 2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 222
  7. 2022Multi-Task Learning as a Bargaining Gamecited by 215
  8. 2020Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropoutcited by 201
  9. 2016UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memorycited by 200
  10. 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 174
  11. 2020Multi-Task Reinforcement Learning with Soft Modularizationcited by 143
  12. 2020Learning to Branch for Multi-Task Learningcited by 115
  13. 2022Auto-Lambda: Disentangling Dynamic Task Relationshipscited by 73
  14. 2022In Defense of the Unitary Scalarization for Deep Multi-Task Learningcited by 67
  15. 2022Do Current Multi-Task Optimization Methods in Deep Learning Even Help?cited by 60
  16. 2020Automated Search for Resource-Efficient Branched Multi-Task Networkscited by 51
  17. 2020MTL-NAS: Task-Agnostic Neural Architecture Search towards General-Purpose Multi-Task Learningcited by 50
  18. 2019Stochastic Filter Groups for Multi-Task CNNs: Learning Specialist and Generalist Convolution Kernelscited by 42
  19. 2021Variational Multi-Task Learning with Gumbel-Softmax Priorscited by 15
  20. 2020Follow the bisector: a simple method for multi-objective optimizationcited by 4
  21. 2021Improved optimization strategies for deep Multi-Task Networkscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 13; 2025: 32; 2026: 32.

78 papers.

Sort:

By area:

  1. 2024Fair Resource Allocation in Multi-Task Learningcited by 35
  2. 2025Imbalanced Gradients in RL Post-Training of Multi-Task LLMscited by 20
  3. 2025Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyondcited by 18
  4. 2024MUNBa: Machine Unlearning via Nash Bargainingcited by 17
  5. 2025Selective Task Group Updates for Multi-Task Optimizationcited by 17
  6. 2024Few for Many: Tchebycheff Set Scalarization for Many-Objective Optimizationcited by 16
  7. 2024Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferencescited by 13
  8. 2024Proactive Gradient Conflict Mitigation in Multi-Task Learning: A Sparse Training Perspectivecited by 12
  9. 2024GO4Align: Group Optimization for Multi-Task Alignmentcited by 11
  10. 2024FERERO: A Flexible Framework for Preference-Guided Multi-Objective Learningcited by 9
  11. 2025PiKE: Adaptive Data Mixing for Large-Scale Multi-Task Learning Under Low Gradient Conflictscited by 8
  12. 2024MGDA Converges under Generalized Smoothness, Provablycited by 7
38 more
  1. 2025LDC-MTL: Balancing Multi-Task Learning through Scalable Loss Discrepancy Controlcited by 7
  2. 2024Analytical Uncertainty-Based Loss Weighting in Multi-Task Learningcited by 6
  3. 2024Multi-objective Deep Learning: Taxonomy and Survey of the State of the Artcited by 6
  4. 2025Balanced Rate-Distortion Optimization in Learned Image Compressioncited by 5
  5. 2025Gradient Similarity Surgery in Multi-Task Deep Learningcited by 5
  6. 2025Interaction-Merged Motion Planning: Effectively Leveraging Diverse Motion Datasets for Robust Planningcited by 5
  7. 2025Synchronizing Task Behavior: Aligning Multiple Tasks during Test-Time Trainingcited by 5
  8. 2025Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Modelscited by 5
  9. 2025Efficient Utility-Preserving Machine Unlearning with Implicit Gradient Surgerycited by 5
  10. 2025Resolving Token-Space Gradient Conflicts: Token Space Manipulation for Transformer-Based Multi-Task Learningcited by 4
  11. 2026Label-Free Cross-Task LoRA Merging with Null-Space Compressioncited by 4
  12. 2025Continual Optimization with Symmetry Teleportation for Multi-Task Learningcited by 3
  13. 2025COPA: Comparing the incomparable in multi-objective model evaluationcited by 3
  14. 2025Revisit the Imbalance Optimization in Multi-task Learning: An Experimental Analysiscited by 3
  15. 2026mSFT: Addressing Dataset Mixtures Overfitting Heterogeneously in Multi-task SFTcited by 3
  16. 2023Examining Common Paradigms in Multi-Task Learningcited by 2
  17. 2024A Parameter Update Balancing Algorithm for Multi-task Ranking Models in Recommendation Systemscited by 2
  18. 2025Injecting Imbalance Sensitivity for Multi-Task Learningcited by 2
  19. 2025Rep-MTL: Unleashing the Power of Representation-level Task Saliency for Multi-Task Learningcited by 2
  20. 2025AutoScale: Linear Scalarization Guided by Multi-Task Optimization Metricscited by 2
  21. 2025AnalysisGNN: Unified Music Analysis with Graph Neural Networkscited by 2
  22. 2025GCond: Gradient Conflict Resolution via Accumulation-based Stabilization for Large-Scale Multi-Task Learningcited by 2
  23. 2025NTKMTL: Mitigating Task Imbalance in Multi-Task Learning from Neural Tangent Kernel Perspectivecited by 2
  24. 2025Language-Conditioned Representations and Mixture-of-Experts Policy for Robust Multi-Task Robotic Manipulationcited by 2
  25. 2026Leveraging Second-Order Curvature for Efficient Learned Image Compression: Theory and Empirical Evidencecited by 2
  26. 2026Uncovering Cross-Objective Interference in Multi-Objective Alignmentcited by 2
  27. 2026Delve into the Applicability of Advanced Optimizers for Multi-Task Learningcited by 2
  28. 2024Robust Analysis of Multi-Task Learning Efficiency: New Benchmarks on Light-Weighed Backbones and Effective Measurement of Multi-Task Learning Challenges by Feature Disentanglementcited by 1
  29. 2024Goal-Conditioned Supervised Learning for Multi-Objective Recommendationcited by 1
  30. 2025SAMO: A Lightweight Sharpness-Aware Approach for Multi-Task Optimization with Joint Global-Local Perturbationcited by 1
  31. 2025Graph Coloring for Multi-Task Learningcited by 1
  32. 2025AIM: Adaptive Intervention for Deep Multi-task Learning of Molecular Propertiescited by 1
  33. 2025Simplifying Multi-Task Architectures Through Task-Specific Normalizationcited by 1
  34. 2026EraseAnything++: Enabling Concept Erasure in Rectified Flow Transformers Leveraging Multi-Object Optimizationcited by 1
  35. 2026Uni-LVC: A Unified Method for Intra- and Inter-Mode Learned Video Compressioncited by 1
  36. 2026One Model for All: Multi-Objective Controllable Language Modelscited by 1
  37. 2026Per-Loss Adapters for Gradient Conflict in Physics-Informed Neural Networkscited by 1
  38. 2026Not All Objectives Are Born Equal: Priority-Constrained Descent for Hierarchical Multi-Objective Optimizationcited by 1

50 most cited of 78 All 78 (JSON)