mlsanityData to 14 Oct 2026

LDC-MTL: Balancing Multi-Task Learning through Scalable Loss Discrepancy Control

arXiv 2502.08585 ↗First posted 12 Feb 2025cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Multi-task learning (MTL) has been widely adopted for its ability to simultaneously learn multiple tasks. While existing gradient manipulation methods often yield more balanced solutions than simple scalarization-based approaches, they typically incur a significant computational overhead of O(K) in both time and memory, where K is the number of tasks. In this paper, we propose LDC-MTL, a simple and scalable loss discrepancy control approach for MTL, formulated from a bilevel optimization perspective. Our method incorporates two key components: (i) a bilevel formulation for fine-grained loss discrepancy control, and (ii) a scalable first-order bilevel algorithm that requires only O(1) time and memory. Theoretically, we prove that LDC-MTL guarantees convergence not only to a stationary point of the bilevel problem with loss discrepancy control but also to an ε-accurate Pareto stationary point for all K loss functions under mild conditions. Extensive experiments on diverse multi-task datasets demonstrate the superior performance of LDC-MTL in both accuracy and efficiency.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 38 of its 56 references that are in the map. First posted on arXiv 12 Feb 2025.

Builds on

38 of 56 references.

References · 38

  1. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  2. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  3. 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,487
  4. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
  5. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  6. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
  7. 2018End-to-End Multi-Task Learning with Attentioncited by 534
  8. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 459
  9. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 360
  10. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  11. 2018Approximation Methods for Bilevel Programmingcited by 236
  12. 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 234
26 more
  1. 2020Bilevel Optimization: Convergence Analysis and Enhanced Designcited by 222
  2. 2022Multi-Task Learning as a Bargaining Gamecited by 215
  3. 2020Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropoutcited by 201
  4. 2016UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memorycited by 200
  5. 2019Attentive Single-Tasking of Multiple Taskscited by 148
  6. 2021Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learningcited by 101
  7. 2023A Fully First-Order Method for Stochastic Bilevel Optimizationcited by 100
  8. 2020FairBatch: Batch Selection for Model Fairnesscited by 99
  9. 2023On Penalty-based Bilevel Gradient Descent Methodcited by 87
  10. 2023FAMO: Fast Adaptive Multitask Optimizationcited by 78
  11. 2022In Defense of the Unitary Scalarization for Deep Multi-Task Learningcited by 67
  12. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  13. 2022Do Current Multi-Task Optimization Methods in Deep Learning Even Help?cited by 60
  14. 2022A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methodscited by 42
  15. 2023Three-Way Trade-Off in Multi-Objective Learning: Optimization, Generalization and Conflict-Avoidancecited by 36
  16. 2024Fair Resource Allocation in Multi-Task Learningcited by 35
  17. 2023Direction-oriented Multi-objective Learning: Simple and Provable Stochastic Algorithmscited by 34
  18. 2020M2GRL: A Multi-task Multi-view Graph Representation Learning Framework for Web-scale Recommender Systemscited by 21
  19. 2021Proxy Convexity: A Unified Framework for the Analysis of Neural Networks Trained by Gradient Descentcited by 21
  20. 2023SimFBO: Towards Simple, Flexible and Communication-efficient Federated Bilevel Learningcited by 14
  21. 2023Communication-Efficient Federated Hypergradient Computation via Aggregated Iterative Differentiationcited by 13
  22. 2024Bayesian Uncertainty for Gradient Aggregation in Multi-Task Learningcited by 11
  23. 2024GO4Align: Group Optimization for Multi-Task Alignmentcited by 11
  24. 2023Improvable Gap Balancing for Multi-Task Learningcited by 9
  25. 2024Tuning-Free Bilevel Optimization: New Algorithms and Convergence Analysiscited by 5
  26. 2024Meta-Learning with Heterogeneous Taskscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 4.

7 papers.

  1. 2025Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAscited by 3
  2. 2025Bilevel Data Curation for LLM Fine-tuning: Offline Selection and Online Self-Refining Generationcited by 2
  3. 2026Delve into the Applicability of Advanced Optimizers for Multi-Task Learningcited by 2
  4. 2025Monotonic Transformation Invariant Multi-task Learningcited by 0
  5. 2026Focus on What Matters: Two-Stage ROI-Aware Refinement for Anatomy-Preserving Fetal Ultrasound Reconstructioncited by 0
  6. 2026When Losses Align: Gradient-Based Composite Loss Weighting for Efficient Pretrainingcited by 0
  7. 2026BRIDGE: Bilevel Retrieval-Credit-Aware Agentic Reinforcement Learningcited by 0