mlsanityData to 7 Oct 2026

Efficient Pareto Manifold Learning with Low-Rank Structure

arXiv 2407.20734 ↗First posted 30 Jul 2024cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Multi-task learning, which optimizes performance across multiple tasks, is inherently a multi-objective optimization problem. Various algorithms are developed to provide discrete trade-off solutions on the Pareto front. Recently, continuous Pareto front approximations using a linear combination of base networks have emerged as a compelling strategy. However, it suffers from scalability issues when the number of tasks is large. To address this issue, we propose a novel approach that integrates a main network with several low-rank matrices to efficiently learn the Pareto manifold. It significantly reduces the number of parameters and facilitates the extraction of shared features. We also introduce orthogonal regularization to further bolster performance. Extensive experimental results demonstrate that the proposed approach outperforms state-of-the-art baselines, especially on datasets with a large number of tasks.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 32 of its 55 references that are in the map. First posted on arXiv 30 Jul 2024.

Builds on

32 of 55 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  4. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  5. 2021Convex optimizationcited by 2,479
  6. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  7. 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,483
  8. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  9. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  10. 2022Editing Models with Task Arithmeticcited by 1,076
  11. 2016HyperNetworkscited by 1,003
  12. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
20 more
  1. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  2. 2017Dynamic Routing Between Capsulescited by 689
  3. 2018End-to-End Multi-Task Learning with Attentioncited by 533
  4. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 441
  5. 2017Age Progression/Regression by Conditional Adversarial Autoencodercited by 426
  6. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
  7. 2019Pareto Multi-Task Learningcited by 224
  8. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  9. 2022Multi-Task Learning as a Bargaining Gamecited by 209
  10. 2020Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropoutcited by 199
  11. 2017Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learningcited by 125
  12. 2019Orthogonal Convolutional Neural Networkscited by 122
  13. 2020Learning the Pareto Front with Hypernetworkscited by 122
  14. 2022Auto-Lambda: Disentangling Dynamic Task Relationshipscited by 73
  15. 2017All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulationcited by 58
  16. 2020Efficient Continuous Pareto Exploration in Multi-Task Learningcited by 57
  17. 2021Scalable Pareto Front Approximation for Deep Multi-Objective Learningcited by 45
  18. 2022Pareto Manifold Learning: Tackling multiple tasks via ensembles of single-task modelscited by 29
  19. 2022Improving Pareto Front Learning via Multi-Sample Hypernetworkscited by 16
  20. 2021Pareto Navigation Gradient Descent: a First-Order Algorithm for Optimization in Pareto Setcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 2.

9 papers.

  1. 2025PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Modelcited by 27
  2. 2025Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasetscited by 17
  3. 2025Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyondcited by 15
  4. 2024LibMOON: A Gradient-based MultiObjective OptimizatioN Library in PyTorchcited by 10
  5. 2025Efficient Knowledge Transfer in Multi-Task Learning through Task-Adaptive Low-Rank Representationcited by 4
  6. 2026UniARM: Towards a Unified Autoregressive Reward Model for Multi-Objective Test-Time Alignmentcited by 3
  7. 2025Learning Pareto manifolds in high dimensions: How can regularization help?cited by 1
  8. 2025Towards Relaxed Multimodal Inputs for Gait-based Parkinson's Disease Assessmentcited by 1
  9. 2026Multi-Objective Exploration and Preference Optimization via Mutual Informationcited by 0