mlsanityData to 14 Oct 2026

Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics

arXiv 1705.07115 ↗First posted 19 May 2017cs.CVcited by 1,358

Not on the map. Why?

Cited by 32 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Numerous deep learning applications benefit from multi-task learning with multiple regression and classification objectives. In this paper we make the observation that the performance of such systems is strongly dependent on the relative weighting between each task's loss. Tuning these weights by hand is a difficult and expensive process, making multi-task learning prohibitive in practice. We propose a principled approach to multi-task deep learning which weighs multiple loss functions by considering the homoscedastic uncertainty of each task. This allows us to simultaneously learn various quantities with different units or scales in both classification and regression settings. We demonstrate our model learning per-pixel depth regression, semantic and instance segmentation from a monocular input image. Perhaps surprisingly, we show our model can learn multi-task weightings and outperform separate models trained individually on each task.

Key facts. Cited by 1,358 papers in the map (data through 14 Oct 2026). First posted on arXiv 19 May 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 222; 2022: 195; 2023: 213; 2024: 208; 2025: 257; 2026: 262.

1,358 papers.

Sort:

By area:

  1. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 459
  2. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 402
  3. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 368
  4. 2023Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspectivecited by 280
  5. 2025MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Detailscited by 261
  6. 2023Efficient Diffusion Training via Min-SNR Weighting Strategycited by 246
  7. 2021FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Camerascited by 239
  8. 2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 222
  9. 2022Multi-Task Learning as a Bargaining Gamecited by 215
  10. 2021Multimodal Motion Prediction with Stacked Transformerscited by 208
  11. 2022AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Timecited by 200
  12. 2021Probabilistic and Geometric Depth: Detecting Objects in Perspectivecited by 191
38 more
  1. 2021Objects are Different: Flexible Monocular 3D Object Detectioncited by 185
  2. 2023FengWu: Pushing the Skillful Global Medium-range Weather Forecast beyond 10 Days Leadcited by 175
  3. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 169
  4. 2022CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flowcited by 166
  5. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 160
  6. 2021Geometry Uncertainty Projection Network for Monocular 3D Object Detectioncited by 156
  7. 2021Deep Learning Through the Lens of Example Difficultycited by 141
  8. 2021Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videoscited by 141
  9. 2023Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentationcited by 129
  10. 2022Simple-BEV: What Really Matters for Multi-Sensor BEV Perception?cited by 128
  11. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 126
  12. 2023SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modelingcited by 115
  13. 2021Uncertainty-aware Joint Salient Object and Camouflaged Object Detectioncited by 114
  14. 2023Gloss-free Sign Language Translation: Improving from Visual-Language Pretrainingcited by 105
  15. 2021Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learningcited by 101
  16. 2021SiT: Self-supervised vIsion Transformercited by 99
  17. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 98
  18. 2024Representation Surgery for Multi-Task Model Mergingcited by 96
  19. 2021Trusted Multi-View Classificationcited by 94
  20. 2023Deep Model Fusion: A Surveycited by 91
  21. 2022InvPT: Inverted Pyramid Multi-task Transformer for Dense Scene Understandingcited by 89
  22. 2021Multi-Objective Loss Balancing for Physics-Informed Deep Learningcited by 87
  23. 2019Urban Driving with Conditional Imitation Learningcited by 86
  24. 2021Multi-Task Self-Training for Learning General Representationscited by 85
  25. 2021Geometry-based Distance Decomposition for Monocular 3D Object Detectioncited by 84
  26. 2021RotoGrad: Gradient Homogenization in Multitask Learningcited by 83
  27. 2021One Model to Serve All: Star Topology Adaptive Recommender for Multi-Domain CTR Predictioncited by 78
  28. 2023FAMO: Fast Adaptive Multitask Optimizationcited by 78
  29. 2024MMEarth: Exploring Multi-Modal Pretext Tasks For Geospatial Representation Learningcited by 77
  30. 2022LidarMultiNet: Towards a Unified Multi-Task Network for LiDAR Perceptioncited by 74
  31. 2021STEP: Segmenting and Tracking Every Pixelcited by 73
  32. 2022Auto-Lambda: Disentangling Dynamic Task Relationshipscited by 73
  33. 2021Replay in Deep Learning: Current Approaches and Missing Biological Elementscited by 72
  34. 2021Exploring Relational Context for Multi-Task Dense Predictioncited by 71
  35. 2021Controlled Text Generation as Continuous Optimization with Multiple Constraintscited by 70
  36. 2023Independent Component Alignment for Multi-Task Learningcited by 70
  37. 2021Self-Supervised Monocular Depth Estimation with Internal Feature Fusioncited by 67
  38. 2022In Defense of the Unitary Scalarization for Deep Multi-Task Learningcited by 67

50 most cited of 1,358 All 1,358 (JSON)