mlsanityData to 14 Oct 2026

Multi-task Learning with 3D-Aware Regularization

arXiv 2310.00986 ↗First posted 2 Oct 2023cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

Deep neural networks have become a standard building block for designing models that can perform multiple dense computer vision tasks such as depth estimation and semantic segmentation thanks to their ability to capture complex correlations in high dimensional feature space across tasks. However, the cross-task correlations that are learned in the unstructured feature space can be extremely noisy and susceptible to overfitting, consequently hurting performance. We propose to address this problem by introducing a structured 3D-aware regularizer which interfaces multiple tasks through the projection of features extracted from an image encoder to a shared 3D feature space and decodes them into their task output space through differentiable rendering. We show that the proposed method is architecture agnostic and can be plugged into various prior multi-task backbones to improve their performance; as we evidence using standard benchmarks NYUv2 and PASCAL-Context.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 55 of its 74 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

55 of 74 references.

References · 55

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,584
  3. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,420
  4. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
  5. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,290
  6. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  7. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,205
  8. 2017A Survey on Multi-Task Learningcited by 840
  9. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
  10. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
  11. 2019PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitizationcited by 818
  12. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
38 more
  1. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  2. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 700
  3. 2018End-to-End Multi-Task Learning with Attentioncited by 534
  4. 2014Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecturecited by 483
  5. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  6. 2016Cross-stitch Networks for Multi-task Learningcited by 467
  7. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 459
  8. 2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 415
  9. 2018DeepVoxels: Learning Persistent 3D Feature Embeddingscited by 405
  10. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Modelscited by 274
  11. 2022Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representationcited by 243
  12. 2014Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Partscited by 232
  13. 2018PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsingcited by 224
  14. 2019Pareto Multi-Task Learningcited by 224
  15. 2020Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropoutcited by 201
  16. 2016UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memorycited by 200
  17. 2022RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generationcited by 162
  18. 2020MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learningcited by 157
  19. 2019Attentive Single-Tasking of Multiple Taskscited by 148
  20. 2019Pattern-Affinitive Propagation across Depth, Surface Normal and Semantic Segmentationcited by 142
  21. 2023Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fieldscited by 123
  22. 2019Neural Rerendering in the Wildcited by 121
  23. 2020Learning to Branch for Multi-Task Learningcited by 115
  24. 2022EpiGRAF: Rethinking training of 3D GANscited by 108
  25. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 98
  26. 2023NDC-Scene: Boost Monocular 3D Semantic Scene Completion in Normalized Device Coordinates Spacecited by 90
  27. 2022InvPT: Inverted Pyramid Multi-task Transformer for Dense Scene Understandingcited by 89
  28. 2019Branched Multi-Task Networks: Deciding What Layers To Sharecited by 76
  29. 2022Pix2NeRF: Unsupervised Conditional π-GAN for Single Image to Neural Radiance Fields Translationcited by 73
  30. 2021Exploring Relational Context for Multi-Task Dense Predictioncited by 71
  31. 2020Knowledge Distillation for Multi-task Learningcited by 51
  32. 2020Automated Search for Resource-Efficient Branched Multi-Task Networkscited by 51
  33. 2019Many Task Learning with Task Routingcited by 50
  34. 2019MultiNet++: Multi-Stream Feature Aggregation and Geometric Loss Strategy for Multi-Task Learningcited by 45
  35. 2023TaskExpert: Dynamically Assembling Multi-Task Representations with Memorial Mixture-of-Expertscited by 45
  36. 2019Stochastic Filter Groups for Multi-Task CNNs: Learning Specialist and Generalist Convolution Kernelscited by 42
  37. 2021Learning Multiple Dense Prediction Tasks from Partially Annotated Datacited by 42
  38. 2019Regularizing Deep Multi-Task Networks using Orthogonal Gradientscited by 36

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 2.

8 papers.

  1. 2024DepthCues: Evaluating Monocular Depth Perception in Large Vision Modelscited by 19
  2. 2024Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learningcited by 7
  3. 2026MTPano: Multi-Task Panoramic Scene Understanding via Label-Free Integration of Dense Prediction Priorscited by 5
  4. 2025Revisit the Imbalance Optimization in Multi-task Learning: An Experimental Analysiscited by 3
  5. 20253D-Aware Multi-Task Learning with Cross-View Correlations for Dense Scene Understandingcited by 2
  6. 2025Enhancing Mamba Decoder with Bidirectional Interaction in Multi-Task Dense Predictioncited by 1
  7. 2025MT-Depth: Multi-task Instance feature analysis for the Depth Completioncited by 0
  8. 2026Learning to Orchestrate Vision Foundation Models for Multi-Task Dense Predictioncited by 0