3D-Aware Multi-Task Learning with Cross-View Correlations for Dense Scene Understanding
Not on the map. Why?
Abstract (arXiv)
This paper addresses the challenge of training a single network to jointly perform multiple dense prediction tasks, such as segmentation and depth estimation, i.e., multi-task learning (MTL). Current approaches mainly capture cross-task relations in the 2D image space, often leading to unstructured features lacking 3D-awareness. We argue that 3D-awareness is vital for modeling cross-task correlations essential for comprehensive scene understanding. We propose to address this problem by integrating correlations across views, i.e., cost volume, as geometric consistency in the MTL network. Specifically, we introduce a lightweight Cross-view Module (CvM), shared across tasks, to exchange information across views and capture cross-view correlations, integrated with a feature from MTL encoder for multi-task predictions. This module is architecture-agnostic and can be applied to both single and multi-view data. Extensive results on NYUv2 and PASCAL-Context demonstrate that our method effectively injects geometric consistency into existing MTL methods to improve performance.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 64 of its 82 references that are in the map. First posted on arXiv 25 Nov 2025.
Builds on
64 of 82 references.
References · 64
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
- 2020GLU Variants Improve Transformercited by 1,804
- 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,584
- 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,567
- 2025VGGT: Visual Geometry Grounded Transformercited by 1,539
- 2025DINOv3cited by 1,414
- 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
- 2020Gradient Surgery for Multi-Task Learningcited by 1,256
- 2024Grounding Image Matching in 3D with MASt3Rcited by 983
38 more
- 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
- 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
- 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
- 2018MVSNet: Depth Inference for Unstructured Multi-view Stereocited by 788
- 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
- 2023pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstructioncited by 684
- 2024MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Imagescited by 566
- 2018End-to-End Multi-Task Learning with Attentioncited by 534
- 2014Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecturecited by 483
- 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
- 2016Cross-stitch Networks for Multi-task Learningcited by 467
- 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 459
- 2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 415
- 2025Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Passcited by 367
- 2021GMFlow: Learning Optical Flow via Global Matchingcited by 327
- 2024DepthSplat: Connecting Gaussian Splatting and Depthcited by 286
- 2022Unifying Flow, Stereo and Depth Estimationcited by 275
- 2022Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representationcited by 243
- 2014Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Partscited by 232
- 2018PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsingcited by 224
- 2019Pareto Multi-Task Learningcited by 224
- 2020Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropoutcited by 201
- 2016UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memorycited by 200
- 2025FLARE: Feed-forward Geometry, Appearance and Camera Estimation from Uncalibrated Sparse Viewscited by 180
- 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 169
- 2020MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learningcited by 157
- 2019Pattern-Affinitive Propagation across Depth, Surface Normal and Semantic Segmentationcited by 142
- 2020Learning to Branch for Multi-Task Learningcited by 115
- 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 98
- 2022InvPT: Inverted Pyramid Multi-task Transformer for Dense Scene Understandingcited by 89
- 2024RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Modelscited by 89
- 2019Branched Multi-Task Networks: Deciding What Layers To Sharecited by 76
- 2022Pix2NeRF: Unsupervised Conditional π-GAN for Single Image to Neural Radiance Fields Translationcited by 73
- 2021Exploring Relational Context for Multi-Task Dense Predictioncited by 71
- 2024Multi-Task Dense Prediction via Mixture of Low-Rank Expertscited by 58
- 2020Knowledge Distillation for Multi-task Learningcited by 51
- 2020Automated Search for Resource-Efficient Branched Multi-Task Networkscited by 51
- 2019Many Task Learning with Task Routingcited by 50
50 most cited of 64 All 64 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.