# MultiNet++: Multi-Stream Feature Aggregation and Geometric Loss Strategy for Multi-Task Learning

arXiv: https://arxiv.org/abs/1904.08492 · HTML: https://mlsanity.com/paper/1904.08492/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Apr 2019.

## Built on by (most cited first, 30 of 45)

- [Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline](https://mlsanity.com/paper/2206.08129/) (2022)
- [Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentation](https://mlsanity.com/paper/2308.02097/) (2023)
- [OmniDet: Surround View Cameras based Multi-task Visual Perception Network for Autonomous Driving](https://mlsanity.com/paper/2102.07448/) (2021)
- [Deep Gait Recognition: A Survey](https://mlsanity.com/paper/2102.09546/) (2021)
- [Smooth Tchebycheff Scalarization for Multi-Objective Optimization](https://mlsanity.com/paper/2402.19078/) (2024)
- [A Deep-Learning Approach For Direct Whole-Heart Mesh Reconstruction](https://mlsanity.com/paper/2102.07899/) (2021)
- [Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimization](https://mlsanity.com/paper/2206.04496/) (2022)
- [Dual-Balancing for Multi-Task Learning](https://mlsanity.com/paper/2308.12029/) (2023)
- [LibMTL: A Python Library for Multi-Task Learning](https://mlsanity.com/paper/2203.14338/) (2022)
- [Universal Representations: A Unified Look at Multiple Task and Domain Learning](https://mlsanity.com/paper/2204.02744/) (2022)
- [An End-to-End Robust Point Cloud Semantic Segmentation Network with Single-Step Conditional Diffusion Models](https://mlsanity.com/paper/2411.16308/) (2024)
- [Weather and Light Level Classification for Autonomous Driving: Dataset, Baseline and Active Learning](https://mlsanity.com/paper/2104.14042/) (2021)
- [AutoMTL: A Programming Framework for Automating Efficient Multi-Task Learning](https://mlsanity.com/paper/2110.13076/) (2021)
- [Adversarial Attacks on Multi-task Visual Perception for Autonomous Driving](https://mlsanity.com/paper/2107.07449/) (2021)
- [Channel Exchanging Networks for Multimodal and Multitask Dense Image Prediction](https://mlsanity.com/paper/2112.02252/) (2021)
- [Bayesian Uncertainty for Gradient Aggregation in Multi-Task Learning](https://mlsanity.com/paper/2402.04005/) (2024)
- [DPF-Nutrition: Food Nutrition Estimation via Depth Prediction and Fusion](https://mlsanity.com/paper/2310.11702/) (2023)
- [TurboTrain: Towards Efficient and Balanced Multi-Task Learning for Multi-Agent Perception and Prediction](https://mlsanity.com/paper/2508.04682/) (2025)
- [Multi-task Learning with 3D-Aware Regularization](https://mlsanity.com/paper/2310.00986/) (2023)
- [UniNet: A Unified Scene Understanding Network and Exploring Multi-Task Relationships through the Lens of Adversarial Attacks](https://mlsanity.com/paper/2108.04584/) (2021)
- [Sample-Level Weighting for Multi-Task Learning with Auxiliary Tasks](https://mlsanity.com/paper/2306.04519/) (2023)
- [When Multi-Task Learning Meets Partial Supervision: A Computer Vision Review](https://mlsanity.com/paper/2307.14382/) (2023)
- [Analytical Uncertainty-Based Loss Weighting in Multi-Task Learning](https://mlsanity.com/paper/2408.07985/) (2024)
- [Instance-Level Task Parameters: A Robust Multi-task Weighting Framework](https://mlsanity.com/paper/2106.06129/) (2021)
- [Improved optimization strategies for deep Multi-Task Networks](https://mlsanity.com/paper/2109.11678/) (2021)
- [radarODE-MTL: A Multi-Task Learning Framework with Eccentric Gradient Alignment for Robust Radar-Based ECG Reconstruction](https://mlsanity.com/paper/2410.08656/) (2024)
- [OFVL-MS: Once for Visual Localization across Multiple Indoor Scenes](https://mlsanity.com/paper/2308.11928/) (2023)
- [Aligned Multi Objective Optimization](https://mlsanity.com/paper/2502.14096/) (2025)
- [Examining Common Paradigms in Multi-Task Learning](https://mlsanity.com/paper/2311.04698/) (2023)
- [CerberusDet: Unified Multi-Dataset Object Detection](https://mlsanity.com/paper/2407.12632/) (2024)

## Abstract (arXiv)

Multi-task learning is commonly used in autonomous driving for solving various visual perception tasks. It offers significant benefits in terms of both performance and computational complexity. Current work on multi-task learning networks focus on processing a single input image and there is no known implementation of multi-task learning handling a sequence of images. In this work, we propose a multi-stream multi-task network to take advantage of using feature representations from preceding frames in a video sequence for joint learning of segmentation, depth, and motion. The weights of the current and previous encoder are shared so that features computed in the previous frame can be leveraged without additional computation. In addition, we propose to use the geometric mean of task losses as a better alternative to the weighted average of task losses. The proposed loss function facilitates better handling of the difference in convergence rates of different tasks. Experimental results on KITTI, Cityscapes and SYNTHIA datasets demonstrate that the proposed strategies outperform various existing multi-task learning solutions.