# TurboTrain: Towards Efficient and Balanced Multi-Task Learning for Multi-Agent Perception and Prediction

arXiv: https://arxiv.org/abs/2508.04682 · HTML: https://mlsanity.com/paper/2508.04682/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 67 of its 87 references that are in the map.
- First posted on arXiv 6 Aug 2025.

## Builds on (most cited first, 30 of 67)

- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [VAD: Vectorized Scene Representation for Efficient Autonomous Driving](https://mlsanity.com/paper/2303.12077/) (2023)
- [RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds](https://mlsanity.com/paper/1911.11236/) (2019)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [Conflict-Averse Gradient Descent for Multi-task Learning](https://mlsanity.com/paper/2110.14048/) (2021)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)
- [V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer](https://mlsanity.com/paper/2203.10638/) (2022)
- [OPV2V: An Open Benchmark Dataset and Fusion Pipeline for Perception with Vehicle-to-Vehicle Communication](https://mlsanity.com/paper/2109.07644/) (2021)
- [V2VNet: Vehicle-to-Vehicle Communication for Joint Perception and Prediction](https://mlsanity.com/paper/2008.07519/) (2020)
- [Where2comm: Communication-Efficient Collaborative Perception via Spatial Confidence Maps](https://mlsanity.com/paper/2209.12836/) (2022)
- [SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representation](https://mlsanity.com/paper/2405.19620/) (2024)
- [PointASNL: Robust Point Clouds Processing using Nonlocal Neural Networks with Adaptive Sampling](https://mlsanity.com/paper/2003.00492/) (2020)
- [Learning Distilled Collaboration Graph for Multi-Agent Perception](https://mlsanity.com/paper/2111.00643/) (2021)
- [BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Driving](https://mlsanity.com/paper/2205.09743/) (2022)
- [F-Cooper: Feature based Cooperative Perception for Autonomous Vehicle Edge Computing System Using 3D Point Clouds](https://mlsanity.com/paper/1909.06459/) (2019)
- [V2X-Sim: Multi-Agent Collaborative Perception Dataset and Benchmark for Autonomous Driving](https://mlsanity.com/paper/2202.08449/) (2022)
- [ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queries](https://mlsanity.com/paper/2208.01582/) (2022)
- [V2X-Seq: A Large-Scale Sequential Dataset for Vehicle-Infrastructure Cooperative Perception and Forecasting](https://mlsanity.com/paper/2305.05938/) (2023)
- [Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learning](https://mlsanity.com/paper/2111.10603/) (2021)
- [An Extensible Framework for Open Heterogeneous Collaborative Perception](https://mlsanity.com/paper/2401.13964/) (2024)
- [LiDAR-based Panoptic Segmentation via Dynamic Shifting Network](https://mlsanity.com/paper/2011.11964/) (2020)
- [UniPAD: A Universal Pre-training Paradigm for Autonomous Driving](https://mlsanity.com/paper/2310.08370/) (2023)
- [TUMTraf V2X Cooperative Perception Dataset](https://mlsanity.com/paper/2403.01316/) (2024)
- [V2X-Real: a Large-Scale Dataset for Vehicle-to-Everything Cooperative Perception](https://mlsanity.com/paper/2403.16034/) (2024)
- [VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding](https://mlsanity.com/paper/2312.08733/) (2023)
- [Random Hypervolume Scalarizations for Provable Multi-Objective Black Box Optimization](https://mlsanity.com/paper/2006.04655/) (2020)
- [Occupancy-MAE: Self-supervised Pre-training Large-scale LiDAR Point Clouds with Masked Occupancy Autoencoders](https://mlsanity.com/paper/2206.09900/) (2022)

## Built on by (most cited first, 9 of 9)

- [SpanVLA: Efficient Action Bridging and Learning from Negative-Recovery Samples for Vision-Language-Action Model](https://mlsanity.com/paper/2604.19710/) (2026)
- [QuantV2X: A Fully Quantized Multi-Agent System for Cooperative Perception](https://mlsanity.com/paper/2509.03704/) (2025)
- [MDrive: Benchmarking Closed-Loop Cooperative Driving for End-to-End Multi-agent Systems](https://mlsanity.com/paper/2605.10904/) (2026)
- [Forging Spatial Intelligence: A Roadmap of Multi-Modal Data Pre-Training for Autonomous Systems](https://mlsanity.com/paper/2512.24385/) (2025)
- [SafeCoop: Unravelling Full Stack Safety in Agentic Collaborative Driving](https://mlsanity.com/paper/2510.18123/) (2025)
- [CMU-Drive and V2V-VLA: Cooperative Multi-agent Unified Driving with Reasoning Benchmark and Vehicle-to-Vehicle Vision-Language-Action Models](https://mlsanity.com/paper/2608.07621/) (2026)
- [When the City Teaches the Car: Label-Free 3D Perception from Infrastructure](https://mlsanity.com/paper/2603.16742/) (2026)
- [ConFixGS: Learning to Fix Feedforward 3D Gaussian Splatting with Confidence-Aware Diffusion Priors in Driving Scenes](https://mlsanity.com/paper/2605.09688/) (2026)
- [VIPS: Vehicle-Infrastructure Cooperative Planning Benchmark via Pseudo-Simulation](https://mlsanity.com/paper/2609.02462/) (2026)

## Abstract (arXiv)

End-to-end training of multi-agent systems offers significant advantages in improving multi-task performance. However, training such models remains challenging and requires extensive manual design and monitoring. In this work, we introduce TurboTrain, a novel and efficient training framework for multi-agent perception and prediction. TurboTrain comprises two key components: a multi-agent spatiotemporal pretraining scheme based on masked reconstruction learning and a balanced multi-task learning strategy based on gradient conflict suppression. By streamlining the training process, our framework eliminates the need for manually designing and tuning complex multi-stage training pipelines, substantially reducing training time and improving performance. We evaluate TurboTrain on a real-world cooperative driving dataset, V2XPnP-Seq, and demonstrate that it further improves the performance of state-of-the-art multi-agent perception and prediction models. Our results highlight that pretraining effectively captures spatiotemporal multi-agent features and significantly benefits downstream tasks. Moreover, the proposed balanced multi-task learning strategy enhances detection and prediction.