# Scaling Datasets for Multi-Sensor, Multi-Agent, and Multi-Domain Learning in Autonomous Systems

arXiv: https://arxiv.org/abs/2606.04444 · HTML: https://mlsanity.com/paper/2606.04444/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 3 Jun 2026.

## Builds on (most cited first, 25 of 25)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [Learning by Cheating](https://mlsanity.com/paper/1912.12294/) (2019)
- [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021)
- [OPV2V: An Open Benchmark Dataset and Fusion Pipeline for Perception with Vehicle-to-Vehicle Communication](https://mlsanity.com/paper/2109.07644/) (2021)
- [End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordances](https://mlsanity.com/paper/1911.10868/) (2019)
- [EagerMOT: 3D Multi-Object Tracking via Sensor Fusion](https://mlsanity.com/paper/2104.14682/) (2021)
- [AB3DMOT: A Baseline for 3D Multi-Object Tracking and New Evaluation Metrics](https://mlsanity.com/paper/2008.08063/) (2020)
- [AVstack: An Open-Source, Reconfigurable Platform for Autonomous Vehicle Development](https://mlsanity.com/paper/2212.13857/) (2022)
- [LLaViDA: A Large Language Vision Driving Assistant for Explicit Reasoning and Enhanced Trajectory Planning](https://mlsanity.com/paper/2512.18211/) (2025)
- [Security-Aware Sensor Fusion with MATE: the Multi-Agent Trust Estimator](https://mlsanity.com/paper/2503.04954/) (2025)
- [A Cooperative Perception Environment for Traffic Operations and Control](https://mlsanity.com/paper/2208.02792/) (2022)
- [Exploiting Trust for Resilient Hypothesis Testing with Malicious Robots](https://mlsanity.com/paper/2209.12285/) (2022)
- [Datasets, Models, and Algorithms for Multi-Sensor, Multi-agent Autonomy Using AVstack](https://mlsanity.com/paper/2312.04970/) (2023)
- [Bayesian Methods for Trust in Collaborative Multi-Agent Autonomy](https://mlsanity.com/paper/2403.16956/) (2024)
- [V2X Misbehavior and Collective Perception Service: Considerations for Standardization](https://mlsanity.com/paper/2112.02184/) (2021)
- [A Modular Platform For Collaborative, Distributed Sensor Fusion](https://mlsanity.com/paper/2303.07430/) (2023)
- [A Multi-Agent Security Testbed for the Analysis of Attacks and Defenses in Collaborative Sensor Fusion](https://mlsanity.com/paper/2401.09387/) (2024)
- [Probabilistic Segmentation for Robust Field of View Estimation](https://mlsanity.com/paper/2503.07375/) (2025)

## Abstract (arXiv)

Existing datasets cannot support large-scale learning in multi-agent, multi-sensor, or multi-domain autonomy, where diversity and coordination are essential. We present a modular dataset generation pipeline that creates terabyte-scale, ground-truth-labeled data for ground, aerial, and infrastructure-based systems using the AVstack framework and CARLA simulator. Supporting single- and multi-agent configurations with flexible sensor suites, the pipeline enables controllable experimentation across challenging conditions. Representative perception and fusion studies show how generated data can support application-specific training and collaborative autonomy.