# Physical AI Smart Spaces: A Large-Scale Benchmark for Multi-Camera 3D Perception in Smart Spaces

arXiv: https://arxiv.org/abs/2610.02580 · HTML: https://mlsanity.com/paper/2610.02580/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 35 references that are in the map.
- First posted on arXiv 1 Oct 2026.

## Builds on (most cited first, 13 of 13)

- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [Performance Measures and a Data Set for Multi-Target, Multi-Camera Tracking](https://mlsanity.com/paper/1609.01775/) (2016)
- [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020)
- [World Simulation with Video Foundation Models for Physical AI](https://mlsanity.com/paper/2511.00062/) (2025)
- [Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control](https://mlsanity.com/paper/2503.14492/) (2025)
- [Multiview Detection with Feature Perspective Transformation](https://mlsanity.com/paper/2007.07247/) (2020)
- [MMPTRACK: Large-scale Densely Annotated Multi-camera Multiple People Tracking Benchmark](https://mlsanity.com/paper/2111.15157/) (2021)
- [The 8th AI City Challenge](https://mlsanity.com/paper/2404.09432/) (2024)
- [The 9th AI City Challenge](https://mlsanity.com/paper/2508.13564/) (2025)
- [MCBLT: Multi-Camera Multi-Object 3D Tracking in Long Videos](https://mlsanity.com/paper/2412.00692/) (2024)
- [MTMMC: A Large-Scale Real-World Multi-Modal Camera Tracking Benchmark](https://mlsanity.com/paper/2403.20225/) (2024)
- [Online 3D Multi-Camera Perception through Robust 2D Tracking and Depth-based Late Aggregation](https://mlsanity.com/paper/2509.09946/) (2025)
- [A Unified 3D Object Perception Framework for Real-Time Outside-In Multi-Camera Systems](https://mlsanity.com/paper/2601.10819/) (2026)

## Abstract (arXiv)

Physical AI Smart Spaces is, to the best of our knowledge, the first benchmark to simultaneously provide large-scale, multi-class, and multi-camera 3D perception data for indoor smart spaces. It contains over 280 hours of synchronized 1080p footage captured by nearly 1,800 cameras in warehouses, hospitals, retail venues, and similar settings, together with automatic annotations for multi-camera identities, 2D bounding boxes, 3D bounding boxes, camera calibration, and depth where available. The benchmark spans Isaac Sim synthetic generation, Cosmos Transfer appearance augmentation, and real-world Sim2Real evaluation. For the real-world target, we include two warehouse deployments with time-synchronized streams, automatic VGGT-based calibration, and a 3D labeling interface that projects world-frame 3D boxes into each view for cross-camera verification. We describe the dataset scope, annotation and calibration schema, generation workflow, benchmark protocols, and official evaluation system, which standardizes submission format, and leaderboard reporting. A central contribution is a 3D instantiation of Higher Order Tracking Accuracy (HOTA), extending the usual 2D box-based tracking evaluation to 3D locations and 3D boxes. We further report empirical baselines from the AI City Challenge leaderboards, showing how methods evolve from person-only 3D location tracking to multi-class 3D box tracking under realistic smart-space constraints. The release is available at https://huggingface.co/datasets/nvidia/PhysicalAI-SmartSpaces.