mlsanityData to 14 Oct 2026

Know Your Surroundings: Panoramic Multi-Object Tracking by Multimodality Collaboration

arXiv 2105.14683 ↗First posted 31 May 2021cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

In this paper, we focus on the multi-object tracking (MOT) problem of automatic driving and robot navigation. Most existing MOT methods track multiple objects using a singular RGB camera, which are prone to camera field-of-view and suffer tracking failures in complex scenarios due to background clutters and poor light conditions. To meet these challenges, we propose a MultiModality PAnoramic multi-object Tracking framework (MMPAT), which takes both 2D panorama images and 3D point clouds as input and then infers target trajectories using the multimodality data. The proposed method contains four major modules, a panorama image detection module, a multimodality data fusion module, a data association module and a trajectory inference model. We evaluate the proposed method on the JRDB dataset, where the MMPAT achieves the top performance in both the detection and tracking tasks and significantly outperforms state-of-the-art methods by a large margin (15.7 and 8.5 improvement in terms of AP and MOTA, respectively).

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 63 of its 92 references that are in the map. First posted on arXiv 31 May 2021.

Builds on

63 of 92 references.

References · 63

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  4. 2020End-to-End Object Detection with Transformerscited by 7,414
  5. 2017Mask R-CNNcited by 7,277
  6. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  7. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  8. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  9. 2015SSD: Single Shot MultiBox Detectorcited by 3,130
  10. 2018YOLOv3: An Incremental Improvementcited by 2,659
  11. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,715
  12. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,577
38 more
  1. 2017Deformable Convolutional Networkscited by 1,488
  2. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  3. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,276
  4. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 941
  5. 2017Simple Online and Realtime Tracking with a Deep Association Metriccited by 910
  6. 2018CornerNet: Detecting Objects as Paired Keypointscited by 763
  7. 2016Multi-View 3D Object Detection Network for Autonomous Drivingcited by 724
  8. 2016Performance Measures and a Data Set for Multi-Target, Multi-Camera Trackingcited by 692
  9. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  10. 2018Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Drivingcited by 457
  11. 2019Bag of Tricks and A Strong Baseline for Deep Person Re-identificationcited by 430
  12. 2019Cascade R-CNN: High Quality Object Detection and Instance Segmentationcited by 429
  13. 2017Soft-NMS -- Improving Object Detection With One Line of Codecited by 382
  14. 2019Tracking without bells and whistlescited by 382
  15. 2019PIXOR: Real-time 3D Object Detection from Point Cloudscited by 358
  16. 2017Joint 3D Proposal Generation and Object Detection from View Aggregationcited by 326
  17. 2020Deep Continuous Fusion for Multi-Sensor 3D Object Detectioncited by 263
  18. 2019M3D-RPN: Monocular 3D Region Proposal Network for Object Detectioncited by 252
  19. 2020Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Netcited by 228
  20. 2020Multi-Task Multi-Sensor Fusion for 3D Object Detectioncited by 203
  21. 2019Learning a Neural Solver for Multiple Object Trackingcited by 198
  22. 2019Stereo R-CNN based 3D Object Detection for Autonomous Drivingcited by 163
  23. 2019Bayesian Loss for Crowd Count Estimation with Point Supervisioncited by 162
  24. 2019Accurate Monocular Object Detection via Color-Embedded 3D Reconstruction for Autonomous Drivingcited by 137
  25. 2016Vehicle Detection from 3D Lidar Using Fully Convolutional Networkcited by 107
  26. 2019Monocular 3D Object Detection Leveraging Accurate Proposals and Shape Reconstructioncited by 106
  27. 2017Deep MANTA: A Coarse-to-fine Many-Task Network for joint 2D and 3D vehicle analysis from monocular imagecited by 95
  28. 2019FreeAnchor: Learning to Match Anchors for Visual Object Detectioncited by 90
  29. 2018MonoGRNet: A Geometric Reasoning Network for Monocular 3D Object Localizationcited by 84
  30. 2018Joint Monocular 3D Vehicle Detection and Trackingcited by 77
  31. 2019Online Multi-Object Tracking with Dual Matching Attention Networkscited by 77
  32. 2017Tracking The Untrackable: Learning To Track Multiple Cues with Long-Term Dependenciescited by 76
  33. 2019Spatial-Temporal Relation Networks for Multi-Object Trackingcited by 74
  34. 2016Vote3Deep: Fast Object Detection in 3D Point Clouds Using Efficient Convolutional Neural Networkscited by 72
  35. 2019FAMNet: Joint Learning of Feature, Affinity and Multi-dimensional Assignment for Online Multiple Object Trackingcited by 65
  36. 20163D Object Proposals using Stereo Imagery for Accurate Object Class Detectioncited by 58
  37. 2020Lifted Disjoint Paths with Application in Multiple Object Trackingcited by 55
  38. 2017Online Multi-Object Tracking Using CNN-based Single Object Tracker with Spatial-Temporal Attention Mechanismcited by 48

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 0; 2023: 2; 2024: 2; 2025: 2; 2026: 1.

8 papers.

  1. 2021JRDB-Act: A Large-scale Dataset for Spatio-temporal Action, Social Group and Activity Detectioncited by 39
  2. 2023Real-time Trajectory-based Social Group Detectioncited by 4
  3. 20233D Multiple Object Tracking on Autonomous Driving: A Literature Reviewcited by 4
  4. 2025OmniTrack++: Omnidirectional Multi-Object Tracking by Learning Large-FoV Trajectory Feedbackcited by 4
  5. 2024AdaFPP: Adapt-Focused Bi-Propagating Prototype Learning for Panoramic Activity Recognitioncited by 2
  6. 2024Social-MAE: Social Masked Autoencoder for Multi-person Motion Representation Learningcited by 0
  7. 2025Inference of Human-derived Specifications of Object Placement via Demonstrationcited by 0
  8. 2026Re-engineering SORT-based algorithms for low-cost small object tracking from omnidirectional footagecited by 0