# LEGO: Learning and Graph-Optimized Modular Tracker for Online Multi-Object Tracking with Point Clouds

arXiv: https://arxiv.org/abs/2308.09908 · HTML: https://mlsanity.com/paper/2308.09908/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 104 references that are in the map.
- First posted on arXiv 19 Aug 2023.

## Builds on (most cited first, 30 of 47)

- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020)
- [FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking](https://mlsanity.com/paper/2004.01888/) (2020)
- [Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud](https://mlsanity.com/paper/2003.01251/) (2020)
- [StrongSORT: Make DeepSORT Great Again](https://mlsanity.com/paper/2202.13514/) (2022)
- [GNN3DMOT: Graph Neural Network for 3D Multi-Object Tracking with Multi-Feature Learning](https://mlsanity.com/paper/2006.07327/) (2020)
- [SimpleTrack: Understanding and Rethinking 3D Multi-object Tracking](https://mlsanity.com/paper/2111.09621/) (2021)
- [EagerMOT: 3D Multi-Object Tracking via Sensor Fusion](https://mlsanity.com/paper/2104.14682/) (2021)
- [Joint Monocular 3D Vehicle Detection and Tracking](https://mlsanity.com/paper/1811.10742/) (2018)
- [Probabilistic 3D Multi-Modal, Multi-Object Tracking for Autonomous Driving](https://mlsanity.com/paper/2012.13755/) (2020)
- [Probabilistic 3D Multi-Object Tracking for Autonomous Driving](https://mlsanity.com/paper/2001.05673/) (2020)
- [LXL: LiDAR Excluded Lean 3D Object Detection with 4D Imaging Radar and Camera Fusion](https://mlsanity.com/paper/2307.00724/) (2023)
- [Robust Multi-Modality Multi-Object Tracking](https://mlsanity.com/paper/1909.03850/) (2019)
- [Exploring Simple 3D Multi-Object Tracking for Autonomous Driving](https://mlsanity.com/paper/2108.10312/) (2021)
- [CAMO-MOT: Combined Appearance-Motion Optimization for 3D Multi-Object Tracking with Camera-LiDAR Fusion](https://mlsanity.com/paper/2209.02540/) (2022)
- [SMURF: Spatial Multi-Representation Fusion for 3D Object Detection with 4D Imaging Radar](https://mlsanity.com/paper/2307.10784/) (2023)
- [Learnable Graph Matching: Incorporating Graph Partitioning with Deep Feature Learning for Multiple Object Tracking](https://mlsanity.com/paper/2103.16178/) (2021)
- [Learning a Proposal Classifier for Multiple Object Tracking](https://mlsanity.com/paper/2103.07889/) (2021)
- [Immortal Tracker: Tracklet Never Dies](https://mlsanity.com/paper/2111.13672/) (2021)
- [Learnable Online Graph Representations for 3D Multi-Object Tracking](https://mlsanity.com/paper/2104.11747/) (2021)
- [DeepFusionMOT: A 3D Multi-Object Tracking Framework Based on Camera-LiDAR Fusion with Deep Association](https://mlsanity.com/paper/2202.12100/) (2022)
- [Score refinement for confidence-based 3D multi-object tracking](https://mlsanity.com/paper/2107.04327/) (2021)
- [PolarMOT: How Far Can Geometric Relations Take Us in 3D Multi-Object Tracking?](https://mlsanity.com/paper/2208.01957/) (2022)
- [FANTrack: 3D Multi-Object Tracking with Feature Association Network](https://mlsanity.com/paper/1905.02843/) (2019)
- [3D Multi-Object Tracking Using Graph Neural Networks with Cross-Edge Modality Attention](https://mlsanity.com/paper/2203.10926/) (2022)

## Built on by (most cited first, 9 of 9)

- [Doracamom: Joint 3D Detection and Occupancy Prediction with Multi-view 4D Radars and Cameras for Omnidirectional Perception](https://mlsanity.com/paper/2501.15394/) (2025)
- [MCTrack: A Unified 3D Multi-Object Tracking Framework for Autonomous Driving](https://mlsanity.com/paper/2409.16149/) (2024)
- [On the Federated Learning Framework for Cooperative Perception](https://mlsanity.com/paper/2404.17147/) (2024)
- [LiDAR MOT-DETR: A LiDAR-based Two-Stage Transformer for 3D Multiple Object Tracking](https://mlsanity.com/paper/2505.12753/) (2025)
- [One Homography is All You Need: IMM-based Joint Homography and Multiple Object State Estimation](https://mlsanity.com/paper/2409.02562/) (2024)
- [OptiPMB: Enhancing 3D Multi-Object Tracking with Optimized Poisson Multi-Bernoulli Filtering](https://mlsanity.com/paper/2503.12968/) (2025)
- [Stable at Any Speed: Speed-Driven Multi-Object Tracking with Learnable Kalman Filtering](https://mlsanity.com/paper/2508.00358/) (2025)
- [FutrTrack: A Camera-LiDAR Fusion Transformer for 3D Multiple Object Tracking](https://mlsanity.com/paper/2510.19981/) (2025)
- [MotionSync: Non-Causal Refinement of Causal Tracker for Label-Efficient 3D Perception](https://mlsanity.com/paper/2608.29567/) (2026)

## Abstract (arXiv)

Online multi-object tracking (MOT) plays a pivotal role in autonomous systems. The state-of-the-art approaches usually employ a tracking-by-detection method, and data association plays a critical role. This paper proposes a learning and graph-optimized (LEGO) modular tracker to improve data association performance in the existing literature. The proposed LEGO tracker integrates graph optimization and self-attention mechanisms, which efficiently formulate the association score map, facilitating the accurate and efficient matching of objects across time frames. To further enhance the state update process, the Kalman filter is added to ensure consistent tracking by incorporating temporal coherence in the object states. Our proposed method utilizing LiDAR alone has shown exceptional performance compared to other online tracking approaches, including LiDAR-based and LiDAR-camera fusion-based methods. LEGO ranked 1st at the time of submitting results to KITTI object tracking evaluation ranking board and remains 2nd at the time of submitting this paper, among all online trackers in the KITTI MOT benchmark for cars1