mlsanityData to 14 Oct 2026

AIDE: An Automatic Data Engine for Object Detection in Autonomous Driving

arXiv 2403.17373 ↗First posted 26 Mar 2024cs.CV cs.AI cs.LGcited by 26

On the map: End-to-end autonomous driving → VLMs for autonomous driving

Abstract (arXiv)

Autonomous vehicle (AV) systems rely on robust perception models as a cornerstone of safety assurance. However, objects encountered on the road exhibit a long-tailed distribution, with rare or unseen categories posing challenges to a deployed perception model. This necessitates an expensive process of continuously curating and annotating data with significant human effort. We propose to leverage recent advances in vision-language and large language models to design an Automatic Data Engine (AIDE) that automatically identifies issues, efficiently curates data, improves the model through auto-labeling, and verifies the model through generation of diverse scenarios. This process operates iteratively, allowing for continuous self-improvement of the model. We further establish a benchmark for open-world detection on AV datasets to comprehensively evaluate various learning paradigms, demonstrating our method's superior performance at a reduced cost.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in VLMs for autonomous driving, one of 9 directions in End-to-end autonomous driving. Builds on 63 of its 80 references that are in the map. First posted on arXiv 26 Mar 2024.

Builds on

63 of 80 references.

References · 63

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  5. 2023Segment Anythingcited by 8,234
  6. 2020End-to-End Object Detection with Transformerscited by 7,414
  7. 2017Mask R-CNNcited by 7,277
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  10. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  11. 2015Fast R-CNNcited by 2,933
  12. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
38 more
  1. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,288
  2. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  3. 2021Grounded Language-Image Pre-trainingcited by 1,156
  4. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  5. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  6. 2020A Survey of Deep Active Learningcited by 554
  7. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  8. 2023Scaling Open-Vocabulary Object Detectioncited by 343
  9. 2021Unbiased Teacher for Semi-Supervised Object Detectioncited by 306
  10. 2021Towards Open World Object Detectioncited by 297
  11. 2023Fast Segment Anythingcited by 290
  12. 2022Generalized Category Discoverycited by 211
  13. 2017Incremental Learning of Object Detectors without Catastrophic Forgettingcited by 208
  14. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 198
  15. 2021A Unified Objective for Novel Class Discoverycited by 162
  16. 2021Offboard 3D Object Detection from Point Cloud Sequencescited by 145
  17. 2019Consistency-based Semi-supervised Active Learning: Towards Minimizing Labeling Costcited by 114
  18. 2021Learning Open-World Object Proposals without Learning to Classifycited by 109
  19. 2020Incremental Few-Shot Object Detectioncited by 102
  20. 2022Exploiting Unlabeled Data with Vision and Language Models for Object Detectioncited by 97
  21. 2022Learning Object-Language Alignments for Open-Vocabulary Object Detectioncited by 96
  22. 2023Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformerscited by 91
  23. 2022Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillationcited by 72
  24. 2022Unbiased Teacher v2: Semi-supervised Object Detection for Anchor-free and Anchor-based Detectorscited by 69
  25. 2021ORDisCo: Effective and Efficient Usage of Incremental Unlabeled Data for Semi-supervised Continual Learningcited by 68
  26. 2020Object Detection with a Unified Label Space from Multiple Datasetscited by 65
  27. 2022CLIP model is an Efficient Continual Learnercited by 65
  28. 2022Discovering Objects that Can Movecited by 44
  29. 2022Towards Open-Set Object Detection and Discoverycited by 44
  30. 2022X-DETR: A Versatile Architecture for Instance-wise Vision-Language Taskscited by 43
  31. 2022Consistent-Teacher: Towards Reducing Inconsistent Pseudo-targets in Semi-supervised Object Detectioncited by 43
  32. 2023Active Teacher for Semi-Supervised Object Detectioncited by 43
  33. 2021Auto4D: Learning to Label 4D Objects from Sequential Point Cloudscited by 42
  34. 2021Learning to Detect Every Thing in an Open Worldcited by 37
  35. 2021Wanderlust: Online Continual Object Detection in the Real Worldcited by 34
  36. 2022Active Learning for Open-set Annotationcited by 34
  37. 2021Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay Buffercited by 30
  38. 2021Not All Labels Are Equal: Rationalizing The Labeling Costs for Training Object Detectioncited by 27

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 10; 2026: 8.

26 papers.

Sort:

By area:

  1. 2024Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generationcited by 55
  2. 2024ROADWork: A Dataset and Benchmark for Learning to Recognize, Observe, Analyze and Drive Through Work Zonescited by 22
  3. 2024MoExtend: Tuning New Experts for Modality and Task Extensioncited by 12
  4. 2025HiBug2: Efficient and Interpretable Error Slice Discovery for Comprehensive Model Debuggingcited by 11
  5. 2025Towards a Multi-Agent Vision-Language System for Zero-Shot Novel Hazardous Object Detection for Autonomous Driving Safetycited by 9
  6. 2025Towards Scalable Spatial Intelligence via 2D-to-3D Data Liftingcited by 8
  7. 2026World Engine: Towards the Era of Post-Training for Autonomous Drivingcited by 8
  8. 2024XLM for Autonomous Driving Systems: A Comprehensive Reviewcited by 5
  9. 2024A Retention-Centric Framework for Continual Learning with Guaranteed Model Developmental Safetycited by 4
  10. 2025No Labels, No Problem: Training Visual Reasoners with Multimodal Verifierscited by 4
  11. 2024MTA: Multimodal Task Alignment for BEV Perception and Captioningcited by 3
  12. 2024Can't Slow me Down: Learning Robust and Hardware-Adaptive Object Detectors against Latency Attacks for Edge Devicescited by 3
14 more
  1. 2025Mcity Data Engine: Iterative Model Improvement Through Open-Vocabulary Data Selectioncited by 3
  2. 2025A Novel Multi-layer Task-centric and Data Quality Framework for Autonomous Drivingcited by 2
  3. 2025PublicAgent: Multi-Agent Design Principles From an LLM-Based Open Data Analysis Frameworkcited by 2
  4. 2024Real-Time Incremental Explanations for Object Detectors in Autonomous Drivingcited by 1
  5. 2026A text-guided vision model for enhanced recognition of small instancescited by 1
  6. 2025Incremental Object Keypoint Learningcited by 0
  7. 2025SAM2Auto: Auto Annotation Using FLASHcited by 0
  8. 2025Dataset Safety in Autonomous Driving: Requirements, Risks, and Assurancecited by 0
  9. 2026SMc2f: Robust Scenario Mining for Robotic Autonomy from Coarse to Finecited by 0
  10. 2026Modeling and Measuring Redundancy in Multisource Multimodal Data for Autonomous Drivingcited by 0
  11. 2026TraVEL: Trajectory-Guided Video Embedding Learning for Driving-Video Retrievalcited by 0
  12. 2026Understanding Autonomous Driving Datasets by Describing Differences between Image Subsets in Natural Languagecited by 0
  13. 2026Auditing Quality Filters for Long-Tail Human Data Curationcited by 0
  14. 2026From internal representations to model improvement through prediction errorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025All You Need for Object Detection: From Pixels, Points, and Prompts to Next-Gen Fusion and Multimodal LLMs/VLMs in Autonomous Vehicles
  2. 2025AutoVDC: Automated Vision Data Cleaning Using Vision-Language Models
  3. 2025CARScenes: Semantic VLM Dataset for Safe Autonomous Driving
  4. 2025AD-EE: Early Exiting for Fast and Reliable Vision-Language Models in Autonomous Driving
  5. 2025Leveraging Text-Driven Semantic Variation for Robust OOD Segmentation
  6. 2021LiDAR Cluster First and Camera Inference Later: A New Perspective Towards Autonomous Driving
  7. 2025Evaluation of Large Language Models for Anomaly Detection in Autonomous Vehicles
  8. 2025Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models