# Talk to Parallel LiDARs: A Human-LiDAR Interaction Method Based on 3D Visual Grounding

arXiv: https://arxiv.org/abs/2405.15274 · HTML: https://mlsanity.com/paper/2405.15274/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 44 references that are in the map.
- First posted on arXiv 24 May 2024.

## Builds on (most cited first, 25 of 25)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation](https://mlsanity.com/paper/2205.13542/) (2022)
- [TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers](https://mlsanity.com/paper/2203.11496/) (2022)
- [ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language](https://mlsanity.com/paper/1912.08830/) (2019)
- [Multi3DRefer: Grounding Text Description to Multiple 3D Objects](https://mlsanity.com/paper/2309.05251/) (2023)
- [Talk2Car: Taking Control of Your Self-Driving Car](https://mlsanity.com/paper/1909.10838/) (2019)
- [Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Clouds](https://mlsanity.com/paper/2112.08879/) (2021)
- [SAT: 2D Semantics Assisted Training for 3D Visual Grounding](https://mlsanity.com/paper/2105.11450/) (2021)
- [Language Prompt for Autonomous Driving](https://mlsanity.com/paper/2309.04379/) (2023)
- [3D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selection](https://mlsanity.com/paper/2204.06272/) (2022)
- [EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Grounding](https://mlsanity.com/paper/2209.14941/) (2022)
- [Referring Multi-Object Tracking](https://mlsanity.com/paper/2303.03366/) (2023)
- [TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Grounding](https://mlsanity.com/paper/2108.02388/) (2021)
- [Mono3DVG: 3D Visual Grounding in Monocular Images](https://mlsanity.com/paper/2312.08022/) (2023)
- [Language-Guided 3D Object Detection in Point Cloud for Autonomous Driving](https://mlsanity.com/paper/2305.15765/) (2023)
- [Emergent Visual Sensors for Autonomous Vehicles](https://mlsanity.com/paper/2205.09383/) (2022)
- [ScanERU: Interactive 3D Visual Grounding based on Embodied Reference Understanding](https://mlsanity.com/paper/2303.13186/) (2023)
- [Cosine meets Softmax: A tough-to-beat baseline for visual grounding](https://mlsanity.com/paper/2009.06066/) (2020)
- [HPL-ViT: A Unified Perception Framework for Heterogeneous Parallel LiDARs in V2V](https://mlsanity.com/paper/2309.15572/) (2023)
- [Commands 4 Autonomous Vehicles (C4AV) Workshop Summary](https://mlsanity.com/paper/2009.08792/) (2020)
- [Predicting Physical World Destinations for Commands Given to Self-Driving Cars](https://mlsanity.com/paper/2112.05419/) (2021)

## Built on by (most cited first, 5 of 5)

- [A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directions](https://mlsanity.com/paper/2406.05785/) (2024)
- [3EED: Ground Everything Everywhere in 3D](https://mlsanity.com/paper/2511.01755/) (2025)
- [LidaRefer: Context-aware Outdoor 3D Visual Grounding for Autonomous Driving](https://mlsanity.com/paper/2411.04351/) (2024)
- [A Coarse-to-Fine Approach to Multi-Modality 3D Occupancy Grounding](https://mlsanity.com/paper/2508.01197/) (2025)
- [Talk2Sensors: 3D Visual Grounding in Autonomous Driving via Sensor-Adaptive Physical Cue Matching](https://mlsanity.com/paper/2608.04568/) (2026)

## Abstract (arXiv)

LiDAR sensors play a crucial role in various applications, especially in autonomous driving. Current research primarily focuses on optimizing perceptual models with point cloud data as input, while the exploration of deeper cognitive intelligence remains relatively limited. To address this challenge, parallel LiDARs have emerged as a novel theoretical framework for the next-generation intelligent LiDAR systems, which tightly integrate physical, digital, and social systems. To endow LiDAR systems with cognitive capabilities, we introduce the 3D visual grounding task into parallel LiDARs and present a novel human-computer interaction paradigm for LiDAR systems. We propose Talk2LiDAR, a large-scale benchmark dataset tailored for 3D visual grounding in autonomous driving. Additionally, we present a two-stage baseline approach and an efficient one-stage method named BEVGrounding, which significantly improves grounding accuracy by fusing coarse-grained sentence and fine-grained word embeddings with visual features. Our experiments on Talk2Car-3D and Talk2LiDAR datasets demonstrate the superior performance of BEVGrounding, laying a foundation for further research in this domain.