mlsanityData to 14 Oct 2026

RCVLA: 4D Radar-Grounded Semantic Reasoning and Trajectory Arbitration for Autonomous Driving

arXiv 2609.32681 ↗First posted 26 Sep 2026cs.CVcited by 0

On the map: End-to-end autonomous driving → VLMs for autonomous driving also fits: Reasoning VLA and VLM driving policies

Abstract (arXiv)

4D radar provides geometric and motion cues that complement visual semantics, but integrating it into vision-language-action (VLA) models requires both radar--language alignment for semantic reasoning and explicit use of radar measurements for trajectory refinement and selection. To support these capabilities, we construct Cap4DR with 86,016 radar-image-text samples for alignment pretraining and OmniHD-QA with 520,161 question-answer pairs for instruction tuning across scene description, key-object reasoning, occupancy understanding, and trajectory planning. Building on these datasets, we propose RCVLA, a radar-camera VLA framework consisting of a radar-grounded semantic reasoning stage (RCVLA-Sem) and a trajectory arbitration stage (RCVLA-Phys). RCVLA-Sem performs gated bidirectional interaction between camera and radar tokens for driving question answering and reference trajectory generation, while auxiliary heads provide object and occupancy queries. RCVLA-Phys refines reference-guided trajectory candidates through truncated diffusion conditioned on these queries and cluster-level radar measurements, then calibrates candidate scores using radar-derived time-to-collision risk. On OmniHD-QA, RCVLA-Sem improves CIDEr by 9.92 points and reduces key-object velocity error by 21.9% relative to OmniDrive. RCVLA-Phys further reduces average L2 error from 0.348 to 0.259,m and average open-loop collision rate from 0.576% to 0.175% relative to RCVLA-Sem. Ablation studies further show that language-aligned radar tokens improve semantic reasoning, while cluster-level radar measurements and risk calibration improve trajectory arbitration. Code will be released.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLMs for autonomous driving, one of 9 directions in End-to-end autonomous driving. Builds on 24 of its 31 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

24 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023DriveLM: Driving with Graph Visual Question Answering21 Dec

Other references · 23

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  5. 2025Qwen3-VL Technical Reportcited by 2,793
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  7. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  8. 2022Planning-oriented Autonomous Drivingcited by 1,087
  9. 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
  10. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 368
  11. 2024DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Drivingcited by 334
  12. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
11 more
  1. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Drivingcited by 156
  2. 2025OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 143
  3. 2022TJ4DRadSet: A 4D Radar Dataset for Autonomous Drivingcited by 75
  4. 2023Dual Radar: A Multi-modal Dataset with Dual 4D Radar for Autonomous Drivingcited by 32
  5. 2024OmniHD-Scenes: A Next-Generation Multimodal Dataset for Autonomous Drivingcited by 27
  6. 2025Doracamom: Joint 3D Detection and Occupancy Prediction with Multi-view 4D Radars and Cameras for Omnidirectional Perceptioncited by 22
  7. 2024Talk2Radar: Bridging Natural Language with 4D mmWave Radar for 3D Referring Expression Comprehensioncited by 18
  8. 2023NTU4DRadLM: 4D Radar-centric Multi-Modal Dataset for Localization and Mappingcited by 17
  9. 2024WaterVG: Waterway Visual Grounding based on Text-Guided Vision and mmWave Radarcited by 13
  10. 2024NanoMVG: USV-Centric Low-Power Multi-Task Visual Grounding based on Prompt-Guided Camera and 4D mmWave Radarcited by 7
  11. 2024SNAIL Radar: A large-scale diverse benchmark for evaluating 4D-radar-based SLAMcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Radar4D-VLM: Proposal-Grounded Temporal 4D Radar Reasoning Across Frozen Language Models
  2. 2026STAR-VLM: Spatiotemporal Grounding Vision-Language Models for Motion and Velocity Estimation via Automotive Radar Supervision
  3. 2025CoT4AD: A Vision-Language-Action Model with Explicit Chain-of-Thought Reasoning for Autonomous Driving
  4. 2026Deferred Exposure of Future Trajectories for Verifiable Reasoning in Autonomous Driving VLMs
  5. 2025RLM: A Vision-Language Model Approach for Radar Scene Understanding
  6. 2026MoRAL: Sensor-Grounded BEV Reasoning for Compact VLMs toward Edge-Oriented Autonomous Driving
  7. 2026S-squared-VLA: Decoupling Semantic and Spatial Streams in Vision-Language-Action Models for Autonomous Driving
  8. 2025A Survey on Vision-Language-Action Models for Autonomous Driving