# Unlock the Power of Unlabeled Data in Language Driving Model

arXiv: https://arxiv.org/abs/2503.10586 · HTML: https://mlsanity.com/paper/2503.10586/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in VLMs for autonomous driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 29 of its 44 references that are in the map.
- First posted on arXiv 13 Mar 2025.

## Placed in

- End-to-end autonomous driving / [VLMs for autonomous driving](https://mlsanity.com/driving/vlms-for-autonomous-driving/)

## Builds on (most cited first, 29 of 29)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [GAIA-1: A Generative World Model for Autonomous Driving](https://mlsanity.com/paper/2309.17080/) (2023)
- [DriveLM: Driving with Graph Visual Question Answering](https://mlsanity.com/paper/2312.14150/) (2023)
- [DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models](https://mlsanity.com/paper/2402.12289/) (2024)
- [DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model](https://mlsanity.com/paper/2310.01412/) (2023)
- [Meta Pseudo Labels](https://mlsanity.com/paper/2003.10580/) (2020)
- [GPT-Driver: Learning to Drive with GPT](https://mlsanity.com/paper/2310.01415/) (2023)
- [DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving](https://mlsanity.com/paper/2309.09777/) (2023)
- [Selective Annotation Makes Language Models Better Few-Shot Learners](https://mlsanity.com/paper/2209.01975/) (2022)
- [DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving](https://mlsanity.com/paper/2312.09245/) (2023)
- [Is GPT-3 a Good Data Annotator?](https://mlsanity.com/paper/2212.10450/) (2022)
- [DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models](https://mlsanity.com/paper/2309.16292/) (2023)
- [Want To Reduce Labeling Cost? GPT-3 Can Help](https://mlsanity.com/paper/2108.13487/) (2021)
- [VLP: Vision Language Planning for Autonomous Driving](https://mlsanity.com/paper/2401.05577/) (2024)
- [End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures](https://mlsanity.com/paper/1911.08460/) (2019)
- [Language Prompt for Autonomous Driving](https://mlsanity.com/paper/2309.04379/) (2023)
- [Real-Time Flying Object Detection with YOLOv8](https://mlsanity.com/paper/2305.09972/) (2023)
- [DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion model](https://mlsanity.com/paper/2310.07771/) (2023)
- [LLM-Assist: Enhancing Closed-Loop Planning with Language-Based Reasoning](https://mlsanity.com/paper/2401.00125/) (2023)
- [LC-LLM: Explainable Lane-Change Intention and Trajectory Predictions with Large Language Models](https://mlsanity.com/paper/2403.18344/) (2024)
- [SemiFed: Semi-supervised Federated Learning with Consistency and Pseudo-Labeling](https://mlsanity.com/paper/2108.09412/) (2021)
- [Language-Guided 3D Object Detection in Point Cloud for Autonomous Driving](https://mlsanity.com/paper/2305.15765/) (2023)
- [BEV-TSR: Text-Scene Retrieval in BEV Space for Autonomous Driving](https://mlsanity.com/paper/2401.01065/) (2024)
- [DriveGenVLM: Real-world Video Generation for Vision Language Model based Autonomous Driving](https://mlsanity.com/paper/2408.16647/) (2024)
- [Ask Me What You Need: Product Retrieval using Knowledge from GPT-3](https://mlsanity.com/paper/2207.02516/) (2022)
- [Driving with InternVL: Oustanding Champion in the Track on Driving with Language of the Autonomous Grand Challenge at CVPR 2024](https://mlsanity.com/paper/2412.07247/) (2024)
- [Augmentation Learning for Semi-Supervised Classification](https://mlsanity.com/paper/2208.01956/) (2022)

## Built on by (most cited first, 1 of 1)

- [VeriDrive: Verifiable Counterfactual Supervision for Cost-Efficient Vision-Language Planning](https://mlsanity.com/paper/2606.07338/) (2026)

## Abstract (arXiv)

Recent Vision-based Large Language Models~(VisionLLMs) for autonomous driving have seen rapid advancements. However, such promotion is extremely dependent on large-scale high-quality annotated data, which is costly and labor-intensive. To address this issue, we propose unlocking the value of abundant yet unlabeled data to improve the language-driving model in a semi-supervised learning manner. Specifically, we first introduce a series of template-based prompts to extract scene information, generating questions that create pseudo-answers for the unlabeled data based on a model trained with limited labeled data. Next, we propose a Self-Consistency Refinement method to improve the quality of these pseudo-annotations, which are later used for further training. By utilizing a pre-trained VisionLLM (e.g., InternVL), we build a strong Language Driving Model (LDM) for driving scene question-answering, outperforming previous state-of-the-art methods. Extensive experiments on the DriveLM benchmark show that our approach performs well with just 5% labeled data, achieving competitive performance against models trained with full datasets. In particular, our LDM achieves 44.85% performance with limited labeled data, increasing to 54.27% when using unlabeled data, while models trained with full datasets reach 60.68% on the DriveLM benchmark.