mlsanityData to 14 Oct 2026

Comparative study of adapting pre-trained models for driving behavior video captioning

arXiv 2609.39542 ↗First posted 30 Sep 2026cs.CV cs.LGcited by 0

On the map: End-to-end autonomous driving → Large language models for driving also fits: VLMs for autonomous driving

Abstract (arXiv)

This report examines and compares some of the many fine tuning and prompting methods existing, applying them within the domain of autonomous driving. The idea is to compare these methods by adapting a Large Language Model (LLM) on a video dataset. LLM's have become extremely good at achieving a good understanding of different forms of data and this study aims to induce a low dimensional understanding of driving situations into our primary test model SpaceTimeGPT. Experiments on BDD-X (Berkeley DeepDrive eXplanation) dataset demonstrate good performance of the full fine tuning framework on some automatic metrics, and in some metrics, it even surpasses the baseline. We also try Low-Rank Adaptation (LoRA) and prompt engineering on VideoLLaVA model and discuss its limitations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Large language models for driving, one of 9 directions in End-to-end autonomous driving. Builds on 14 of its 19 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

14 of 19 references.

References · 14

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  5. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  6. 2021ViViT: A Video Vision Transformercited by 1,620
  7. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  8. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,288
  9. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  10. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
  11. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  12. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 234
2 more
  1. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  2. 2019End-to-End Video Captioningcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023ADAPT: Action-aware Driving Caption Transformer
  2. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving
  3. 2024Probing Multimodal LLMs as World Models for Driving
  4. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model
  5. 2026Mind the Temporal Gap: Evaluating Video Models for Temporal Understanding in Driving
  6. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Driving
  7. 2024Testing Large Language Models on Driving Theory Knowledge and Skills for Connected Autonomous Vehicles
  8. 2025Are Vision LLMs Road-Ready? A Comprehensive Benchmark for Safety-Critical Driving Video Understanding