mlsanityData to 7 Oct 2026

Can you text what is happening? Integrating pre-trained language encoders into trajectory prediction models for autonomous driving

arXiv 2309.05282 ↗First posted 11 Sep 2023cs.CV cs.AI cs.LGcited by 30

Not on the map. Why?

Abstract (arXiv)

In autonomous driving tasks, scene understanding is the first step towards predicting the future behavior of the surrounding traffic participants. Yet, how to represent a given scene and extract its features are still open research questions. In this study, we propose a novel text-based representation of traffic scenes and process it with a pre-trained language encoder. First, we show that text-based representations, combined with classical rasterized image representations, lead to descriptive scene embeddings. Second, we benchmark our predictions on the nuScenes dataset and show significant improvements compared to baselines. Third, we show in an ablation study that a joint encoder of text and rasterized images outperforms the individual encoders confirming that both representations have their complementary strengths.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 25 of its 27 references that are in the map. First posted on arXiv 11 Sep 2023.

Builds on

25 of 27 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  10. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
13 more
  1. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  2. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  3. 2020A Survey on Visual Transformercited by 653
  4. 2020VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representationcited by 528
  5. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 417
  6. 2023UniSim: A Neural Closed-Loop Sensor Simulatorcited by 294
  7. 2018Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networkscited by 289
  8. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 273
  9. 2019CoverNet: Multimodal Behavior Prediction using Trajectory Setscited by 259
  10. 2021GOHOME: Graph-Oriented Heatmap Output for future Motion Estimationcited by 146
  11. 2021Multimodal Trajectory Prediction Conditioned on Lane-Graph Traversalscited by 112
  12. 2023Language Conditioned Traffic Generationcited by 86
  13. 2023Cheap and Deterministic Inference for Deep State-Space Models of Interacting Dynamical Systemscited by 5

Built on by

Cited by year

Citing papers per year: 2023: 8; 2024: 13; 2025: 9.

30 papers.

Sort:

By area:

  1. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  2. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 188
  3. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  4. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 124
  5. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervisioncited by 85
  6. 2024OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Drivingcited by 82
  7. 2024Embodied Understanding of Driving Scenarioscited by 63
  8. 2024A Survey for Foundation Models in Autonomous Drivingcited by 51
  9. 2024Asynchronous Large Language Model Enhanced Planner for Autonomous Drivingcited by 38
  10. 2023Empowering Autonomous Driving with Large Language Models: A Safety Perspectivecited by 34
  11. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  12. 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
18 more
  1. 2024Towards Urban General Intelligence: A Review and Outlook of Urban Foundation Modelscited by 23
  2. 2024Hallucination Detection in Foundation Models for Decision-Making: A Flexible Definition and Review of the State of the Artcited by 17
  3. 2025Embodied Scene Understanding for Vision Language Models via MetaVQAcited by 16
  4. 2023Applications of Large Scale Foundation Models for Autonomous Drivingcited by 14
  5. 2025Trajectory Prediction Meets Large Language Models: A Surveycited by 13
  6. 2024SemanticFormer: Holistic and Semantic Traffic Scene Representation for Trajectory Prediction using Knowledge Graphscited by 10
  7. 2024Generating and Evolving Reward Functions for Highway Driving with Large Language Modelscited by 8
  8. 2023Prospective Role of Foundation Models in Advancing Autonomous Vehiclescited by 7
  9. 2024Hard Cases Detection in Motion Prediction by Vision-Language Foundation Modelscited by 4
  10. 2024A Survey on Large Language Model-empowered Autonomous Drivingcited by 3
  11. 2025Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Surveycited by 3
  12. 2024A Comprehensive Review of 3D Object Detection in Autonomous Driving: Technological Advances and Future Directionscited by 2
  13. 2025A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspectivecited by 2
  14. 2025Towards Physics-informed Spatial Intelligence with Human Priors: An Autonomous Driving Pilot Studycited by 2
  15. 2024Pedestrian motion prediction evaluation for urban autonomous drivingcited by 1
  16. 2025TRIDE: A Text-assisted Radar-Image weather-aware fusion network for Depth Estimationcited by 1
  17. 2025VLMs Guided Interpretable Decision Making for Autonomous Drivingcited by 1
  18. 2025A Review of Learning-Based Motion Planning: Toward a Data-Driven Optimal Control Approachcited by 1