# Tapping in a Remote Vehicle's onboard LLM to Complement the Ego Vehicle's Field-of-View

arXiv: https://arxiv.org/abs/2408.10794 · HTML: https://mlsanity.com/paper/2408.10794/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 22 references that are in the map.
- First posted on arXiv 20 Aug 2024.

## Builds on (most cited first, 9 of 9)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [The VIA Annotation Software for Images, Audio and Video](https://mlsanity.com/paper/1904.10699/) (2019)
- [Large Language Models Understand and Can be Enhanced by Emotional Stimuli](https://mlsanity.com/paper/2307.11760/) (2023)
- [Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigation](https://mlsanity.com/paper/2303.03480/) (2023)
- [Penetrative AI: Making LLMs Comprehend the Physical World](https://mlsanity.com/paper/2310.09605/) (2023)
- [ChatGPT as a tool for User Story Quality Evaluation: Trustworthy Out of the Box?](https://mlsanity.com/paper/2306.12132/) (2023)
- [An LLM-Based Digital Twin for Optimizing Human-in-the Loop Systems](https://mlsanity.com/paper/2403.16809/) (2024)
- [CarExpert: Leveraging Large Language Models for In-Car Conversational Question Answering](https://mlsanity.com/paper/2310.09536/) (2023)

## Built on by (most cited first, 1 of 1)

- [Multi-Agent Autonomous Driving Systems with Large Language Models: A Survey of Recent Advances](https://mlsanity.com/paper/2502.16804/) (2025)

## Abstract (arXiv)

Today's advanced automotive systems are turning into intelligent Cyber-Physical Systems (CPS), bringing computational intelligence to their cyber-physical context. Such systems power advanced driver assistance systems (ADAS) that observe a vehicle's surroundings for their functionality. However, such ADAS have clear limitations in scenarios when the direct line-of-sight to surrounding objects is occluded, like in urban areas. Imagine now automated driving (AD) systems that ideally could benefit from other vehicles' field-of-view in such occluded situations to increase traffic safety if, for example, locations about pedestrians can be shared across vehicles. Current literature suggests vehicle-to-infrastructure (V2I) via roadside units (RSUs) or vehicle-to-vehicle (V2V) communication to address such issues that stream sensor or object data between vehicles. When considering the ongoing revolution in vehicle system architectures towards powerful, centralized processing units with hardware accelerators, foreseeing the onboard presence of large language models (LLMs) to improve the passengers' comfort when using voice assistants becomes a reality. We are suggesting and evaluating a concept to complement the ego vehicle's field-of-view (FOV) with another vehicle's FOV by tapping into their onboard LLM to let the machines have a dialogue about what the other vehicle ``sees''. Our results show that very recent versions of LLMs, such as GPT-4V and GPT-4o, understand a traffic situation to an impressive level of detail, and hence, they can be used even to spot traffic participants. However, better prompts are needed to improve the detection quality and future work is needed towards a standardised message interchange format between vehicles.