# LLM-Driven Autonomous Vehicles Inherit Human Driver Biases in Pedestrian Yielding: Results and Implications From A New Benchmark

arXiv: https://arxiv.org/abs/2609.00192 · HTML: https://mlsanity.com/paper/2609.00192/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in VLMs for autonomous driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 24 of its 39 references that are in the map.
- First posted on arXiv 31 Aug 2026.

## Placed in

- End-to-end autonomous driving / [VLMs for autonomous driving](https://mlsanity.com/driving/vlms-for-autonomous-driving/)

## Builds on (most cited first, 24 of 24)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [DriveLM: Driving with Graph Visual Question Answering](https://mlsanity.com/paper/2312.14150/) (2023)
- [Toxicity in ChatGPT: Analyzing Persona-assigned Language Models](https://mlsanity.com/paper/2304.05335/) (2023)
- [SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models](https://mlsanity.com/paper/2311.07575/) (2023)
- [CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIP](https://mlsanity.com/paper/2301.04926/) (2023)
- [Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Driving](https://mlsanity.com/paper/2410.22313/) (2024)
- [Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challenges](https://mlsanity.com/paper/2311.03287/) (2023)
- [The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models](https://mlsanity.com/paper/2601.10387/) (2026)
- [Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving](https://mlsanity.com/paper/2310.02251/) (2023)
- [Current state of LLM Risks and AI Guardrails](https://mlsanity.com/paper/2406.12934/) (2024)
- [Joint Attention in Autonomous Driving (JAAD)](https://mlsanity.com/paper/1609.04741/) (2016)
- [Agreeing to Cross: How Drivers and Pedestrians Communicate](https://mlsanity.com/paper/1702.03555/) (2017)
- [Benchmarking Adversarial Robustness to Bias Elicitation in Large Language Models: Scalable Automated Assessment with LLM-as-a-Judge](https://mlsanity.com/paper/2504.07887/) (2025)
- [LLM-Driven Robots Risk Enacting Discrimination, Violence, and Unlawful Actions](https://mlsanity.com/paper/2406.08824/) (2024)
- [Age and gender bias in pedestrian detection algorithms](https://mlsanity.com/paper/1906.10490/) (2019)
- [Robots Enact Malignant Stereotypes](https://mlsanity.com/paper/2207.11569/) (2022)
- [Causal Mode Multiplexer: A Novel Framework for Unbiased Multispectral Pedestrian Detection](https://mlsanity.com/paper/2403.01300/) (2024)
- [A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models](https://mlsanity.com/paper/2402.13636/) (2024)

## Abstract (arXiv)

Public trust in Autonomous Vehicles (AVs) may depend not only on technical success but also on the fairness of their decision making. While a recent trend in AV research involves using general purpose "common sense" models to guide AV decision making, the degree to which these inherit human biases in driving is still understudied. Given that psychology studies have shown human driver biases exist, such as lower pedestrian-yielding rates to Black pedestrians in the US, we argue that analyses of model bias should also be part of AV evaluation. Concretely, in this paper we propose two new bias testing methodologies for Large Language Models (LLMs) and Visual-Language Models (VLMs)-"All Else Being Equal" tests and "Self-Consistency" tests-in order to assess bias in pedestrian-yielding decisions. Our findings show that both LLMs and VLMs make yielding decisions which are influenced by pedestrian gender, ethnicity, religion, disability, age, skin tone and socio-economic status. While the type and degree of bias is different from model to model, we highlight common patterns-and raise questions about the "common sense" model paradigm, particularly the need to either revise the paradigm or address issues of downstream bias.