# Optimizing Visual Question Answering Models for Driving: Bridging the Gap Between Human and Machine Attention Patterns

arXiv: https://arxiv.org/abs/2406.09203 · HTML: https://mlsanity.com/paper/2406.09203/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 16 references that are in the map.
- First posted on arXiv 13 Jun 2024.

## Builds on (most cited first, 8 of 8)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Patch Slimming for Efficient Vision Transformers](https://mlsanity.com/paper/2106.02852/) (2021)
- [NuScenes-MQA: Integrated Evaluation of Captions and QA for Autonomous Driving Datasets using Markup Annotations](https://mlsanity.com/paper/2312.06352/) (2023)
- [How Transferable are Reasoning Patterns in VQA?](https://mlsanity.com/paper/2104.03656/) (2021)
- [Prompting Vision Language Model with Knowledge from Large Language Model for Knowledge-Based VQA](https://mlsanity.com/paper/2308.15851/) (2023)
- [Towards a performance analysis on pre-trained Visual Question Answering models for autonomous driving](https://mlsanity.com/paper/2307.09329/) (2023)

## Built on by (most cited first, 4 of 4)

- [ReasonDrive: Efficient Visual Question Answering for Autonomous Vehicles with Reasoning-Enhanced Small Vision-Language Models](https://mlsanity.com/paper/2504.10757/) (2025)
- [Large Language Models and Their Applications in Roadway Safety and Mobility Enhancement: A Comprehensive Review](https://mlsanity.com/paper/2506.06301/) (2025)
- [Robusto-1 Dataset: Comparing Humans and VLMs on real out-of-distribution Autonomous Driving VQA from Peru](https://mlsanity.com/paper/2503.07587/) (2025)
- [Efficient Visual Question Answering Pipeline for Autonomous Driving via Scene Region Compression](https://mlsanity.com/paper/2601.07092/) (2026)

## Abstract (arXiv)

Visual Question Answering (VQA) models play a critical role in enhancing the perception capabilities of autonomous driving systems by allowing vehicles to analyze visual inputs alongside textual queries, fostering natural interaction and trust between the vehicle and its occupants or other road users. This study investigates the attention patterns of humans compared to a VQA model when answering driving-related questions, revealing disparities in the objects observed. We propose an approach integrating filters to optimize the model's attention mechanisms, prioritizing relevant objects and improving accuracy. Utilizing the LXMERT model for a case study, we compare attention patterns of the pre-trained and Filter Integrated models, alongside human answers using images from the NuImages dataset, gaining insights into feature prioritization. We evaluated the models using a Subjective scoring framework which shows that the integration of the feature encoder filter has enhanced the performance of the VQA model by refining its attention mechanisms.