# Neuro-Cognitive Reward Modeling for Human-Centered Autonomous Vehicle Control

arXiv: https://arxiv.org/abs/2603.25968 · HTML: https://mlsanity.com/paper/2603.25968/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning VLA and VLM driving policies, one of 9 directions in End-to-end autonomous driving.
- Builds on 34 of its 67 references that are in the map.
- First posted on arXiv 26 Mar 2026.

## Placed in

- End-to-end autonomous driving / [Reasoning VLA and VLM driving policies](https://mlsanity.com/driving/reasoning-vla-and-vlm-driving-policies/)
- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving](https://mlsanity.com/paper/2406.03877/) (2024)
- [Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline](https://mlsanity.com/paper/2206.08129/) (2022)
- [Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer](https://mlsanity.com/paper/2207.14024/) (2022)
- [End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordances](https://mlsanity.com/paper/1911.10868/) (2019)
- [Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmark](https://mlsanity.com/paper/2310.12567/) (2023)
- [Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?](https://mlsanity.com/paper/2006.14911/) (2020)
- [Hidden Biases of End-to-End Driving Models](https://mlsanity.com/paper/2306.07957/) (2023)
- [ReasonNet: End-to-End Driving with Temporal and Global Reasoning](https://mlsanity.com/paper/2305.10507/) (2023)
- [Predicting Driver Attention in Critical Situations](https://mlsanity.com/paper/1711.06406/) (2017)
- [Imitation Learning: Progress, Taxonomies and Challenges](https://mlsanity.com/paper/2106.12177/) (2021)
- [DADA-2000: Can Driving Accident be Predicted by Driver Attention? Analyzed by A Benchmark](https://mlsanity.com/paper/1904.12634/) (2019)
- [ICLabel: An automated electroencephalographic independent component classifier, dataset, and website](https://mlsanity.com/paper/1901.07915/) (2019)
- [Multi-channel EEG recordings during a sustained-attention driving task](https://mlsanity.com/paper/1809.06534/) (2018)
- [Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching](https://mlsanity.com/paper/2303.02569/) (2023)
- [Prioritized Experience-based Reinforcement Learning with Human Guidance for Autonomous Driving](https://mlsanity.com/paper/2109.12516/) (2021)
- [Brain Decodes Deep Nets](https://mlsanity.com/paper/2312.01280/) (2023)
- [Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Models](https://mlsanity.com/paper/2410.01532/) (2024)
- [MAAD: A Model and Dataset for "Attended Awareness" in Driving](https://mlsanity.com/paper/2110.08610/) (2021)

## Built on by (most cited first, 3 of 3)

- [Post-Training in End-to-End Autonomous Driving](https://mlsanity.com/paper/2607.08072/) (2026)
- [Learning Dynamic Neural Evidence Representations for Time-Adaptive Brain-Computer Interfaces](https://mlsanity.com/paper/2609.22088/) (2026)
- [Trajectory Prediction for Autonomous Driving in Road Construction Zones Using Human-in-the-Loop Driving Data](https://mlsanity.com/paper/2610.12747/) (2026)

## Abstract (arXiv)

Recent advancements in computer vision have accelerated the development of autonomous driving. Despite these advancements, training machines to drive in a way that aligns with human expectations remains a significant challenge. Human factors are still essential, as humans possess a sophisticated cognitive system capable of rapidly interpreting scene information and making accurate decisions. Aligning machine with human intent has been explored with Reinforcement Learning with Human Feedback (RLHF). Conventional RLHF methods rely on collecting human preference data by manually ranking generated outputs, which is time-consuming and indirect. In this work, we propose an electroencephalography (EEG)-guided decision-making framework to incorporate human cognitive insights without behaviour response interruption into reinforcement learning (RL) for autonomous driving. We collected EEG signals from 20 participants in a realistic driving simulator and analyzed event-related potentials (ERP) in response to sudden environmental changes. Our proposed framework employs a neural network to predict the strength of ERP based on the cognitive information from visual scene information. Moreover, we explore the integration of such cognitive information into the reward signal of the RL algorithm. Experimental results show that our framework can improve the collision avoidance ability of the RL algorithm, highlighting the potential of neuro-cognitive feedback in enhancing autonomous driving systems. Our project page is: https://alex95gogo.github.io/Cognitive-Reward/.