See2Refine: Vision-Language Feedback Improves LLM-Based eHMI Action Designers

arXiv 2602.02063 ↗First posted 2 Feb 2026cs.HC cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Automated vehicles lack natural communication channels with other road users, making external Human-Machine Interfaces (eHMIs) essential for conveying intent and maintaining trust in shared environments. However, most eHMI studies rely on developer-crafted message-action pairs, which are difficult to adapt to diverse and dynamic traffic contexts. A promising alternative is to use Large Language Models (LLMs) as action designers that generate context-conditioned eHMI actions, yet such designers lack perceptual verification and typically depend on fixed prompts or costly human-annotated feedback for improvement. We present See2Refine, a human-free, closed-loop framework that uses vision-language model (VLM) perceptual evaluation as automated visual feedback to improve an LLM-based eHMI action designer. Given a driving context and a candidate eHMI action, the VLM evaluates the perceived appropriateness of the action, and this feedback is used to iteratively revise the designer's outputs, enabling systematic refinement without human supervision. We evaluate our framework across three eHMI modalities (lightbar, eyes, and arm) and multiple LLM model sizes. Across settings, our framework consistently outperforms prompt-only LLM designers and manually specified baselines in both VLM-based metrics and human-subject evaluations. Results further indicate that the improvements generalize across modalities and that VLM evaluations are well aligned with human preferences, supporting the robustness and effectiveness of See2Refine for scalable action design.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 15 of its 57 references that are in the map. First posted on arXiv 2 Feb 2026.

Builds on

15 of 57 references.

Closest ancestors · 1

Same direction, newest first

2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May

Other references · 14

2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 299
2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 123
2024WildVision: Evaluating Vision-Language Models in the Wild with Human Preferencescited by 76
2024Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedbackcited by 36
2024Exploring the Capabilities and Limitations of Large Language Models in the Electric Energy Sectorcited by 29
2024BlenderLLM: Training Large Language Models for Computer-Aided Design with Self-improvementcited by 26

2 more

2023Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Languagecited by 22
2025Automating eHMI Action Design with LLMs for Automated Vehicle Communicationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way