# Through Their Eyes: Fixation-aligned Tuning for Personalized User Emulation

arXiv: https://arxiv.org/abs/2604.09368 · HTML: https://mlsanity.com/paper/2604.09368/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 43 references that are in the map.
- First posted on arXiv 10 Apr 2026.

## Builds on (most cited first, 23 of 23)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [A Survey on the Memory Mechanism of Large Language Model based Agents](https://mlsanity.com/paper/2404.13501/) (2024)
- [AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers](https://mlsanity.com/paper/2402.05602/) (2024)
- [MAI-UI Technical Report: Real-World Centric Foundation GUI Agents](https://mlsanity.com/paper/2512.22047/) (2025)
- [Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training](https://mlsanity.com/paper/2403.02325/) (2024)
- [A Survey on LLM-powered Agents for Recommender Systems](https://mlsanity.com/paper/2502.10050/) (2025)
- [LLM-Powered User Simulator for Recommender System](https://mlsanity.com/paper/2412.16984/) (2024)
- [SimUSER: Simulating User Behavior with Large Language Models for Recommender System Evaluation](https://mlsanity.com/paper/2504.12722/) (2025)
- [Voila-A: Aligning Vision-Language Models with User's Gaze Attention](https://mlsanity.com/paper/2401.09454/) (2023)
- [Personalized Saliency and its Prediction](https://mlsanity.com/paper/1710.03011/) (2017)
- [Gaze-VLM:Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding](https://mlsanity.com/paper/2510.21356/) (2025)
- [The Eye of Sherlock Holmes: Uncovering User Private Attribute Profiling via Vision-Language Model Agentic Framework](https://mlsanity.com/paper/2505.19139/) (2025)
- [Diagnostic-Guided Dynamic Profile Optimization for LLM-based User Simulators in Sequential Recommendation](https://mlsanity.com/paper/2508.12645/) (2025)
- [GLIMPSE: Holistic Cross-Modal Explainability for Large Vision-Language Models](https://mlsanity.com/paper/2506.18985/) (2025)
- [Learning User Embeddings from Human Gaze for Personalised Saliency Prediction](https://mlsanity.com/paper/2403.13653/) (2024)
- [CIRP: Cross-Item Relational Pre-training for Multimodal Product Bundling](https://mlsanity.com/paper/2404.01735/) (2024)
- [Generating Negative Samples for Multi-Modal Recommendation](https://mlsanity.com/paper/2501.15183/) (2025)
- [RecGaze: The First Eye Tracking and User Interaction Dataset for Carousel Interfaces](https://mlsanity.com/paper/2504.20792/) (2025)
- [Improving Visual Recommendation on E-commerce Platforms Using Vision-Language Models](https://mlsanity.com/paper/2510.13359/) (2025)
- [Weighted KL-Divergence for Document Ranking Model Refinement](https://mlsanity.com/paper/2406.05977/) (2024)
- [A Versatile Dataset of Mouse and Eye Movements on Search Engine Results Pages](https://mlsanity.com/paper/2507.08003/) (2025)

## Abstract (arXiv)

Large language model (LLM) agents are increasingly deployed as scalable user simulators for recommender system evaluation. Yet existing simulators perceive recommendations through text or structured metadata rather than the visual interfaces real users browse-a critical gap, since attention over recommendation layouts is both visually driven and highly personalized. We investigate whether aligning a vision-language model's (VLM's) visual attention with user-specific gaze patterns can improve simulation fidelity. Analysis of a real-world eye-tracking dataset collected in a carousel-based recommendation setting reveals that users exhibit stable individual gaze patterns strongly predictive of click behavior. Building on this finding, we propose Fixation-Aligned Tuning for user Emulation (FixATE). Our approach first probes the VLM's internal visual attention via interpretability operators to obtain a slot-level relevance distribution comparable with human fixation, and then learns personalized soft prompts to steer the model's attention toward each user's characteristic fixation pattern. Experiments across three interpretability-based probing operators and two architecturally distinct VLM backbones demonstrate consistent improvements in both attention alignment and click prediction accuracy. These results suggest that making the model "see like the user" is a viable path toward simulators that more faithfully reproduce how users perceive and act in recommendation interfaces.