# Personalized Image Descriptions from Attention Sequences

arXiv: https://arxiv.org/abs/2512.06662 · HTML: https://mlsanity.com/paper/2512.06662/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 67 references that are in the map.
- First posted on arXiv 7 Dec 2025.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Connecting Vision and Language with Localized Narratives](https://mlsanity.com/paper/1912.03098/) (2019)
- [Caption Anything: Interactive Image Description with Diverse Multimodal Controls](https://mlsanity.com/paper/2305.02677/) (2023)
- [Yo'LLaVA: Your Personalized Language and Vision Assistant](https://mlsanity.com/paper/2406.09400/) (2024)
- [MyVLM: Personalizing VLMs for User-Specific Queries](https://mlsanity.com/paper/2403.14599/) (2024)
- [Engaging Image Captioning Via Personality](https://mlsanity.com/paper/1810.10665/) (2018)
- [Polos: Multimodal Metric Learning from Human Feedback for Image Captioning](https://mlsanity.com/paper/2402.18091/) (2024)
- [Modeling Caption Diversity in Contrastive Vision-Language Pretraining](https://mlsanity.com/paper/2405.00740/) (2024)
- [Attend to You: Personalized Image Captioning with Context Sequence Memory Networks](https://mlsanity.com/paper/1704.06485/) (2017)
- [Towards Knowledge-Based Personalized Product Description Generation in E-commerce](https://mlsanity.com/paper/1903.12457/) (2019)
- [Personalized Visual Instruction Tuning](https://mlsanity.com/paper/2410.07113/) (2024)
- [RAP: Retrieval-Augmented Personalization for Multimodal Large Language Models](https://mlsanity.com/paper/2410.13360/) (2024)
- [Diverse Image Captioning with Context-Object Split Latent Spaces](https://mlsanity.com/paper/2011.00966/) (2020)
- [Unifying Top-down and Bottom-up Scanpath Prediction Using Transformers](https://mlsanity.com/paper/2303.09383/) (2023)
- [YoChameleon: Personalized Vision and Language Generation](https://mlsanity.com/paper/2504.20998/) (2025)
- [Generating Image Descriptions via Sequential Cross-Modal Alignment Guided by Human Gaze](https://mlsanity.com/paper/2011.04592/) (2020)
- [Meta-Personalizing Vision-Language Models to Find Named Instances in Video](https://mlsanity.com/paper/2306.10169/) (2023)
- [Connecting What to Say With Where to Look by Modeling Human Attention Traces](https://mlsanity.com/paper/2105.05964/) (2021)
- [Beyond Average: Individualized Visual Scanpath Prediction](https://mlsanity.com/paper/2404.12235/) (2024)
- ["It's Kind of Context Dependent": Understanding Blind and Low Vision People's Video Accessibility Preferences Across Viewing Scenarios](https://mlsanity.com/paper/2403.10792/) (2024)

## Built on by (most cited first, 2 of 2)

- [Pathologist Attention-Aligned Report Generation for Prostate Histopathology](https://mlsanity.com/paper/2607.19624/) (2026)
- [Gaze-to-text Generation: Beyond Categorical Decoding of Human Attention](https://mlsanity.com/paper/2607.23917/) (2026)

## Abstract (arXiv)

People can view the same image differently: they focus on different regions, objects, and details in varying orders and describe them in distinct linguistic styles. This leads to substantial variability in image descriptions. However, existing models for personalized image description focus on linguistic style alone, with no prior work leveraging individual viewing patterns. We address this gap by explicitly modeling personalized viewing behavior as a core factor in description generation. Our method, DEPER (DEscription-PERception persona encoder), learns a subject embedding that captures both linguistic style and viewing behavior, guided by an auxiliary attention-prediction task. A lightweight adapter aligns these embeddings with a frozen vision-language model, enabling few-shot personalization without retraining. Across four datasets spanning diverse viewing tasks and both short and detailed descriptions, DEPER achieves a 24% average improvement, showing that modeling personalized attention produces more human-aligned and high-quality descriptions. We posit that understanding how people see helps predict what they say; modeling human diversity in perception can improve both performance and human alignment in multimodal systems.