# PerSRV: Personalized Sticker Retrieval with Vision-Language Model

arXiv: https://arxiv.org/abs/2410.21801 · HTML: https://mlsanity.com/paper/2410.21801/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 17 of its 48 references that are in the map.
- First posted on arXiv 29 Oct 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/)

## Builds on (most cited first, 17 of 17)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Generating Images with Multimodal Language Models](https://mlsanity.com/paper/2305.17216/) (2023)
- [PP-OCR: A Practical Ultra Lightweight OCR System](https://mlsanity.com/paper/2009.09941/) (2020)
- [Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese](https://mlsanity.com/paper/2211.01335/) (2022)
- [A Survey of Deep Learning Approaches for OCR and Document Understanding](https://mlsanity.com/paper/2011.13534/) (2020)
- [STICKERCONV: Generating Multimodal Empathetic Responses from Scratch](https://mlsanity.com/paper/2402.01679/) (2024)
- [Zero-Shot Learners for Natural Language Understanding via a Unified Multiple Choice Perspective](https://mlsanity.com/paper/2210.08590/) (2022)
- [Learning to Respond with Stickers: A Framework of Unifying Multi-Modality in Multi-Turn Dialog](https://mlsanity.com/paper/2003.04679/) (2020)
- [Understanding Chat Messages for Sticker Recommendation in Messaging Apps](https://mlsanity.com/paper/1902.02704/) (2019)
- [Sticker820K: Empowering Interactive Retrieval with Stickers](https://mlsanity.com/paper/2306.06870/) (2023)
- [EmojiLM: Modeling the New Emoji Language](https://mlsanity.com/paper/2311.01751/) (2023)
- [Learning to Respond with Your Favorite Stickers: A Framework of Unifying Multi-Modality and User Preference in Multi-Turn Dialog](https://mlsanity.com/paper/2011.03322/) (2020)
- [Reply with Sticker: New Dataset and Model for Sticker Retrieval](https://mlsanity.com/paper/2403.05427/) (2024)
- [Towards Real-World Stickers Use: A New Dataset for Multi-Tag Sticker Recognition](https://mlsanity.com/paper/2403.05428/) (2024)
- [Reciprocity, Homophily, and Social Network Effects in Pictorial Communication: A Case Study of Bitmoji Stickers](https://mlsanity.com/paper/2211.01426/) (2022)

## Built on by (most cited first, 1 of 1)

- [GSE: Evaluating Sticker Visual Semantic Similarity via a General Sticker Encoder](https://mlsanity.com/paper/2511.04977/) (2025)

## Abstract (arXiv)

Instant Messaging is a popular means for daily communication, allowing users to send text and stickers. As the saying goes, "a picture is worth a thousand words", so developing an effective sticker retrieval technique is crucial for enhancing user experience. However, existing sticker retrieval methods rely on labeled data to interpret stickers, and general-purpose Vision-Language Models (VLMs) often struggle to capture the unique semantics of stickers. Additionally, relevant-based sticker retrieval methods lack personalization, creating a gap between diverse user expectations and retrieval results. To address these, we propose the Personalized Sticker Retrieval with Vision-Language Model framework, namely PerSRV, structured into offline calculations and online processing modules. The online retrieval part follows the paradigm of relevant recall and personalized ranking, supported by the offline pre-calculation parts, which are sticker semantic understanding, utility evaluation and personalization modules. Firstly, for sticker-level semantic understanding, we supervised fine-tuned LLaVA-1.5-7B to generate human-like sticker semantics, complemented by textual content extracted from figures and historical interaction queries. Secondly, we investigate three crowd-sourcing metrics for sticker utility evaluation. Thirdly, we cluster style centroids based on users' historical interactions to achieve personal preference modeling. Finally, we evaluate our proposed PerSRV method on a public sticker retrieval dataset from WeChat, containing 543,098 candidates and 12,568 interactions. Experimental results show that PerSRV significantly outperforms existing methods in multi-modal sticker retrieval. Additionally, our fine-tuned VLM delivers notable improvements in sticker semantic understandings.