# PinnerFormer: Sequence Modeling for User Representation at Pinterest

arXiv: https://arxiv.org/abs/2205.04507 · HTML: https://mlsanity.com/paper/2205.04507/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 32 references that are in the map.
- First posted on arXiv 9 May 2022.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [Graph Convolutional Neural Networks for Web-Scale Recommender Systems](https://mlsanity.com/paper/1806.01973/) (2018)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [Deep Interest Evolution Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1809.03672/) (2018)
- [Learning Deep Transformer Models for Machine Translation](https://mlsanity.com/paper/1906.01787/) (2019)
- [Time2Vec: Learning a Vector Representation of Time](https://mlsanity.com/paper/1907.05321/) (2019)
- [Top-K Off-Policy Correction for a REINFORCE Recommender System](https://mlsanity.com/paper/1812.02353/) (2018)
- [Search-based User Interest Modeling with Lifelong Sequential Behavior Data for Click-Through Rate Prediction](https://mlsanity.com/paper/2006.05639/) (2020)
- [Recurrent Neural Networks with Top-k Gains for Session-based Recommendations](https://mlsanity.com/paper/1706.03847/) (2017)
- [Transformers without Tears: Improving the Normalization of Self-Attention](https://mlsanity.com/paper/1910.05895/) (2019)
- [Behavior Sequence Transformer for E-commerce Recommendation in Alibaba](https://mlsanity.com/paper/1905.06874/) (2019)
- [A Simple Convolutional Generative Network for Next Item Recommendation](https://mlsanity.com/paper/1808.05163/) (2018)
- [Practice on Long Sequential User Behavior Modeling for Click-Through Rate Prediction](https://mlsanity.com/paper/1905.09248/) (2019)
- [Multi-Interest Network with Dynamic Routing for Recommendation at Tmall](https://mlsanity.com/paper/1904.08030/) (2019)
- [Towards Personalized and Semantic Retrieval: An End-to-End Solution for E-commerce Search via Embedding Learning](https://mlsanity.com/paper/2006.02282/) (2020)
- [Is a Single Vector Enough? Exploring Node Polysemy for Network Embedding](https://mlsanity.com/paper/1905.10668/) (2019)

## Built on by (most cited first, 30 of 57)

- [Better Generalization with Semantic IDs: A Case Study in Ranking for Recommendations](https://mlsanity.com/paper/2306.08121/) (2023)
- [TransAct: Transformer-based Realtime User Action Model for Recommendation at Pinterest](https://mlsanity.com/paper/2306.00248/) (2023)
- [Sparse Meets Dense: Unified Generative Recommendations with Cascaded Sparse-Dense Representations](https://mlsanity.com/paper/2503.02453/) (2025)
- [LiGNN: Graph Neural Networks at LinkedIn](https://mlsanity.com/paper/2402.11139/) (2024)
- [Large Language Models for User Interest Journeys](https://mlsanity.com/paper/2305.15498/) (2023)
- [Scaling User Modeling: Large-scale Online User Representations for Ads Personalization in Meta](https://mlsanity.com/paper/2311.09544/) (2023)
- [LEARN: Knowledge Adaptation from Large Language Model to Recommendation for Practical Industrial Application](https://mlsanity.com/paper/2405.03988/) (2024)
- [Rethinking Personalized Ranking at Pinterest: An End-to-End Approach](https://mlsanity.com/paper/2209.08435/) (2022)
- [LiRank: Industrial Large Scale Ranking Models at LinkedIn](https://mlsanity.com/paper/2402.06859/) (2024)
- [Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders](https://mlsanity.com/paper/2507.16289/) (2025)
- [MMQ: Multimodal Mixture-of-Quantization Tokenization for Semantic ID Generation and User Behavioral Adaptation](https://mlsanity.com/paper/2508.15281/) (2025)
- [Massive Memorization with Hundreds of Trillions of Parameters for Sequential Transducer Generative Recommenders](https://mlsanity.com/paper/2510.22049/) (2025)
- [PinRec: Unified Generative Retrieval for Pinterest Recommender Systems](https://mlsanity.com/paper/2504.10507/) (2025)
- [DV365: Extremely Long User History Modeling at Instagram](https://mlsanity.com/paper/2506.00450/) (2025)
- [PinFM: Foundation Model for User Activity Sequences at a Billion-scale Visual Discovery Platform](https://mlsanity.com/paper/2507.12704/) (2025)
- [KuaiFormer: Transformer-Based Retrieval at Kuaishou](https://mlsanity.com/paper/2411.10057/) (2024)
- [Request-Only Optimization for Recommendation Systems](https://mlsanity.com/paper/2508.05640/) (2025)
- [LLaTTE: Scaling Laws for Multi-Stage Sequence Modeling in Large-Scale Ads Recommendation](https://mlsanity.com/paper/2601.20083/) (2026)
- [adSformers: Personalization from Short-Term Sequences and Diversity of Representations in Etsy Ads](https://mlsanity.com/paper/2302.01255/) (2023)
- [OmniSage: Large Scale, Multi-Entity Heterogeneous Graph Representation Learning](https://mlsanity.com/paper/2504.17811/) (2025)
- [FinTRec: Transformer Based Unified Contextual Ads Targeting and Personalization for Financial Applications](https://mlsanity.com/paper/2511.14865/) (2025)
- [Denoising Pre-Training and Customized Prompt Learning for Efficient Multi-Behavior Sequential Recommendation](https://mlsanity.com/paper/2408.11372/) (2024)
- [Building a Scalable, Effective, and Steerable Search and Ranking Platform](https://mlsanity.com/paper/2409.02856/) (2024)
- [TRACE: Transformer-based user Representations from Attributed Clickstream Event sequences](https://mlsanity.com/paper/2409.12972/) (2024)
- [Scaling Laws for Online Advertisement Retrieval](https://mlsanity.com/paper/2411.13322/) (2024)
- [On the Practice of Deep Hierarchical Ensemble Network for Ad Conversion Rate Prediction](https://mlsanity.com/paper/2504.08169/) (2025)
- [Revisiting Self-attention for Cross-domain Sequential Recommendation](https://mlsanity.com/paper/2505.21811/) (2025)
- [Rethinking E-Commerce Search](https://mlsanity.com/paper/2312.03217/) (2023)
- [InteractRank: Personalized Web-Scale Search Pre-Ranking with Cross Interaction Features](https://mlsanity.com/paper/2504.06609/) (2025)
- [Describe What You See with Multimodal Large Language Models to Enhance Video Recommendations](https://mlsanity.com/paper/2508.09789/) (2025)

## Abstract (arXiv)

Sequential models have become increasingly popular in powering personalized recommendation systems over the past several years. These approaches traditionally model a user's actions on a website as a sequence to predict the user's next action. While theoretically simplistic, these models are quite challenging to deploy in production, commonly requiring streaming infrastructure to reflect the latest user activity and potentially managing mutable data for encoding a user's hidden state. Here we introduce PinnerFormer, a user representation trained to predict a user's future long-term engagement using a sequential model of a user's recent actions. Unlike prior approaches, we adapt our modeling to a batch infrastructure via our new dense all-action loss, modeling long-term future actions instead of next action prediction. We show that by doing so, we significantly close the gap between batch user embeddings that are generated once a day and realtime user embeddings generated whenever a user takes an action. We describe our design decisions via extensive offline experimentation and ablations and validate the efficacy of our approach in A/B experiments showing substantial improvements in Pinterest's user retention and engagement when comparing PinnerFormer against our previous user representation. PinnerFormer is deployed in production as of Fall 2021.