# PinFM: Foundation Model for User Activity Sequences at a Billion-scale Visual Discovery Platform

arXiv: https://arxiv.org/abs/2507.12704 · HTML: https://mlsanity.com/paper/2507.12704/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 17 Jul 2025.

## Builds on (most cited first, 22 of 22)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [Graph Convolutional Neural Networks for Web-Scale Recommender Systems](https://mlsanity.com/paper/1806.01973/) (2018)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://mlsanity.com/paper/2008.13535/) (2020)
- [Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations](https://mlsanity.com/paper/2402.17152/) (2024)
- [M6-Rec: Generative Pretrained Language Models are Open-Ended Recommender Systems](https://mlsanity.com/paper/2205.08084/) (2022)
- [Text Is All You Need: Learning Language Representations for Sequential Recommendation](https://mlsanity.com/paper/2305.13731/) (2023)
- [Towards Universal Sequence Representation Learning for Recommender Systems](https://mlsanity.com/paper/2206.05941/) (2022)
- [PinnerFormer: Sequence Modeling for User Representation at Pinterest](https://mlsanity.com/paper/2205.04507/) (2022)
- [FBGEMM: Enabling High-Performance Low-Precision Deep Learning Inference](https://mlsanity.com/paper/2101.05615/) (2021)
- [TWIN: TWo-stage Interest Network for Lifelong User Behavior Modeling in CTR Prediction at Kuaishou](https://mlsanity.com/paper/2302.02352/) (2023)
- [TransAct: Transformer-based Realtime User Action Model for Recommendation at Pinterest](https://mlsanity.com/paper/2306.00248/) (2023)
- [Scaling Law of Large Sequential Recommendation Models](https://mlsanity.com/paper/2311.11351/) (2023)
- [PTUM: Pre-training User Model from Unlabeled User Behaviors via Self-supervision](https://mlsanity.com/paper/2010.01494/) (2020)
- [Multi-Behavior Generative Recommendation](https://mlsanity.com/paper/2405.16871/) (2024)
- [External Large Foundation Model: How to Efficiently Serve Trillions of Parameters for Online Ads Recommendation](https://mlsanity.com/paper/2502.17494/) (2025)
- [360Brew: A Decoder-only Foundation Model for Personalized Ranking and Recommendation](https://mlsanity.com/paper/2501.16450/) (2025)
- [SessionRec: Next Session Prediction Paradigm For Generative Sequential Recommendation](https://mlsanity.com/paper/2502.10157/) (2025)

## Built on by (most cited first, 8 of 8)

- [PLUM: Adapting Pre-trained Language Models for Industrial-scale Generative Recommendations](https://mlsanity.com/paper/2510.07784/) (2025)
- [DenseRec: Revisiting Dense Content Embeddings for Sequential Transformer-based Recommendation](https://mlsanity.com/paper/2508.18442/) (2025)
- [LocalSUG: City-Preference-Enhanced LLM for Query Suggestion in Local-Life Services](https://mlsanity.com/paper/2603.04946/) (2026)
- [UniPinRec: Unifying Generative Retrieval and Ranking at Pinterest Scale](https://mlsanity.com/paper/2606.00422/) (2026)
- [RankGraph-2: Lifecycle Co-Design for Billion-Node Graph Learning in Recommendation](https://mlsanity.com/paper/2606.18379/) (2026)
- [A Foundation Model for Multimodal Event Sequences in Financial Applications](https://mlsanity.com/paper/2607.09955/) (2026)
- [Building a User Foundation Model for the Open Web](https://mlsanity.com/paper/2607.28019/) (2026)
- [Progressive Alignment of Recommender Foundation Model through Multi-Phase Post-Training](https://mlsanity.com/paper/2608.06792/) (2026)

## Abstract (arXiv)

User activity sequences have emerged as one of the most important signals in recommender systems. We present a foundational model, PinFM, for understanding user activity sequences across multiple applications at a billion-scale visual discovery platform. We pretrain a transformer model with 20B+ parameters using extensive user activity data, then fine-tune it for specific applications, efficiently coupling it with existing models. While this pretraining-and-fine-tuning approach has been popular in other domains, such as Vision and NLP, its application in industrial recommender systems presents numerous challenges. The foundational model must be scalable enough to score millions of items every second while meeting tight cost and latency constraints imposed by these systems. Additionally, it should capture the interactions between user activities and other features and handle new items that were not present during the pretraining stage. We developed innovative techniques to address these challenges. Our infrastructure and algorithmic optimizations, such as the Deduplicated Cross-Attention Transformer (DCAT), improved our throughput by 600% on Pinterest internal data. We demonstrate that PinFM can learn interactions between user sequences and candidate items by altering input sequences, leading to a 20% increase in engagement with new items. PinFM is now deployed to help improve the experience of more than half a billion users across various applications.