# Training Large-Scale News Recommenders with Pretrained Language Models in the Loop

arXiv: https://arxiv.org/abs/2102.09268 · HTML: https://mlsanity.com/paper/2102.09268/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 44 references that are in the map.
- First posted on arXiv 18 Feb 2021.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [Graph Convolutional Neural Networks for Web-Scale Recommender Systems](https://mlsanity.com/paper/1806.01973/) (2018)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Deep Interest Evolution Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1809.03672/) (2018)
- [UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training](https://mlsanity.com/paper/2002.12804/) (2020)
- [Pre-training Tasks for Embedding-based Large-scale Retrieval](https://mlsanity.com/paper/2002.03932/) (2020)
- [DKN: Deep Knowledge-Aware Network for News Recommendation](https://mlsanity.com/paper/1801.08284/) (2018)
- [NPA: Neural News Recommendation with Personalized Attention](https://mlsanity.com/paper/1907.05559/) (2019)
- [Neural News Recommendation with Attentive Multi-View Learning](https://mlsanity.com/paper/1907.05576/) (2019)
- [Graph Enhanced Representation Learning for News Recommendation](https://mlsanity.com/paper/2003.14292/) (2020)
- [Training Recommender Systems at Scale: Communication-Efficient Model and Data Parallelism](https://mlsanity.com/paper/2010.08899/) (2020)

## Built on by (most cited first, 18 of 18)

- [A Survey on Large Language Models for Recommendation](https://mlsanity.com/paper/2305.19860/) (2023)
- [Large Language Models are Zero-Shot Rankers for Recommender Systems](https://mlsanity.com/paper/2305.08845/) (2023)
- [RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder](https://mlsanity.com/paper/2205.12035/) (2022)
- [IDGenRec: LLM-RecSys Alignment with Textual ID Learning](https://mlsanity.com/paper/2403.19021/) (2024)
- [Prompt Learning for News Recommendation](https://mlsanity.com/paper/2304.05263/) (2023)
- [T-RAG: Lessons from the LLM Trenches](https://mlsanity.com/paper/2402.07483/) (2024)
- [Distill-VQ: Learning Retrieval Oriented Vector Quantization By Distilling Knowledge from Dense Embeddings](https://mlsanity.com/paper/2204.00185/) (2022)
- [Knowledge Plugins: Enhancing Large Language Models for Domain-Specific Recommendations](https://mlsanity.com/paper/2311.10779/) (2023)
- [Taxonomy-Guided Zero-Shot Recommendations with LLMs](https://mlsanity.com/paper/2406.14043/) (2024)
- [RetroMAE-2: Duplex Masked Auto-Encoder For Pre-Training Retrieval-Oriented Language Models](https://mlsanity.com/paper/2305.02564/) (2023)
- [Knowledge Prompt-tuning for Sequential Recommendation](https://mlsanity.com/paper/2308.08459/) (2023)
- [GRAM: Fast Fine-tuning of Pre-trained Language Models for Content-based Collaborative Filtering](https://mlsanity.com/paper/2204.04179/) (2022)
- [RetroMAE v2: Duplex Masked Auto-Encoder For Pre-Training Retrieval-Oriented Language Models](https://mlsanity.com/paper/2211.08769/) (2022)
- [A Survey on LLM-based News Recommender Systems](https://mlsanity.com/paper/2502.09797/) (2025)
- [HADSF: Aspect Aware Semantic Control for Explainable Recommendation](https://mlsanity.com/paper/2510.26994/) (2025)
- [GateFormer: Speeding Up News Feed Recommendation with Input Gated Transformers](https://mlsanity.com/paper/2201.04406/) (2022)
- [GREAT: Guiding Query Generation with a Trie for Recommending Related Search about Video at Kuaishou](https://mlsanity.com/paper/2507.15267/) (2025)
- [GrocLM: Grocery Category Recommendation in E-Commerce with Large Language Models](https://mlsanity.com/paper/2607.24764/) (2026)

## Abstract (arXiv)

News recommendation calls for deep insights of news articles' underlying semantics. Therefore, pretrained language models (PLMs), like BERT and RoBERTa, may substantially contribute to the recommendation quality. However, it's extremely challenging to have news recommenders trained together with such big models: the learning of news recommenders requires intensive news encoding operations, whose cost is prohibitive if PLMs are used as the news encoder. In this paper, we propose a novel framework, SpeedyFeed, which efficiently trains PLMs-based news recommenders of superior quality. SpeedyFeed is highlighted for its light-weighted encoding pipeline, which gives rise to three major advantages. Firstly, it makes the intermedia results fully reusable for the training workflow, which removes most of the repetitive but redundant encoding operations. Secondly, it improves the data efficiency of the training workflow, where non-informative data can be eliminated from encoding. Thirdly, it further saves the cost by leveraging simplified news encoding and compact news representation. Extensive experiments show that SpeedyFeed leads to more than 100× acceleration of the training process, which enables big models to be trained efficiently and effectively over massive user data. The well-trained PLMs-based model from SpeedyFeed demonstrates highly competitive performance, where it outperforms the state-of-the-art news recommenders with significant margins. SpeedyFeed is also a model-agnostic framework, which is potentially applicable to a wide spectrum of content-based recommender systems; therefore, the whole framework is open-sourced to facilitate the progress in related areas.