# NewsBERT: Distilling Pre-trained Language Model for Intelligent News Application arXiv: https://arxiv.org/abs/2102.04887 ยท HTML: https://mlsanity.com/paper/2102.04887/ ## Key facts - Cited by 17 papers in the map (data through 14 Oct 2026). - Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. - Builds on 29 of its 48 references that are in the map. - First posted on arXiv 9 Feb 2021. ## Placed in - Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/) ## Builds on (most cited first, 29 of 29) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019) - [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019) - [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019) - [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016) - [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020) - [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019) - [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016) - [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017) - [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019) - [Patient Knowledge Distillation for BERT Model Compression](https://mlsanity.com/paper/1908.09355/) (2019) - [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020) - [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://mlsanity.com/paper/2004.02984/) (2020) - [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019) - [Fake News Detection on Social Media: A Data Mining Perspective](https://mlsanity.com/paper/1708.01967/) (2017) - [MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers](https://mlsanity.com/paper/2012.15828/) (2020) - [InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training](https://mlsanity.com/paper/2007.07834/) (2020) - [UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training](https://mlsanity.com/paper/2002.12804/) (2020) - [Distilling Task-Specific Knowledge from BERT into Simple Neural Networks](https://mlsanity.com/paper/1903.12136/) (2019) - [DKN: Deep Knowledge-Aware Network for News Recommendation](https://mlsanity.com/paper/1801.08284/) (2018) - [BERT-of-Theseus: Compressing BERT by Progressive Module Replacing](https://mlsanity.com/paper/2002.02925/) (2020) - [NPA: Neural News Recommendation with Personalized Attention](https://mlsanity.com/paper/1907.05559/) (2019) - [Neural News Recommendation with Attentive Multi-View Learning](https://mlsanity.com/paper/1907.05576/) (2019) - [Self-Attentive Model for Headline Generation](https://mlsanity.com/paper/1901.07786/) (2019) - [Simplified TinyBERT: Knowledge Distillation for Document Retrieval](https://mlsanity.com/paper/2009.07531/) (2020) ## Built on by (most cited first, 17 of 17) - [ONCE: Boosting Content-based Recommendation with Both Open- and Closed-source Large Language Models](https://mlsanity.com/paper/2305.06566/) (2023) - [One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teachers](https://mlsanity.com/paper/2106.01023/) (2021) - [No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices](https://mlsanity.com/paper/2202.08036/) (2022) - [EB-NeRD: A Large-Scale Dataset for News Recommendation](https://mlsanity.com/paper/2410.03432/) (2024) - [SPAR: Personalized Content-Based Recommendation via Long Engagement Attention](https://mlsanity.com/paper/2402.10555/) (2024) - [Gradient Knowledge Distillation for Pre-trained Language Models](https://mlsanity.com/paper/2211.01071/) (2022) - [Unsupervised Story Discovery from Continuous News Streams via Scalable Thematic Embedding](https://mlsanity.com/paper/2304.04099/) (2023) - [When Gradient Descent Meets Derivative-Free Optimization: A Match Made in Black-Box Scenario](https://mlsanity.com/paper/2305.10013/) (2023) - [SCStory: Self-supervised and Continual Online Story Discovery](https://mlsanity.com/paper/2312.03725/) (2023) - [EmbSum: Leveraging the Summarization Capabilities of Large Language Models for Content-Based Recommendations](https://mlsanity.com/paper/2405.11441/) (2024) - [Few-shot News Recommendation via Cross-lingual Transfer](https://mlsanity.com/paper/2207.14370/) (2022) - [Knowledge-aware Collaborative Filtering with Pre-trained Language Model for Personalized Review-based Rating Prediction](https://mlsanity.com/paper/2308.02555/) (2023) - [A Survey on LLM-based News Recommender Systems](https://mlsanity.com/paper/2502.09797/) (2025) - [GateFormer: Speeding Up News Feed Recommendation with Input Gated Transformers](https://mlsanity.com/paper/2201.04406/) (2022) - [Does Knowledge Distillation Matter for Large Language Model based Bundle Generation?](https://mlsanity.com/paper/2504.17220/) (2025) - [Routing Distilled Knowledge via Mixture of LoRA Experts for Large Language Model based Bundle Generation](https://mlsanity.com/paper/2508.17250/) (2025) - [User recommendation system based on MIND dataset](https://mlsanity.com/paper/2209.06131/) (2022) ## Abstract (arXiv) Pre-trained language models (PLMs) like BERT have made great progress in NLP. News articles usually contain rich textual information, and PLMs have the potentials to enhance news text modeling for various intelligent news applications like news recommendation and retrieval. However, most existing PLMs are in huge size with hundreds of millions of parameters. Many online news applications need to serve millions of users with low latency tolerance, which poses huge challenges to incorporating PLMs in these scenarios. Knowledge distillation techniques can compress a large PLM into a much smaller one and meanwhile keeps good performance. However, existing language models are pre-trained and distilled on general corpus like Wikipedia, which has some gaps with the news domain and may be suboptimal for news intelligence. In this paper, we propose NewsBERT, which can distill PLMs for efficient and effective news intelligence. In our approach, we design a teacher-student joint learning and distillation framework to collaboratively learn both teacher and student models, where the student model can learn from the learning experience of the teacher model. In addition, we propose a momentum distillation method by incorporating the gradients of teacher model into the update of student model to better transfer useful knowledge learned by the teacher model. Extensive experiments on two real-world datasets with three tasks show that NewsBERT can effectively improve the model performance in various intelligent news applications with much smaller models.