# Drinking from a Firehose: Continual Learning with Web-scale Natural Language

arXiv: https://arxiv.org/abs/2007.09335 · HTML: https://mlsanity.com/paper/2007.09335/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jul 2020.

## Built on by (most cited first, 25 of 25)

- [Continual Learning for Large Language Models: A Survey](https://mlsanity.com/paper/2402.01364/) (2024)
- [StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models](https://mlsanity.com/paper/2205.11388/) (2022)
- [The CLEAR Benchmark: Continual LEArning on Real-World Imagery](https://mlsanity.com/paper/2201.06289/) (2022)
- [Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Data](https://mlsanity.com/paper/2108.09020/) (2021)
- [Continual Learning: Applications and the Road Forward](https://mlsanity.com/paper/2311.11908/) (2023)
- [Recent Advances of Foundation Language Models-based Continual Learning: A Survey](https://mlsanity.com/paper/2405.18653/) (2024)
- [Learning Fast and Slow for Online Time Series Forecasting](https://mlsanity.com/paper/2202.11672/) (2022)
- [Continual Learning as Computationally Constrained Reinforcement Learning](https://mlsanity.com/paper/2307.04345/) (2023)
- [Online Continual Learning Without the Storage Constraint](https://mlsanity.com/paper/2305.09253/) (2023)
- [A Practitioner's Guide to Continual Multimodal Pretraining](https://mlsanity.com/paper/2408.14471/) (2024)
- [Pretrained Vision-Language-Action Models are Surprisingly Resistant to Forgetting in Continual Learning](https://mlsanity.com/paper/2603.03818/) (2026)
- [New Insights for the Stability-Plasticity Dilemma in Online Continual Learning](https://mlsanity.com/paper/2302.08741/) (2023)
- [LCA-on-the-Line: Benchmarking Out-of-Distribution Generalization with Class Taxonomies](https://mlsanity.com/paper/2407.16067/) (2024)
- [Feature Forgetting in Continual Representation Learning](https://mlsanity.com/paper/2205.13359/) (2022)
- [Dynamically Addressing Unseen Rumor via Continual Learning](https://mlsanity.com/paper/2104.08775/) (2021)
- [Improving information retention in large scale online continual learning](https://mlsanity.com/paper/2210.06401/) (2022)
- [Domain Generalization without Excess Empirical Risk](https://mlsanity.com/paper/2308.15856/) (2023)
- [CoLLEGe: Concept Embedding Generation for Large Language Models](https://mlsanity.com/paper/2403.15362/) (2024)
- [HPCR: Holistic Proxy-based Contrastive Replay for Online Continual Learning](https://mlsanity.com/paper/2309.15038/) (2023)
- [Multi-Stream Cellular Test-Time Adaptation of Real-Time Models Evolving in Dynamic Environments](https://mlsanity.com/paper/2404.17930/) (2024)
- [A Practical Guide to Streaming Continual Learning](https://mlsanity.com/paper/2603.01677/) (2026)
- [Taming the Beast: Learning to Control Neural Conversational Models](https://mlsanity.com/paper/2108.10561/) (2021)
- [DoDo Learning: DOmain-DemOgraphic Transfer in Language Models for Detecting Abuse Targeted at Public Figures](https://mlsanity.com/paper/2307.16811/) (2023)
- [ER-FSL: Experience Replay with Feature Subspace Learning for Online Continual Learning](https://mlsanity.com/paper/2407.12279/) (2024)
- [Task Vector Descent: Learning from Non-IID Batches](https://mlsanity.com/paper/2610.05402/) (2026)

## Abstract (arXiv)

Continual learning systems will interact with humans, with each other, and with the physical world through time -- and continue to learn and adapt as they do. An important open problem for continual learning is a large-scale benchmark that enables realistic evaluation of algorithms. In this paper, we study a natural setting for continual learning on a massive scale. We introduce the problem of personalized online language learning (POLL), which involves fitting personalized language models to a population of users that evolves over time. To facilitate research on POLL, we collect massive datasets of Twitter posts. These datasets, Firehose10M and Firehose100M, comprise 100 million tweets, posted by one million users over six years. Enabled by the Firehose datasets, we present a rigorous evaluation of continual learning algorithms on an unprecedented scale. Based on this analysis, we develop a simple algorithm for continual gradient descent (ConGraD) that outperforms prior continual learning methods on the Firehose datasets as well as earlier benchmarks. Collectively, the POLL problem setting, the Firehose datasets, and the ConGraD algorithm enable a complete benchmark for reproducible research on web-scale continual learning.