Future Language Modeling from Temporal Document History
Not on the map. Why?
Abstract (arXiv)
Predicting the future is of great interest across many aspects of human activity. Businesses are interested in future trends, traders are interested in future stock prices, and companies are highly interested in future technological breakthroughs. While there are many automated systems for predicting future numerical data, such as weather, stock prices, and demand for products, there is relatively little work in automatically predicting textual data. Humans are interested in textual data predictions because it is a natural format for our consumption, and experts routinely make predictions in a textual format (Christensen et al., 2004; Tetlock & Gardner, 2015; Frick, 2015). However, there has been relatively little formalization of this general problem in the machine learning or natural language processing communities. To address this gap, we introduce the task of future language modeling: probabilistic modeling of texts in the future based on a temporal history of texts. To our knowledge, our work is the first work to formalize the task of predicting the future in this way. We show that it is indeed possible to build future language models that improve upon strong non-temporal language model baselines, opening the door to working on this important, and widely applicable problem.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 11 of its 23 references that are in the map. First posted on arXiv 16 Apr 2024.
Builds on
11 of 23 references.
References · 11
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
- 2021Time-Aware Language Models as Temporal Knowledge Basescited by 296
- 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 230
- 2022TimeLMs: Diachronic Language Models from Twittercited by 131
- 2021BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Banglacited by 125
- 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
- 2021Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Mediacited by 51
- 2021Time Masking for Temporal Language Modelscited by 46
- 2021Temporal Effects on Pre-trained Models for Language Processing Taskscited by 46
- 2020Dynamic Contextualized Word Embeddingscited by 20
Built on by
Not cited yet.