# Anticipating Innovation Using Large Language Models

arXiv: https://arxiv.org/abs/2605.04875 · HTML: https://mlsanity.com/paper/2605.04875/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 43 references that are in the map.
- First posted on arXiv 6 May 2026.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders](https://mlsanity.com/paper/2404.05961/) (2024)
- [Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders](https://mlsanity.com/paper/2104.08027/) (2021)
- [PatentSBERTa: A Deep NLP based Hybrid Model for Patent Distance and Classification using Augmented SBERT](https://mlsanity.com/paper/2103.11933/) (2021)
- [Natural Language Processing in the Patent Domain: A Survey](https://mlsanity.com/paper/2403.04105/) (2024)
- [PaECTER: Patent-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2402.19411/) (2024)
- [Invention as a Combinatorial Process: Evidence from U.S. Patents](https://mlsanity.com/paper/1406.2938/) (2014)
- [The dynamics of correlated novelties](https://mlsanity.com/paper/1310.1953/) (2013)
- [Inferring monopartite projections of bipartite networks: an entropy-based approach](https://mlsanity.com/paper/1607.02481/) (2016)

## Abstract (arXiv)

Forecasting innovation, intended as the emergence of new technological combinations, is a fundamental challenge for science and policy. We show that forthcoming combinations leave an early trace in the collective language of patents, with predictive signals detectable even decades in advance. We show that signal is not attributable to any single inventor, but emerges as a collective shift in how technologies are described across thousands of patents. To this end, we introduce TechToken, a transformer-based model that treats technologies, classified by International Patent Classification codes, as words in its vocabulary, learning the language of technologies by embedding these codes during fine-tuning. We define context similarity between code embeddings as a measure of linguistic convergence and show that it accurately predicts first technological combinations. TechToken also improves general representation quality, outperforming state-of-the-art models across different patent-related tasks.