mlsanityData to 14 Oct 2026

On Self-improving Token Embeddings

arXiv 2504.14808 ↗First posted 21 Apr 2025cs.CL cs.AI cs.IRcited by 0

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

This article introduces a novel and fast method for refining pre-trained static word or, more generally, token embeddings. By incorporating the embeddings of neighboring tokens in text corpora, it continuously updates the representation of each token, including those without pre-assigned embeddings. This approach effectively addresses the out-of-vocabulary problem, too. Operating independently of large language models and shallow neural networks, it enables versatile applications such as corpus exploration, conceptual search, and word sense disambiguation. The method is designed to enhance token representations within topically homogeneous corpora, where the vocabulary is restricted to a specific domain, resulting in more meaningful embeddings compared to general-purpose pre-trained vectors. As an example, the methodology is applied to explore storm events and their impacts on infrastructure and communities using narratives from a subset of the NOAA Storm Events database. The article also demonstrates how the approach improves the representation of storm-related terms over time, providing valuable insights into the evolving nature of disaster narratives.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 6 of its 16 references that are in the map. First posted on arXiv 21 Apr 2025.

Builds on

6 of 16 references.

References · 6

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  4. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  5. 2014Distributed Representations of Sentences and Documentscited by 678
  6. 2018A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectorscited by 17

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkit
  2. 2024Extracting Sentence Embeddings from Pretrained Transformer Models
  3. 2026World Properties without World Models: Recovering Spatial and Temporal Structure from Co-occurrence Statistics in Static Word Embeddings
  4. 2024Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark
  5. 2025CALE : Concept-Aligned Embeddings for Both Within-Lemma and Inter-Lemma Sense Differentiation
  6. 2025Learning to Compress: Unlocking the Potential of Large Language Models for Text Representation
  7. 2025Dewey Long Context Embedding Model: A Technical Report
  8. 2025Luxical: High-Speed Lexical-Dense Text Embeddings