# A Text is Worth Several Tokens: Text Embedding from LLMs Secretly Aligns Well with The Key Tokens

arXiv: https://arxiv.org/abs/2406.17378 · HTML: https://mlsanity.com/paper/2406.17378/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jun 2024.

## Built on by (most cited first, 8 of 8)

- [LLMs are Also Effective Embedding Models: An In-depth Overview](https://mlsanity.com/paper/2412.12591/) (2024)
- [Two Heads are Better Than One: Test-time Scaling of Multi-agent Collaborative Reasoning](https://mlsanity.com/paper/2504.09772/) (2025)
- [Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization](https://mlsanity.com/paper/2510.13554/) (2025)
- [How Does Reasoning Flow? Tracing Attention-Induced Information Flow for Targeted RL in LLMs](https://mlsanity.com/paper/2606.10646/) (2026)
- [SemPA: Improving Sentence Embeddings of Large Language Models through Semantic Preference Alignment](https://mlsanity.com/paper/2601.05075/) (2026)
- [Your UnEmbedding Matrix is Secretly a Feature Lens for Text Embeddings](https://mlsanity.com/paper/2606.07502/) (2026)
- [Text2Token: Unsupervised Text Representation Learning with Token Target Prediction](https://mlsanity.com/paper/2510.10224/) (2025)
- [RGAlign-Rec: Ranking-Guided Alignment for Latent Query Reasoning in Recommendation Systems](https://mlsanity.com/paper/2602.12968/) (2026)

## Abstract (arXiv)

Text embeddings from large language models (LLMs) have achieved excellent results in tasks such as information retrieval, semantic textual similarity, etc. In this work, we show an interesting finding: when feeding a text into the LLM-based embedder, the obtained text embedding will be able to be aligned with the key tokens in the input text. We first fully analyze this phenomenon on eight LLM-based embedders and show that this phenomenon is universal and is not affected by model architecture, training strategy, and embedding method. With a deeper analysis, we find that the main change in embedding space between these embedders and their LLM backbones is in the first principal component. By adjusting the first principal component, we can align text embedding with the key tokens. Finally, we give several examples to demonstrate the vast application potential of this finding: (1) we propose a simple and practical sparse retrieval method based on the aligned tokens, which can achieve 80% of the dense retrieval effect of the same model while reducing the computation significantly; (2) we show that our findings provide a novel perspective to help understand novel technologies (e.g., instruction-following embedding) and fuzzy concepts (e.g., semantic relatedness vs. similarity) in this field.