# DySem: Uncovering Dynamic Semantic Components of Large Language Models for Calculating Semantic Textual Similarity

arXiv: https://arxiv.org/abs/2605.29751 · HTML: https://mlsanity.com/paper/2605.29751/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 47 references that are in the map.
- First posted on arXiv 28 May 2026.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models](https://mlsanity.com/paper/2405.17428/) (2024)
- [LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders](https://mlsanity.com/paper/2404.05961/) (2024)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019)
- [Layer by Layer: Uncovering Hidden Representations in Language Models](https://mlsanity.com/paper/2502.02013/) (2025)
- [Gemini Embedding: Generalizable Embeddings from Gemini](https://mlsanity.com/paper/2503.07891/) (2025)
- [Token Prepending: A Training-Free Approach for Eliciting Better Sentence Embeddings from LLMs](https://mlsanity.com/paper/2412.11556/) (2024)
- [Contrastive Prompting Enhances Sentence Embeddings in LLMs through Inference-Time Steering](https://mlsanity.com/paper/2505.12831/) (2025)
- [Interpreting Context Look-ups in Transformers: Investigating Attention-MLP Interactions](https://mlsanity.com/paper/2402.15055/) (2024)
- [On the Dimensionality of Sentence Embeddings](https://mlsanity.com/paper/2310.15285/) (2023)
- [LLM-based Embeddings: Attention Values Encode Sentence Semantics Better Than Hidden States](https://mlsanity.com/paper/2602.01572/) (2026)
- [SemPA: Improving Sentence Embeddings of Large Language Models through Semantic Preference Alignment](https://mlsanity.com/paper/2601.05075/) (2026)

## Abstract (arXiv)

Calculating semantic textual similarity is a foundational task in natural language processing. Current large language models (LLMs) based methods typically rely on extracting last-layer hidden states with fixed dimensions to compute similarity for every text pairs. We argue that this paradigm is suffer from two limitations: (i) The last hidden layer encodes more general knowledge rather than just semantic knowledge, making it suboptimal for semantic similarity computation; (ii) The hidden layer dimensions of LLMs are generally very large, which introduces some redundancy and noise for representing semantics. In this work, we propose DySem, a novel training-free framework that investigates more semantic-related internal components of LLMs via multilingual consensus, and shifts away from static representation spaces in favor of dynamic, sample-specific semantic dimensions by constructing text-dependent joint semantic set and computes similarity over this shared dimensional subset. Extensive experiments across various LLMs show that our method consistently outperforms recent baselines while maintaining lower dimensions for similarity calculation. The code is released at https://github.com/szu-tera/DySem.