Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaning
Not on the map. Why?
Abstract (arXiv)
This position paper argues that text embedding research should move beyond surface meaning and embrace implicit semantics as a central modeling objective. Text embeddings are a foundational component of modern NLP, underpinning a wide range of applications and driving sustained research progress. Despite rapid progress, most embedding models remain narrowly focused on surface-level semantics, whereas linguistic theory emphasizes that much of human meaning is implicit, shaped by pragmatics, speaker intent, and sociocultural context. Current models are typically trained on datasets that lack such depth and evaluated using benchmarks that reward surface similarity. As a result, they struggle with tasks that require interpretive reasoning, stance recognition, or socially grounded understanding. Our pilot study makes this limitation explicit, showing that even state-of-the-art embeddings achieve only marginal improvements over simple lexical baselines on tasks probing implicit semantics. We therefore call for a paradigm shift: embedding research should prioritize linguistically grounded and diverse training data, develop benchmarks that probe deeper semantic understanding, and treat implicit meaning as a core modeling objective to better align embeddings with real-world language complexity. The code is available at http://github.com/dukesun99/Implicit-Embeddings.
Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 148 of its 236 references that are in the map. First posted on arXiv 10 Jun 2025.
Builds on
148 of 236 references.
References · 148
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
- 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
- 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
- 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
- 2018Deep contextualized word representationscited by 2,290
- 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
38 more
- 2015A large annotated corpus for learning natural language inferencecited by 1,847
- 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
- 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
- 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
- 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
- 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
- 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
- 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
- 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 801
- 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
- 2022MTEB: Massive Text Embedding Benchmarkcited by 640
- 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
- 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 476
- 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
- 2021Large Dual Encoders Are Generalizable Retrieverscited by 456
- 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 432
- 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
- 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 380
- 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 372
- 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 361
- 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 333
- 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 331
- 2015Skip-Thought Vectorscited by 322
- 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 312
- 2023Improving Text Embeddings with Large Language Modelscited by 303
- 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 283
- 2022Matryoshka Representation Learningcited by 282
- 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queriescited by 257
- 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 236
- 2024Generative Representational Instruction Tuningcited by 233
- 2021Latent Hatred: A Benchmark for Understanding Implicit Hate Speechcited by 217
- 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
- 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 158
- 2022SciRepEval: A Multi-Format Benchmark for Scientific Document Representationscited by 129
- 2025MMTEB: Massive Multilingual Text Embedding Benchmarkcited by 125
- 2013A Theoretical Analysis of NDCG Type Ranking Measurescited by 119
- 2021Mr. TyDi: A Multi-lingual Benchmark for Dense Retrievalcited by 119
- 2024LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domaincited by 118
50 most cited of 148 All 148 (JSON)
Built on by
Citing papers per year: 2025: 2; 2026: 8.
10 papers.
- 2026Corpus2Skill: Distilling Enterprise Knowledge into Navigable Agent Skills for QA and RAGcited by 7
- 2026Very Efficient Listwise Multimodal Reranking for Long Documentscited by 2
- 2025Towards Bridging Review Sparsity in Recommendation with Textual Edge Graph Representationcited by 1
- 2026When Models Hear What They Expect: Diagnosing Prosodic Heuristics in Multimodal Sarcasm Detectioncited by 1
- 2025The Semantic Illusion: Certified Limits of Embedding-Based Hallucination Detection in RAG Systemscited by 0
- 2026LLMs Should Incorporate Explicit Mechanisms for Human Empathycited by 0
- 2026ACAT: A Collaborative Platform for Efficient Aspect-Based Sentiment Dataset Annotationcited by 0
- 2026An Investigation of the NeurIPS and ICML 2025 Position Trackscited by 0
- 2026Do General NLP Embeddings Capture Ontological Reasoning?cited by 0
- 2026Embedding Models Measure in Peculiar Wayscited by 0
- 2026Embedding Models Measure in Peculiar Ways17 Sep 2026
- 2026Do General NLP Embeddings Capture Ontological Reasoning?31 Aug 2026
- 2026When Models Hear What They Expect: Diagnosing Prosodic Heuristics in Multimodal Sarcasm Detection31 Aug 2026
- 2026An Investigation of the NeurIPS and ICML 2025 Position Tracks6 Jul 2026
- 2026ACAT: A Collaborative Platform for Efficient Aspect-Based Sentiment Dataset Annotation2 Jun 2026
- 2026Very Efficient Listwise Multimodal Reranking for Long Documents12 May 2026
- 2026Corpus2Skill: Distilling Enterprise Knowledge into Navigable Agent Skills for QA and RAG16 Apr 2026
- 2026LLMs Should Incorporate Explicit Mechanisms for Human Empathy12 Apr 2026
- 2025The Semantic Illusion: Certified Limits of Embedding-Based Hallucination Detection in RAG Systems17 Dec 2025
- 2025Towards Bridging Review Sparsity in Recommendation with Textual Edge Graph Representation2 Aug 2025