mlsanityData to 14 Oct 2026

Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaning

arXiv 2506.08354 ↗First posted 10 Jun 2025cs.CL cs.AI cs.IRcited by 10

Not on the map. Why?

Abstract (arXiv)

This position paper argues that text embedding research should move beyond surface meaning and embrace implicit semantics as a central modeling objective. Text embeddings are a foundational component of modern NLP, underpinning a wide range of applications and driving sustained research progress. Despite rapid progress, most embedding models remain narrowly focused on surface-level semantics, whereas linguistic theory emphasizes that much of human meaning is implicit, shaped by pragmatics, speaker intent, and sociocultural context. Current models are typically trained on datasets that lack such depth and evaluated using benchmarks that reward surface similarity. As a result, they struggle with tasks that require interpretive reasoning, stance recognition, or socially grounded understanding. Our pilot study makes this limitation explicit, showing that even state-of-the-art embeddings achieve only marginal improvements over simple lexical baselines on tasks probing implicit semantics. We therefore call for a paradigm shift: embedding research should prioritize linguistically grounded and diverse training data, develop benchmarks that probe deeper semantic understanding, and treat implicit meaning as a core modeling objective to better align embeddings with real-world language complexity. The code is available at http://github.com/dukesun99/Implicit-Embeddings.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 148 of its 236 references that are in the map. First posted on arXiv 10 Jun 2025.

Builds on

148 of 236 references.

References · 148

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  9. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  11. 2018Deep contextualized word representationscited by 2,290
  12. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
38 more
  1. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  2. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  3. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  4. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  5. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  6. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  7. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  8. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  9. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 801
  10. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  11. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  12. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
  13. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 476
  14. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
  15. 2021Large Dual Encoders Are Generalizable Retrieverscited by 456
  16. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 432
  17. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
  18. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 380
  19. 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 372
  20. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 361
  21. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 333
  22. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 331
  23. 2015Skip-Thought Vectorscited by 322
  24. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 312
  25. 2023Improving Text Embeddings with Large Language Modelscited by 303
  26. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 283
  27. 2022Matryoshka Representation Learningcited by 282
  28. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queriescited by 257
  29. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 236
  30. 2024Generative Representational Instruction Tuningcited by 233
  31. 2021Latent Hatred: A Benchmark for Understanding Implicit Hate Speechcited by 217
  32. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  33. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 158
  34. 2022SciRepEval: A Multi-Format Benchmark for Scientific Document Representationscited by 129
  35. 2025MMTEB: Massive Multilingual Text Embedding Benchmarkcited by 125
  36. 2013A Theoretical Analysis of NDCG Type Ranking Measurescited by 119
  37. 2021Mr. TyDi: A Multi-lingual Benchmark for Dense Retrievalcited by 119
  38. 2024LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domaincited by 118

50 most cited of 148 All 148 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 8.

10 papers.