mlsanityData to 7 Oct 2026

SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models

arXiv 2002.06652 ↗First posted 16 Feb 2020cs.CL cs.LG cs.MMcited by 55

Not on the map. Why?

Abstract (arXiv)

Sentence embedding is an important research topic in natural language processing (NLP) since it can transfer knowledge to downstream tasks. Meanwhile, a contextualized word representation, called BERT, achieves the state-of-the-art performance in quite a few NLP tasks. Yet, it is an open problem to generate a high quality sentence representation from BERT-based word models. It was shown in previous study that different layers of BERT capture different linguistic properties. This allows us to fusion information across layers to find better sentence representation. In this work, we study the layer-wise pattern of the word representation of deep contextualized models. Then, we propose a new sentence embedding method by dissecting BERT-based word models through geometric analysis of the space spanned by the word representation. It is called the SBERT-WK method. No further training is required in SBERT-WK. We evaluate SBERT-WK on semantic textual similarity and downstream supervised tasks. Furthermore, ten sentence-level probing tasks are presented for detailed linguistic analysis. Experiments show that SBERT-WK achieves the state-of-the-art performance. Our codes are publicly available.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 14; 2023: 10; 2024: 10; 2025: 5; 2026: 2.

55 papers.

Sort:

By area:

  1. 2024Generative Representational Instruction Tuningcited by 227
  2. 2021NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Mediacited by 106
  3. 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
  4. 2021Open-Domain, Content-based, Multi-modal Fact-checking of Out-of-Context Images via Online Resourcescited by 71
  5. 2021WhiteningBERT: An Easy Unsupervised Sentence Embedding Approachcited by 68
  6. 2022SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Featurescited by 33
  7. 2021COSMOS: Catching Out-of-Context Misinformation with Self-Supervised Learningcited by 26
  8. 2021Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factorscited by 25
  9. 2023An Overview on Language Models: Recent Developments and Outlookcited by 22
  10. 2022Just Rank: Rethinking Evaluation with Word and Sentence Similaritiescited by 20
  11. 2023Nugget: Neural Agglomerative Embeddings of Textcited by 19
  12. 2022Entity-centered Cross-document Relation Extractioncited by 18
38 more
  1. 2023SpeechFormer++: A Hierarchical Efficient Framework for Paralinguistic Speech Processingcited by 16
  2. 2024MFC-Bench: Benchmarking Multimodal Fact-Checking with Large Vision-Language Modelscited by 15
  3. 2022Addressing Token Uniformity in Transformers via Singular Value Transformationcited by 14
  4. 2024Multimodal Misinformation Detection by Learning from Synthetic Data with Multimodal LLMscited by 13
  5. 2021Sentence Similarity Based on Contextscited by 12
  6. 2022Analyzing and Evaluating Faithfulness in Dialogue Summarizationcited by 12
  7. 2025Interpretable Text Embeddings and Text Similarity Explanation: A Surveycited by 12
  8. 2024Similarity over Factuality: Are we making progress on multimodal out-of-context misinformation detection?cited by 11
  9. 2022L3Cube-MahaSBERT and HindSBERT: Sentence BERT Models and Benchmarking BERT Sentence Representations for Hindi and Marathicited by 10
  10. 2024Extracting Sentence Embeddings from Pretrained Transformer Modelscited by 7
  11. 2025DioR: Adaptive Cognitive Detection and Contextual Retrieval Optimization for Dynamic Retrieval-Augmented Generationcited by 7
  12. 2021Natural Language Understanding for Argumentative Dialogue Systems in the Opinion Building Domaincited by 6
  13. 2023Harnessing the Power of Text-image Contrastive Models for Automatic Detection of Online Misinformationcited by 6
  14. 2024From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Modelscited by 6
  15. 2022Compressing Sentence Representation for Semantic Retrieval via Homomorphic Projective Distillationcited by 5
  16. 2024SlideSpawn: An Automatic Slides Generation System for Research Publicationscited by 5
  17. 2021Using BERT Encoding and Sentence-Level Language Model for Sentence Orderingcited by 4
  18. 2022SynWMD: Syntax-aware Word Mover's Distance for Sentence Similarity Evaluationcited by 4
  19. 2022Relational Sentence Embedding for Flexible Semantic Matchingcited by 4
  20. 2021Task-Specific Dependency-based Word Embedding Methodscited by 3
  21. 2022Comparison and Combination of Sentence Embeddings Derived from Different Supervision Signalscited by 3
  22. 2022A Survey on Sentence Embedding Models Performance for Patent Analysiscited by 3
  23. 2022Unsupervised Simplification of Legal Textscited by 3
  24. 2022Multi-stage Information Retrieval for Vietnamese Legal Textscited by 3
  25. 2021Multitask Learning for Class-Imbalanced Discourse Classificationcited by 2
  26. 2021Interpretable contrastive word mover's embeddingcited by 2
  27. 2023Probing Out-of-Distribution Robustness of Language Models with Parameter-Efficient Transfer Learningcited by 2
  28. 2023Detecting Out-of-Context Image-Caption Pairs in News: A Counter-Intuitive Methodcited by 2
  29. 2024SetCSE: Set Operations using Contrastive Learning of Sentence Embeddingscited by 2
  30. 2025Emergent musical properties of a transformer under contrastive self-supervised learningcited by 2
  31. 2021A New Sentence Ordering Method Using BERT Pretrained Modelcited by 1
  32. 2023Enhancing Black-Box Few-Shot Text Classification with Prompt-Based Data Augmentationcited by 1
  33. 2024Leveraging MLLM Embeddings and Attribute Smoothing for Compositional Zero-Shot Learningcited by 1
  34. 2021Transferring Semantic Knowledge Into Language Encoderscited by 0
  35. 2022Pruned Graph Neural Network for Short Story Orderingcited by 0
  36. 2023NoPPA: Non-Parametric Pairwise Attention Random Walk Model for Sentence Representationcited by 0
  37. 2023Leveraging Language Identification to Enhance Code-Mixed Text Classificationcited by 0
  38. 2023Sentiment analysis in Tourism: Fine-tuning BERT or sentence embeddings concatenation?cited by 0

50 most cited of 55 All 55 (JSON)