mlsanityData to 7 Oct 2026

Dual-View Distilled BERT for Sentence Embedding

arXiv 2104.08675 ↗First posted 18 Apr 2021cs.AI cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Recently, BERT realized significant progress for sentence matching via word-level cross sentence attention. However, the performance significantly drops when using siamese BERT-networks to derive two sentence embeddings, which fall short in capturing the global semantic since the word-level attention between two sentences is absent. In this paper, we propose a Dual-view distilled BERT~(DvBERT) for sentence matching with sentence embeddings. Our method deals with a sentence pair from two distinct views, i.e., Siamese View and Interaction View. Siamese View is the backbone where we generate sentence embeddings. Interaction View integrates the cross sentence interaction as multiple teachers to boost the representation ability of sentence embeddings. Experiments on six STS tasks show that our method outperforms the state-of-the-art sentence embedding methods significantly.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 19 of its 26 references that are in the map. First posted on arXiv 18 Apr 2021.

Builds on

19 of 26 references.

References · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  8. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  9. 2018Born Again Neural Networkscited by 539
  10. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  11. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  12. 2013A Survey on Multi-view Learningcited by 152
7 more
  1. 2019BAM! Born-Again Multi-Task Networks for Natural Language Understandingcited by 85
  2. 2018Semi-Supervised Sequence Modeling with Cross-View Trainingcited by 75
  3. 2019Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understandingcited by 37
  4. 2018Neural Network Models for Paraphrase Identification, Semantic Textual Similarity, Natural Language Inference, and Question Answeringcited by 23
  5. 2019Toward Fast and Accurate Neural Chinese Word Segmentation with Multi-Criteria Learningcited by 11
  6. 2019Symmetric Regularization based BERT for Pair-wise Semantic Reasoningcited by 4
  7. 2019Making Neural Machine Reading Comprehension Fastercited by 2

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 0; 2023: 3; 2024: 1; 2025: 1.

7 papers.

  1. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  2. 2021Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillationscited by 20
  3. 2023A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the ChatGPT Era and Beyondcited by 11
  4. 2023Zero-Shot Multi-Label Topic Inference with Sentence Encoderscited by 6
  5. 2024SlideSpawn: An Automatic Slides Generation System for Research Publicationscited by 5
  6. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  7. 2023NoPPA: Non-Parametric Pairwise Attention Random Walk Model for Sentence Representationcited by 0