mlsanityData to 7 Oct 2026

Improving word mover's distance by leveraging self-attention matrix

arXiv 2211.06229 ↗First posted 11 Nov 2022cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Measuring the semantic similarity between two sentences is still an important task. The word mover's distance (WMD) computes the similarity via the optimal alignment between the sets of word embeddings. However, WMD does not utilize word order, making it challenging to distinguish sentences with significant overlaps of similar words, even if they are semantically very different. Here, we attempt to improve WMD by incorporating the sentence structure represented by BERT's self-attention matrix (SAM). The proposed method is based on the Fused Gromov-Wasserstein distance, which simultaneously considers the similarity of the word embedding and the SAM for calculating the optimal transport between two sentences. Experiments demonstrate the proposed method enhances WMD and its variants in paraphrase identification with near-equivalent performance in semantic textual similarity. Our code is available at urlhttps://github.com/ymgw55/WSMD.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 24 of its 44 references that are in the map. First posted on arXiv 11 Nov 2022.

Builds on

24 of 44 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  6. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  7. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  8. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  9. 2018Computational Optimal Transportcited by 765
  10. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  11. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  12. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 241
12 more
  1. 2018Optimal Transport for structured data with application on graphscited by 170
  2. 2018Fused Gromov-Wasserstein distance for structured objects: theoretical foundations and mathematical propertiescited by 93
  3. 2019Do Attention Heads in BERT Track Syntactic Dependencies?cited by 87
  4. 2019Improving Sequence-to-Sequence Learning via Optimal Transportcited by 33
  5. 2020Word Rotator's Distancecited by 32
  6. 2017Structured Optimal Transportcited by 30
  7. 2019Don't Settle for Average, Go for the Max: Fuzzy Sets and Max-Pooled Word Vectorscited by 7
  8. 2018Matching Natural Language Sentences with Hierarchical Sentence Factorizationcited by 5
  9. 2021Have Attention Heads in BERT Learned Constituency Grammar?cited by 5
  10. 2022SynWMD: Syntax-aware Word Mover's Distance for Sentence Similarity Evaluationcited by 4
  11. 2022Unsupervised Sentence Textual Similarity with Compositional Phrase Semanticscited by 4
  12. 2020Structural-Aware Sentence Similarity with Recursive Optimal Transportcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 2.

3 papers.

  1. 2024Paying Attention to Deflections: Mining Pragmatic Nuances for Whataboutism Detection in Online Discoursecited by 1
  2. 2026Automated Large-scale CVRP Solver Design via LLM-assisted Flexible MCTScited by 1
  3. 2026Approaches to Semantic Textual Similarity in Slovak Language: From Algorithms to Transformerscited by 0