# Improving word mover's distance by leveraging self-attention matrix

arXiv: https://arxiv.org/abs/2211.06229 · HTML: https://mlsanity.com/paper/2211.06229/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 44 references that are in the map.
- First posted on arXiv 11 Nov 2022.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Computational Optimal Transport](https://mlsanity.com/paper/1803.00567/) (2018)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [PAWS: Paraphrase Adversaries from Word Scrambling](https://mlsanity.com/paper/1904.01130/) (2019)
- [Analyzing the Structure of Attention in a Transformer Language Model](https://mlsanity.com/paper/1906.04284/) (2019)
- [Optimal Transport for structured data with application on graphs](https://mlsanity.com/paper/1805.09114/) (2018)
- [Fused Gromov-Wasserstein distance for structured objects: theoretical foundations and mathematical properties](https://mlsanity.com/paper/1811.02834/) (2018)
- [Do Attention Heads in BERT Track Syntactic Dependencies?](https://mlsanity.com/paper/1911.12246/) (2019)
- [Improving Sequence-to-Sequence Learning via Optimal Transport](https://mlsanity.com/paper/1901.06283/) (2019)
- [Word Rotator's Distance](https://mlsanity.com/paper/2004.15003/) (2020)
- [Structured Optimal Transport](https://mlsanity.com/paper/1712.06199/) (2017)
- [Don't Settle for Average, Go for the Max: Fuzzy Sets and Max-Pooled Word Vectors](https://mlsanity.com/paper/1904.13264/) (2019)
- [Matching Natural Language Sentences with Hierarchical Sentence Factorization](https://mlsanity.com/paper/1803.00179/) (2018)
- [Have Attention Heads in BERT Learned Constituency Grammar?](https://mlsanity.com/paper/2102.07926/) (2021)
- [SynWMD: Syntax-aware Word Mover's Distance for Sentence Similarity Evaluation](https://mlsanity.com/paper/2206.10029/) (2022)
- [Unsupervised Sentence Textual Similarity with Compositional Phrase Semantics](https://mlsanity.com/paper/2210.02284/) (2022)
- [Structural-Aware Sentence Similarity with Recursive Optimal Transport](https://mlsanity.com/paper/2002.00745/) (2020)

## Built on by (most cited first, 3 of 3)

- [Paying Attention to Deflections: Mining Pragmatic Nuances for Whataboutism Detection in Online Discourse](https://mlsanity.com/paper/2402.09934/) (2024)
- [Automated Large-scale CVRP Solver Design via LLM-assisted Flexible MCTS](https://mlsanity.com/paper/2605.03339/) (2026)
- [Approaches to Semantic Textual Similarity in Slovak Language: From Algorithms to Transformers](https://mlsanity.com/paper/2602.04659/) (2026)

## Abstract (arXiv)

Measuring the semantic similarity between two sentences is still an important task. The word mover's distance (WMD) computes the similarity via the optimal alignment between the sets of word embeddings. However, WMD does not utilize word order, making it challenging to distinguish sentences with significant overlaps of similar words, even if they are semantically very different. Here, we attempt to improve WMD by incorporating the sentence structure represented by BERT's self-attention matrix (SAM). The proposed method is based on the Fused Gromov-Wasserstein distance, which simultaneously considers the similarity of the word embedding and the SAM for calculating the optimal transport between two sentences. Experiments demonstrate the proposed method enhances WMD and its variants in paraphrase identification with near-equivalent performance in semantic textual similarity. Our code is available at urlhttps://github.com/ymgw55/WSMD.