# On Length Divergence Bias in Textual Matching Models

arXiv: https://arxiv.org/abs/2109.02431 · HTML: https://mlsanity.com/paper/2109.02431/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 26 references that are in the map.
- First posted on arXiv 6 Sep 2021.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [Probing Neural Network Comprehension of Natural Language Arguments](https://mlsanity.com/paper/1907.07355/) (2019)
- [Stress Test Evaluation for Natural Language Inference](https://mlsanity.com/paper/1806.00692/) (2018)
- [Enhanced LSTM for Natural Language Inference](https://mlsanity.com/paper/1609.06038/) (2016)
- [Bilateral Multi-Perspective Matching for Natural Language Sentences](https://mlsanity.com/paper/1702.03814/) (2017)
- [A Continuously Growing Dataset of Sentential Paraphrases](https://mlsanity.com/paper/1708.00391/) (2017)
- [Text Matching as Image Recognition](https://mlsanity.com/paper/1602.06359/) (2016)
- [A Deep Architecture for Semantic Matching with Multiple Positional Sentence Representations](https://mlsanity.com/paper/1511.08277/) (2015)
- [MatchZoo: A Learning, Practicing, and Developing System for Neural Text Matching](https://mlsanity.com/paper/1905.10289/) (2019)
- [FAQ-based Question Answering via Word Alignment](https://mlsanity.com/paper/1507.02628/) (2015)
- [Multi-Perspective Relevance Matching with Hierarchical ConvNets for Social Media Search](https://mlsanity.com/paper/1805.08159/) (2018)

## Built on by (most cited first, 4 of 4)

- [A Long Way to Go: Investigating Length Correlations in RLHF](https://mlsanity.com/paper/2310.03716/) (2023)
- [ROSE: Robust Selective Fine-tuning for Pre-trained Language Models](https://mlsanity.com/paper/2210.09658/) (2022)
- [Code Search Debiasing:Improve Search Results beyond Overall Ranking Performance](https://mlsanity.com/paper/2311.14901/) (2023)
- [M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models](https://mlsanity.com/paper/2608.03803/) (2026)

## Abstract (arXiv)

Despite the remarkable success deep models have achieved in Textual Matching (TM) tasks, it still remains unclear whether they truly understand language or measure the semantic similarity of texts by exploiting statistical bias in datasets. In this work, we provide a new perspective to study this issue -- via the length divergence bias. We find the length divergence heuristic widely exists in prevalent TM datasets, providing direct cues for prediction. To determine whether TM models have adopted such heuristic, we introduce an adversarial evaluation scheme which invalidates the heuristic. In this adversarial setting, all TM models perform worse, indicating they have indeed adopted this heuristic. Through a well-designed probing experiment, we empirically validate that the bias of TM models can be attributed in part to extracting the text length information during training. To alleviate the length divergence bias, we propose an adversarial training method. The results demonstrate we successfully improve the robustness and generalization ability of models at the same time.