# The Role of Complex NLP in Transformers for Text Ranking?

arXiv: https://arxiv.org/abs/2207.02522 · HTML: https://mlsanity.com/paper/2207.02522/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 36 references that are in the map.
- First posted on arXiv 6 Jul 2022.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [A Decomposable Attention Model for Natural Language Inference](https://mlsanity.com/paper/1606.01933/) (2016)
- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://mlsanity.com/paper/2104.06644/) (2021)
- [Out of Order: How Important Is The Sequential Order of Words in a Sentence in Natural Language Understanding Tasks?](https://mlsanity.com/paper/2012.15180/) (2020)
- [UnNatural Language Inference](https://mlsanity.com/paper/2101.00010/) (2020)
- [BERT & Family Eat Word Salad: Experiments with Text Understanding](https://mlsanity.com/paper/2101.03453/) (2021)
- [ABNIRML: Analyzing the Behavior of Neural IR Models](https://mlsanity.com/paper/2011.00696/) (2020)
- [Rethinking Search: Making Domain Experts out of Dilettantes](https://mlsanity.com/paper/2105.02274/) (2021)
- [A White Box Analysis of ColBERT](https://mlsanity.com/paper/2012.09650/) (2020)
- [Diagnosing BERT with Retrieval Heuristics](https://mlsanity.com/paper/2201.04458/) (2022)

## Built on by (most cited first, 1 of 1)

- [NevIR: Negation in Neural Information Retrieval](https://mlsanity.com/paper/2305.07614/) (2023)

## Abstract (arXiv)

Even though term-based methods such as BM25 provide strong baselines in ranking, under certain conditions they are dominated by large pre-trained masked language models (MLMs) such as BERT. To date, the source of their effectiveness remains unclear. Is it their ability to truly understand the meaning through modeling syntactic aspects? We answer this by manipulating the input order and position information in a way that destroys the natural sequence order of query and passage and shows that the model still achieves comparable performance. Overall, our results highlight that syntactic aspects do not play a critical role in the effectiveness of re-ranking with BERT. We point to other mechanisms such as query-passage cross-attention and richer embeddings that capture word meanings based on aggregated context regardless of the word order for being the main attributions for its superior performance.