# Critically Examining the "Neural Hype": Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models

arXiv: https://arxiv.org/abs/1904.09171 · HTML: https://mlsanity.com/paper/1904.09171/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Apr 2019.

## Built on by (most cited first, 24 of 24)

- [Overview of the TREC 2020 deep learning track](https://mlsanity.com/paper/2102.07662/) (2021)
- [The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models](https://mlsanity.com/paper/2101.05667/) (2021)
- [All for One, and One for All: UrbanSyn Dataset, the third Musketeer of Synthetic Driving Scenes](https://mlsanity.com/paper/2312.12176/) (2023)
- [Retrieval-augmented Generation to Improve Math Question-Answering: Trade-offs Between Groundedness and Human Preference](https://mlsanity.com/paper/2310.03184/) (2023)
- [Keyphrase Generation for Scientific Document Retrieval](https://mlsanity.com/paper/2106.14726/) (2021)
- [A Next Basket Recommendation Reality Check](https://mlsanity.com/paper/2109.14233/) (2021)
- [Redefining Absent Keyphrases and their Effect on Retrieval Effectiveness](https://mlsanity.com/paper/2103.12440/) (2021)
- [Studying Catastrophic Forgetting in Neural Ranking Models](https://mlsanity.com/paper/2101.06984/) (2021)
- [Long Document Re-ranking with Modular Re-ranker](https://mlsanity.com/paper/2205.04275/) (2022)
- [Mistral-SPLADE: LLMs for better Learned Sparse Retrieval](https://mlsanity.com/paper/2408.11119/) (2024)
- [CsFEVER and CTKFacts: Acquiring Czech data for fact verification](https://mlsanity.com/paper/2201.11115/) (2022)
- [Natural Language Inference with Self-Attention for Veracity Assessment of Pandemic Claims](https://mlsanity.com/paper/2205.02596/) (2022)
- [Simple Yet Effective Neural Ranking and Reranking Baselines for Cross-Lingual Information Retrieval](https://mlsanity.com/paper/2304.01019/) (2023)
- [Pipeline and Dataset Generation for Automated Fact-checking in Almost Any Language](https://mlsanity.com/paper/2312.10171/) (2023)
- ["Ask Me Anything": How Comcast Uses LLMs to Assist Agents in Real Time](https://mlsanity.com/paper/2405.00801/) (2024)
- [Questionable practices in machine learning](https://mlsanity.com/paper/2407.12220/) (2024)
- [VLQA: The First Comprehensive, Large, and High-Quality Vietnamese Dataset for Legal Question Answering](https://mlsanity.com/paper/2507.19995/) (2025)
- [torchdistill Meets Hugging Face Libraries for Reproducible, Coding-Free Deep Learning Studies: A Case Study on NLP](https://mlsanity.com/paper/2310.17644/) (2023)
- [A Reproducibility Study of LLM-Based Query Reformulation](https://mlsanity.com/paper/2604.27421/) (2026)
- [An Evaluation Study of Generative Adversarial Networks for Collaborative Filtering](https://mlsanity.com/paper/2201.01815/) (2022)
- [ReFormeR: Learning and Applying Explicit Query Reformulation Patterns](https://mlsanity.com/paper/2604.01417/) (2026)
- [Mirror Matching: Document Matching Approach in Seed-driven Document Ranking for Medical Systematic Reviews](https://mlsanity.com/paper/2112.14318/) (2021)
- [Reproducibility and Artifact Consistency of the SIGIR 2022 Recommender Systems Papers Based on Message Passing](https://mlsanity.com/paper/2503.07823/) (2025)
- [Fenced Citation-Context Retrieval for Case Law: Temporal Leakage and Degree Control Across Two Jurisdictions](https://mlsanity.com/paper/2607.17142/) (2026)

## Abstract (arXiv)

Is neural IR mostly hype? In a recent SIGIR Forum article, Lin expressed skepticism that neural ranking models were actually improving ad hoc retrieval effectiveness in limited data scenarios. He provided anecdotal evidence that authors of neural IR papers demonstrate "wins" by comparing against weak baselines. This paper provides a rigorous evaluation of those claims in two ways: First, we conducted a meta-analysis of papers that have reported experimental results on the TREC Robust04 test collection. We do not find evidence of an upward trend in effectiveness over time. In fact, the best reported results are from a decade ago and no recent neural approach comes close. Second, we applied five recent neural models to rerank the strong baselines that Lin used to make his arguments. A significant improvement was observed for one of the models, demonstrating additivity in gains. While there appears to be merit to neural IR approaches, at least some of the gains reported in the literature appear illusory.