mlsanityData to 14 Oct 2026

State Space Models are Strong Text Rerankers

arXiv 2412.14354 ↗First posted 18 Dec 2024cs.CL cs.IRcited by 10

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

Transformers dominate NLP and IR; but their inference inefficiencies and challenges in extrapolating to longer contexts have sparked interest in alternative model architectures. Among these, state space models (SSMs) like Mamba offer promising advantages, particularly O(1) time complexity in inference. Despite their potential, SSMs' effectiveness at text reranking -- a task requiring fine-grained query-document interaction and long-context understanding -- remains underexplored. This study benchmarks SSM-based architectures (specifically, Mamba-1 and Mamba-2) against transformer-based models across various scales, architectures, and pre-training objectives, focusing on performance and efficiency in text reranking tasks. We find that (1) Mamba architectures achieve competitive text ranking performance, comparable to transformer-based models of similar size; (2) they are less efficient in training and inference compared to transformers with flash attention; and (3) Mamba-2 outperforms Mamba-1 in both performance and efficiency. These results underscore the potential of state space models as a transformer alternative and highlight areas for improvement in future IR applications.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 64 of its 89 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

64 of 89 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Models27 Aug
  2. 2024Jamba-1.5: Hybrid Transformer-Mamba Models at Scale22 Aug
  3. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
  4. 2024Jamba: A Hybrid Transformer-Mamba Language Model28 Mar
  5. 2024RankMamba: Benchmarking Mamba's Document Ranking Performance in the Era of Transformers27 Mar
  6. 2023Gated Linear Attention Transformers with Hardware-Efficient Training11 Dec
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  8. 2022Simplified State Space Layers for Sequence Modeling9 Aug
  9. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 55

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  11. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  12. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
38 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  3. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  4. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  5. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  6. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  7. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  8. 2019Passage Re-ranking with BERTcited by 645
  9. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 608
  10. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 605
  11. 2020Fact or Fiction: Verifying Scientific Claimscited by 554
  12. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 463
  13. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 430
  14. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  15. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 376
  16. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 361
  17. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  18. 2020Overview of the TREC 2019 deep learning trackcited by 286
  19. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 272
  20. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 267
  21. 2019Document Expansion by Query Predictioncited by 231
  22. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  23. 2020CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claimscited by 196
  24. 2021Overview of the TREC 2020 deep learning trackcited by 196
  25. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 127
  26. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 124
  27. 2022RankT5: Fine-Tuning T5 for Text Ranking with Ranking Lossescited by 115
  28. 2019Deeper Text Understanding for IR with Contextual Neural Language Modelingcited by 113
  29. 2022SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrievalcited by 90
  30. 2021The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Modelscited by 81
  31. 2024OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMscited by 72
  32. 2021Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipelinecited by 65
  33. 2020PARADE: Passage Representation Aggregation for Document Rerankingcited by 44
  34. 2024Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compressioncited by 33
  35. 2022Exploring The Landscape of Distributional Robustness for Question Answering Modelscited by 23
  36. 2024In-Context Example Ordering Guided by Label Distributionscited by 12
  37. 2023Context-aware Decoding Reduces Hallucination in Query-focused Summarizationcited by 11
  38. 2021Intra-Document Cascading: Learning to Select Passages for Neural Document Rankingcited by 10

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 5.

10 papers.

  1. 2024MAIN-RAG: Multi-Agent Filtering Retrieval-Augmented Generationcited by 44
  2. 2025CSPLADE: Learned Sparse Retrieval with Causal Language Modelscited by 10
  3. 2025Distillation versus Contrastive Learning: How to Train Your Rerankerscited by 6
  4. 2026LACONIC: Dense-Level Effectiveness for Scalable Sparse Retrieval via a Two-Phase Training Curriculumcited by 5
  5. 2025RECON: Reasoning with Condensation for Efficient Retrieval-Augmented Generationcited by 3
  6. 2025Do You Really Need a GPU to Guard Your LLM? CPU-Class Classifiers and Multi-Stage Pipelines for Safety Enforcement at Scalecited by 2
  7. 2026Towards Understanding What State Space Models Learn About Codecited by 1
  8. 2026Tevatron Meets Megatron: Expert-Parallel LLM Reranker Training on an Academic Budgetcited by 1
  9. 2026Efficient Extractive Summarization with MAMBA-Transformer Hybrids for Low-Resource Scenarioscited by 0
  10. 2026Tevatron-Elastic: A Unified Abstraction for Training Elastic Retrievers and Rerankerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024How Effective are State Space Models for Machine Translation?
  2. 2025ProRank: Prompt Warmup via Reinforcement Learning for Small Language Models Reranking
  3. 2025Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanism
  4. 2024Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers
  5. 2025The Evolution of Reranking Models in Information Retrieval: From Heuristic Methods to Large Language Models
  6. 2026Think Thrice Before Reranking: Multi-perspective Evidence and Reasoning Integration for Text Reranking
  7. 2026Retrievit: In-context Retrieval Capabilities of Transformers, State Space Models, and Hybrid Architectures
  8. 2024JudgeRank: Leveraging Large Language Models for Reasoning-Intensive Reranking