# Efficient Extractive Summarization with MAMBA-Transformer Hybrids for Low-Resource Scenarios

arXiv: https://arxiv.org/abs/2603.01288 · HTML: https://mlsanity.com/paper/2603.01288/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 15 of its 23 references that are in the map.
- First posted on arXiv 1 Mar 2026.

## Placed in

- State-space sequence models (S4, Mamba) / [Core SSM architectures for language](https://mlsanity.com/ssm/core-ssm-architectures-for-language/)

## Builds on (most cited first, 15 of 15)

- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018)
- [Extractive Summarization as Text Matching](https://mlsanity.com/paper/2004.08795/) (2020)
- [Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning](https://mlsanity.com/paper/2002.08307/) (2020)
- [Fine-tune BERT for Extractive Summarization](https://mlsanity.com/paper/1903.10318/) (2019)
- [Sequence Length is a Domain: Length-based Overfitting in Transformer Models](https://mlsanity.com/paper/2109.07276/) (2021)
- [DebateSum: A large-scale argument mining and summarization dataset](https://mlsanity.com/paper/2011.07251/) (2020)
- [DiffuSum: Generation Enhanced Extractive Summarization with Diffusion](https://mlsanity.com/paper/2305.01735/) (2023)
- [Fair Abstractive Summarization of Diverse Perspectives](https://mlsanity.com/paper/2311.07884/) (2023)
- [Alleviating Exposure Bias via Contrastive Learning for Abstractive Text Summarization](https://mlsanity.com/paper/2108.11846/) (2021)
- [State Space Models are Strong Text Rerankers](https://mlsanity.com/paper/2412.14354/) (2024)
- [COLO: A Contrastive Learning based Re-ranking Framework for One-Stage Summarization](https://mlsanity.com/paper/2209.14569/) (2022)
- [How Effective are State Space Models for Machine Translation?](https://mlsanity.com/paper/2407.05489/) (2024)
- [Enhancing Abstractive Summarization of Scientific Papers Using Structure Information](https://mlsanity.com/paper/2505.14179/) (2025)

## Abstract (arXiv)

Extractive summarization of long documents is bottlenecked by quadratic complexity, often forcing truncation and limiting deployment in resource-constrained settings. We introduce the first Mamba-Transformer hybrid for extractive summarization, combining the semantic strength of pre-trained transformers with the linear-time processing of state space models. Leveraging Mamba's ability to process full documents without truncation, our approach preserves context while maintaining strong summarization quality. The architecture includes: (1) a transformer encoder for sentence-level semantics, (2) a Mamba state space model to capture inter-sentence dependencies efficiently, and (3) a linear classifier for sentence relevance prediction. Across news, argumentative, and scientific domains under low-resource conditions, our method achieves: (1) large gains over BERTSUM and MATCHSUM, including +0.23 ROUGE-1 on ArXiv and statistically significant improvements on all datasets (p < 0.001); (2) consistent advantages across domains, strongest on the longest documents; (3) robust performance with limited training data; and (4) 24-27% faster inference on news summarization (CNN/DailyMail). We introduce the first hybrid Transformer-state space architecture for summarization, showing significant ROUGE improvements in low-resource scenarios.