# ReSSFormer: A Recursive Sparse Structured Transformer for Scalable and Long-Context Reasoning

arXiv: https://arxiv.org/abs/2510.01585 · HTML: https://mlsanity.com/paper/2510.01585/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 32 of its 37 references that are in the map.
- First posted on arXiv 2 Oct 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 30 of 32)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentation](https://mlsanity.com/paper/2409.05591/) (2024)
- [CoLT5: Faster Long-Range Transformers with Conditional Computation](https://mlsanity.com/paper/2303.09752/) (2023)
- [Frequency-Aligned Knowledge Distillation for Lightweight Spatiotemporal Forecasting](https://mlsanity.com/paper/2507.02939/) (2025)
- [Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers](https://mlsanity.com/paper/2406.16747/) (2024)
- [On the Stability of Expressive Positional Encodings for Graphs](https://mlsanity.com/paper/2310.02579/) (2023)
- [SepPrune: Structured Pruning for Efficient Deep Speech Separation](https://mlsanity.com/paper/2505.12079/) (2025)
- [SGLP: A Similarity Guided Fast Layer Partition Pruning for Compressing Large Deep Models](https://mlsanity.com/paper/2410.14720/) (2024)
- [Enhancing Graph Transformers with Hierarchical Distance Structural Encoding](https://mlsanity.com/paper/2308.11129/) (2023)
- [COMAE: COMprehensive Attribute Exploration for Zero-shot Hashing](https://mlsanity.com/paper/2402.16424/) (2024)
- [Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing](https://mlsanity.com/paper/2505.00315/) (2025)
- [Syntax-aware Neural Semantic Role Labeling](https://mlsanity.com/paper/1907.09312/) (2019)
- [Modular MeanFlow: Towards Stable and Scalable One-Step Generative Modeling](https://mlsanity.com/paper/2508.17426/) (2025)
- [MOVER: Multimodal Optimal Transport with Volume-based Embedding Regularization](https://mlsanity.com/paper/2508.12149/) (2025)
- [Efficient Sparse Attention needs Adaptive Token Release](https://mlsanity.com/paper/2407.02328/) (2024)
- [Fovea Transformer: Efficient Long-Context Modeling with Structured Fine-to-Coarse Attention](https://mlsanity.com/paper/2311.07102/) (2023)

## Abstract (arXiv)

While Transformer architectures have demonstrated impressive scalability across domains, they continue to face challenges in long-context reasoning, computational efficiency, and structural generalization - largely due to rigid layer stacking, dense attention, and reliance on positional encodings. We present ReSSFormer, a Recursive Sparse Structured Transformer that integrates three complementary innovations: Recurrent Reasoning & Memory Unit (R2MU) for iterative reasoning with bounded depth, Adaptive Sparse Attention Module (ASAM) for efficient and focused context selection, and Self-Organizing Encoder Structure (SOES) for position-free structure induction. ReSSFormer replaces conventional depth stacking with recurrent inference, substitutes full attention with token- and expert-level sparsity, and models latent token topology directly from content. Across language modeling, multi-hop QA, and structure-sensitive tasks, ReSSFormer consistently outperforms strong baselines under comparable FLOPs and parameter budgets, highlighting its scalability, efficiency, and structural flexibility.