mlsanityData to 14 Oct 2026

ReSSFormer: A Recursive Sparse Structured Transformer for Scalable and Long-Context Reasoning

arXiv 2510.01585 ↗First posted 2 Oct 2025cs.CL cs.NIcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

While Transformer architectures have demonstrated impressive scalability across domains, they continue to face challenges in long-context reasoning, computational efficiency, and structural generalization - largely due to rigid layer stacking, dense attention, and reliance on positional encodings. We present ReSSFormer, a Recursive Sparse Structured Transformer that integrates three complementary innovations: Recurrent Reasoning & Memory Unit (R2MU) for iterative reasoning with bounded depth, Adaptive Sparse Attention Module (ASAM) for efficient and focused context selection, and Self-Organizing Encoder Structure (SOES) for position-free structure induction. ReSSFormer replaces conventional depth stacking with recurrent inference, substitutes full attention with token- and expert-level sparsity, and models latent token topology directly from content. Across language modeling, multi-hop QA, and structure-sensitive tasks, ReSSFormer consistently outperforms strong baselines under comparable FLOPs and parameter budgets, highlighting its scalability, efficiency, and structural flexibility.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 32 of its 37 references that are in the map. First posted on arXiv 2 Oct 2025.

Builds on

32 of 37 references.

References · 32

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  5. 2020Longformer: The Long-Document Transformercited by 3,413
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  9. 2020Reformer: The Efficient Transformercited by 1,668
  10. 2020Rethinking Attention with Performerscited by 1,601
  11. 2018Self-Attention with Relative Position Representationscited by 1,120
  12. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
20 more
  1. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  2. 2020TabTransformer: Tabular Data Modeling Using Contextual Embeddingscited by 409
  3. 2022Multimodal Learning with Transformers: A Surveycited by 400
  4. 2024MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentationcited by 119
  5. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  6. 2025Frequency-Aligned Knowledge Distillation for Lightweight Spatiotemporal Forecastingcited by 55
  7. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 52
  8. 2023On the Stability of Expressive Positional Encodings for Graphscited by 33
  9. 2025SepPrune: Structured Pruning for Efficient Deep Speech Separationcited by 31
  10. 2024SGLP: A Similarity Guided Fast Layer Partition Pruning for Compressing Large Deep Modelscited by 25
  11. 2023Enhancing Graph Transformers with Hierarchical Distance Structural Encodingcited by 16
  12. 2024COMAE: COMprehensive Attribute Exploration for Zero-shot Hashingcited by 16
  13. 2025Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routingcited by 14
  14. 2019Syntax-aware Neural Semantic Role Labelingcited by 8
  15. 2025Modular MeanFlow: Towards Stable and Scalable One-Step Generative Modelingcited by 8
  16. 2025MOVER: Multimodal Optimal Transport with Volume-based Embedding Regularizationcited by 6
  17. 2024Efficient Sparse Attention needs Adaptive Token Releasecited by 4
  18. 2023Fovea Transformer: Efficient Long-Context Modeling with Structured Fine-to-Coarse Attentioncited by 2
  19. 2025Enhancing RWKV-based Language Models for Long-Sequence Text Generationcited by 1
  20. 2025TRA: Better Length Generalisation with Threshold Relative Attentioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026T^2MLR: Transformer with Temporal Middle-Layer Recurrence
  2. 2026Penelope: Localized Latent Recurrence for Efficient Structured Reasoning
  3. 2026CausalGate: Causal Importance Distillation for Transformer Module Pruning
  4. 2026LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation
  5. 2025TreeGPT: Pure TreeFFN Encoder-Decoder Architecture for Structured Reasoning Without Attention Mechanisms
  6. 2026Think Deep, Speak Once: Relit, A Recursive Latent Implicit Transformer Framework
  7. 2026Beyond Dense States: Sparse Transcoders as Causally Testable Operators for LLM Latent Reasoning
  8. 2025Hydra: A Modular Architecture for Efficient Long-Context Reasoning