mlsanityData to 7 Oct 2026

Nautile-370M: Spectral Memory Meets Attention in a Small Reasoning Model

arXiv 2604.24809 ↗First posted 27 Apr 2026cs.LG cs.AIcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

We present Nautile-370M, a 371-million-parameter small language model designed for efficient reasoning under strict parameter and inference budgets. Nautile-370M uses a hybrid backbone in which two SeqCond Attention (SCA) layers, a linear-time spectral sequence operator inspired by SeqCondenser, alternate with one transformer layer. This design aims to retain the long-context efficiency and state-tracking benefits of structured sequential models while preserving the expressive token-to-token routing of attention. The model was trained on a single Cloud TPU v4-64 pod slice provided through the Google TPU Research Cloud (TRC) program; the subsequent reinforcement learning stage was carried out on a single NVIDIA DGX Spark. We prove that the SCA readout mechanism can exactly retrieve any individual token from the prefix summary and can reproduce any output of softmax attention as a special case, establishing that SCA is at least as expressive as full self-attention in the continuous limit. We also describe the training data pipeline and outline a reinforcement learning stage specialized for reasoning, verification, and response quality.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 16 of its 19 references that are in the map. First posted on arXiv 27 Apr 2026.

Builds on

16 of 19 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022STaR: Bootstrapping Reasoning With Reasoning28 Mar

Other references · 15

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  9. 2020GLU Variants Improve Transformercited by 1,724
  10. 2019Root Mean Square Layer Normalizationcited by 1,425
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  12. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
3 more
  1. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  2. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  3. 2026Embarrassingly Simple Self-Distillation Improves Code Generationcited by 36

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025DELTA: Dynamic Layer-Aware Token Attention for Efficient Long-Context Reasoning
  2. 2026ARM: Attention with Routed-Memory for Learnable Sparse Control
  3. 2026LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning
  4. 2025Pretraining with hierarchical memories: separating long-tail and common knowledge
  5. 2025Hydra: A Modular Architecture for Efficient Long-Context Reasoning
  6. 2025Breadcrumbs Reasoning: Memory-Efficient Reasoning with Compression Beacons
  7. 2025Reasoning Under 1 Billion: Memory-Augmented Reinforcement Learning for Large Language Models
  8. 2026MKA: Memory-Keyed Attention for Efficient Long-Context Reasoning