Lost in State Space: Probing Frozen Mamba Representations
On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language
Abstract (arXiv)
Mamba's recurrent state h_t is, by construction, a compressed summary of every token seen so far. This raises a tempting hypothesis: if we extract token-level outputs y_t at fixed patch boundaries, we obtain semantic sentence summaries for free, with no pooling head, no fine-tuning, and no [CLS] token. We test this hypothesis carefully. Across five benchmarks (SST-2, CoLA, MRPC, STS-B, IMDb), we compare four strategies for extracting frozen sentence representations from a pretrained Mamba-130M backbone under a strict frozen-feature probing protocol, using three random seeds where computationally feasible. The results do not support the hypothesis: patch boundary readouts do not consistently outperform simple mean pooling. We identify and quantify two structural pathologies: severe anisotropy (mean pairwise cosine similarity 0.9999, std 0.000044) and representational collapse in the raw final SSM state (MCC = 0.000 on CoLA across all three seeds, confirmed via confusion matrix). We further propose orthogonal injection, a modified recurrence that constrains new information per
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 13 of its 15 references that are in the map. First posted on arXiv 30 Apr 2026.
Builds on
13 of 15 references.
Closest ancestors · 4
Same direction, newest first
- 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
- 2024Jamba: A Hybrid Transformer-Mamba Language Model28 Mar
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
- 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct
Other references · 9
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
- 2016Understanding intermediate layers using linear classifier probescited by 1,279
- 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
- 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
- 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
- 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 250
- 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 205
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025LaTIM: Measuring Latent Token-to-Token Interactions in Mamba Models
- 2025MemMamba: Rethinking Memory Patterns in State Space Model
- 2024ReMamba: Equip Mamba with Effective Long-Sequence Modeling
- 2026Residual Streams Read, Recurrent States Remember: The Global Workspace in Mamba Models
- 2024MambaLRP: Explaining Selective State Space Sequence Models
- 2025LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargement
- 2025Characterizing Mamba's Selective Memory using Auto-Encoders
- 2024Sparse Mamba: Introducing Controllability, Observability, And Stability To Structural State Space Models