mlsanityData to 14 Oct 2026

The Mask Is Not the Model: Auditing Prefix Invariance in Attention, State-Space, and Hybrid Sequence Models

arXiv 2608.22876 ↗First posted 24 Aug 2026cs.LG cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Hybrid sequence models must satisfy prefix invariance: representations at position t must not depend on future inputs, yet this is rarely verified. We formalize prefix invariance and give a lightweight audit, two forward passes, no training or gradients, yielding a per-layer score localizing where causality breaks. Attention-mask inspection, the field's default check, is incomplete: causality is a graph-level property, and leaks can occur via scans, aggregations, or normalization despite correct masks. Across 192 injected-fault trials on eight checkpoints, mask inspection detected none, while our audit localized all 192/192 to the exact layer. Static/dynamic analysis of chunked-scan code in transformers found the same defect in Zamba2 and Nemotron-H, an inter-chunk axis error fixed via the reference implementation. The method fits on one page and runs in seconds.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 26 of its 40 references that are in the map. First posted on arXiv 24 Aug 2026.

Builds on

26 of 40 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2020Longformer: The Long-Document Transformercited by 3,413
  9. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  11. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  12. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
14 more
  1. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  2. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  3. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  4. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  5. 2020Efficient Transformers: A Surveycited by 790
  6. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  7. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
  8. 2017DeepXplore: Automated Whitebox Testing of Deep Learning Systemscited by 311
  9. 2017DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Carscited by 251
  10. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 242
  11. 2019Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approachescited by 185
  12. 2019A Troubling Analysis of Reproducibility and Progress in Recommender Systems Researchcited by 65
  13. 2024RecurrentGemma: Moving Past Transformers for Efficient Open Language Modelscited by 64
  14. 2019Deep Neural Network Ensembles for Time Series Classificationcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Beyond Accuracy: Prefix-Invariant Realizations of Low-Precision Fast Matrix Multiplicationcited by 0