mlsanityData to 7 Oct 2026

TinyCeNN-LM: Quality-Gated Conversion of Pretrained Attention with CeNN-Inspired Cellular-Recurrent Layers

arXiv 2609.21139 ↗First posted 17 Sep 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Replacing attention in a pretrained language model is a compatibility problem: a plausible substitute may alter representations expected by later layers. TinyCeNN-LM introduces a quality-gated post-training conversion framework using CeNN-inspired cellular-recurrent layers with bounded local processing, compact recurrent memory, routing, fusion, and accept-or-rollback validation. Three implementations are studied: Integrated Memory, MemoryFusion, and PDelta3-GDN2-CLVR+Local32. Strict PDelta3 conversion accepts a layer only when representation and NLL criteria pass fixed thresholds. On SmolLM2-135M, layers 0-2 are accepted with cumulative DeltaNLL=+0.01209, while layer 3 is rejected despite acceptable NLL because representation fidelity fails. On Qwen3.5-0.8B, full-attention layers 3, 7, and 11 are accepted with final DeltaNLL=+0.02073. Integrated Memory keeps perplexity within -0.07% to +0.93% while reducing total cache by up to 6.01%. A sampled 200-item downstream sanity check gives 28.5%--32.0% overall accuracy for converted Qwen releases. The results support conservative, quality-gated structural conversion rather than universal attention replacement or speedup.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 19 of its 22 references that are in the map. First posted on arXiv 17 Sep 2026.

Builds on

19 of 22 references.

References · 19

  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  4. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  5. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  6. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  7. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  8. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  9. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  10. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  11. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  12. 2024xLSTM: Extended Long Short-Term Memorycited by 320
7 more
  1. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 307
  2. 2024Titans: Learning to Memorize at Test Timecited by 303
  3. 2024Simple linear attention language models balance the recall-throughput tradeoffcited by 197
  4. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  5. 2024RecurrentGemma: Moving Past Transformers for Efficient Open Language Modelscited by 61
  6. 2025RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scalecited by 24
  7. 2025Distilling to Hybrid Attention Models via KL-Guided Layer Selectioncited by 17

Built on by

Not cited yet.