mlsanityData to 7 Oct 2026

CodeSSM: Towards State Space Models for Code Understanding

arXiv 2505.01475 ↗First posted 2 May 2025cs.SE cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Although transformers dominate many code-specific tasks, they have significant limitations. This paper explores State Space Models (SSMs) as a promising alternative for code understanding tasks such as retrieval, classification, and clone detection. We introduce CodeSSM, the first SSM-based model trained on code corpora to assess its effectiveness. Our results demonstrate that SSMs are more sample-efficient and can extrapolate to longer contexts beyond the pretraining length. Extensive experiments show that SSMs offer a viable alternative to transformers, addressing several their limitations. Additionally, CodeSSM reduces memory usage by up to 64% compared to transformers at a context length of 2048, with greater savings as context length grows.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 41 of its 63 references that are in the map. First posted on arXiv 2 May 2025.

Builds on

41 of 63 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  10. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  11. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  12. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
29 more
  1. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  2. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  3. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  4. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  5. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
  6. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  7. 2019Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networkscited by 266
  8. 2024The Illusion of State in State-Space Modelscited by 181
  9. 2023LongCoder: A Long-Range Pre-trained Language Model for Code Completioncited by 110
  10. 2022The Devil in Linear Transformercited by 98
  11. 2023DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detectioncited by 98
  12. 2020Detecting Code Clones with Graph Neural Networkand Flow-Augmented Abstract Syntax Treecited by 68
  13. 2024Iterative Refinement of Project-Level Code Context for Precise Code Generation with Compiler Feedbackcited by 45
  14. 2024Dataflow-Guided Retrieval Augmentation for Repository-Level Code Completioncited by 40
  15. 2024Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalizationcited by 37
  16. 2023StableSSM: Alleviating the Curse of Memory in State-space Models through Stable Reparameterizationcited by 29
  17. 2021Are Pre-trained Convolutions Better than Pre-trained Transformers?cited by 27
  18. 2023Fortify the Shortest Stave in Attention: Enhancing Context Awareness of Large Language Models for Effective Tool Usecited by 27
  19. 2024STALL+: Boosting LLM-based Repository-level Code Completion with Static Analysiscited by 21
  20. 2024Do Efficient Transformers Really Save Computation?cited by 20
  21. 2024Mixture of In-Context Experts Enhance LLMs' Long Context Awarenesscited by 17
  22. 2022LSG Attention: Extrapolation of pretrained Transformers to long sequencescited by 16
  23. 2021Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchycited by 14
  24. 2024LOCOST: State-Space Models for Long Document Abstractive Summarizationcited by 8
  25. 2024HiRoPE: Length Extrapolation for Code Models Using Hierarchical Positioncited by 8
  26. 2024SMR: State Memory Replay for Long Sequence Modelingcited by 6
  27. 2024State Space Models are Provably Comparable to Transformers in Dynamic Token Selectioncited by 5
  28. 2024Quantifying Positional Biases in Text Embedding Modelscited by 4
  29. 2024ContextModule: Improving Code Completion via Repository-level Contextual Informationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Analysis of Long Range Dependency Understanding in State Space Modelscited by 1
  2. 2026Towards Understanding What State Space Models Learn About Codecited by 1
  3. 2026Beyond the Autoregressive Horizon: A Comprehensive Survey of Diffusion Models, World Modelling, and State Space Models for Codecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs
  2. 2026Are you Synthesizing or Recalling? Evaluating LLMs on Algorithmic Code Retrieval
  3. 2026CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval
  4. 2026Seeing is Coding: On the Effectiveness of Vision Language Models in Code Understanding
  5. 2024CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrieval
  6. 2025Characterizing Mamba's Selective Memory using Auto-Encoders
  7. 2026Memorization Diagnostics for Code LLMs Should be Scale-Aware
  8. 2025Functional Consistency of LLM Code Embeddings: A Self-Evolving Data Synthesis Framework for Benchmarking