mlsanityData to 7 Oct 2026

LongCoder: A Long-Range Pre-trained Language Model for Code Completion

arXiv 2306.14893 ↗First posted 26 Jun 2023cs.SE cs.AI cs.CLcited by 110

Not on the map. Why?

Cited by 31 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we introduce a new task for code completion that focuses on handling long code input and propose a sparse Transformer model, called LongCoder, to address this task. LongCoder employs a sliding window mechanism for self-attention and introduces two types of globally accessible tokens - bridge tokens and memory tokens - to improve performance and efficiency. Bridge tokens are inserted throughout the input sequence to aggregate local information and facilitate global interaction, while memory tokens are included to highlight important statements that may be invoked later and need to be memorized, such as package imports and definitions of classes, functions, or structures. We conduct experiments on a newly constructed dataset that contains longer code context and the publicly available CodeXGLUE benchmark. Experimental results demonstrate that LongCoder achieves superior performance on code completion tasks compared to previous models while maintaining comparable efficiency in terms of computational resources during inference. All the codes and data are available at https://github.com/microsoft/CodeBERT.

Key facts. Cited by 110 papers in the map (data through 7 Oct 2026). Builds on 26 of its 35 references that are in the map. First posted on arXiv 26 Jun 2023.

Builds on

26 of 35 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  8. 2020Reformer: The Efficient Transformercited by 1,658
  9. 2020Rethinking Attention with Performerscited by 1,584
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  12. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
14 more
  1. 2020Efficient Transformers: A Surveycited by 784
  2. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  3. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  4. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  5. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  6. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  7. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
  8. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  9. 2020IntelliCode Compose: Code Generation Using Transformercited by 225
  10. 2022cosFormer: Rethinking Softmax in Attentioncited by 212
  11. 2018The Adverse Effects of Code Duplication in Machine Learning Models of Codecited by 123
  12. 2020Multi-task Learning based Pre-trained Language Model for Code Completioncited by 58
  13. 2014Mining Idioms from Source Codecited by 19
  14. 2021Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchycited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 32; 2025: 40; 2026: 31.

110 papers.

Sort:

By area:

  1. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  3. 2024A Survey on Large Language Models for Code Generationcited by 646
  4. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  5. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  6. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  7. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 230
  8. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  9. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  10. 2025SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agentscited by 80
  11. 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains Morecited by 69
  12. 2024Palu: Compressing KV-Cache with Low-Rank Projectioncited by 59
38 more
  1. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 57
  2. 2024Self-evolving Agents with reflective and memory-augmented abilitiescited by 52
  3. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 51
  4. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 50
  5. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 48
  6. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  7. 2024DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMscited by 42
  8. 2026SWE-Pruner: Self-Adaptive Context Pruning for Coding Agentscited by 40
  9. 2025LongCodeZip: Compress Long Context for Code Language Modelscited by 38
  10. 2024An LLM-Assisted Easy-to-Trigger Backdoor Attack on Code Completion Models: Injecting Disguised Vulnerabilities against Strong Detectioncited by 32
  11. 2024LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Acceleratorscited by 31
  12. 2024Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLPcited by 27
  13. 2023LLM for SoC Security: A Paradigm Shiftcited by 22
  14. 2024Are Long-LLMs A Necessity For Long-Context Tasks?cited by 22
  15. 2024Performance-Aligned LLMs for Generating Fast Codecited by 20
  16. 2025LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verificationcited by 19
  17. 2025Code Researcher: Deep Research Agent for Large Systems Code and Commit Historycited by 19
  18. 2025SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruningcited by 19
  19. 2024AMXFP4: Taming Activation Outliers with Asymmetric Microscaling Floating-Point for 4-bit LLM Inferencecited by 17
  20. 2025LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocationcited by 16
  21. 2024Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Modelscited by 15
  22. 2025Taming the Fragility of KV Cache Eviction in LLM Inferencecited by 14
  23. 2024E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoningcited by 13
  24. 2025xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extractioncited by 13
  25. 2025Mamba Modulation: On the Length Generalization of Mambacited by 13
  26. 2026Seeing is Coding: On the Effectiveness of Vision Language Models in Code Understandingcited by 13
  27. 2026PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defensescited by 13
  28. 2023Nova: Generative Language Models for Assembly Code with Hierarchical Attention and Contrastive Learningcited by 12
  29. 2025A Survey on Transformer Context Extension: Approaches and Evaluationcited by 11
  30. 2024More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compressioncited by 10
  31. 2025Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inferencecited by 10
  32. 2025Towards an Understanding of Context Utilization in Code Intelligencecited by 10
  33. 2025Alignment-Augmented Speculative Decoding with Alignment Sampling and Conditional Verificationcited by 10
  34. 2025EvolKV: Evolutionary KV Cache Compression for LLM Inferencecited by 10
  35. 2026PIArena: A Platform for Prompt Injection Evaluationcited by 10
  36. 2023Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolationcited by 9
  37. 2024Scaling Granite Code Models to 128K Contextcited by 9
  38. 2025SwiftSpec: Ultra-Low Latency LLM Decoding by Scaling Asynchronous Speculative Decodingcited by 9

50 most cited of 110 All 110 (JSON)