mlsanityData to 7 Oct 2026

An Evaluation of Context Length Extrapolation in Long Code via Positional Embeddings and Efficient Attention

arXiv 2602.21800 ↗First posted 25 Feb 2026cs.SE cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

The rapid advancement of large language models (LLMs) has led to a significant increase in automated tools in the software engineering, capable of performing various code-related tasks such as code generation, completion, and translation. Despite these advancements, its effectiveness is constrained by fixed context lengths, limiting its ability to generalize across long, domain-specific code sequences. To address this challenge, we investigate zero-shot, inference-only methods aimed at improving position encodings and optimizing attention mechanisms. Our goal is to provide a thorough analysis of current approaches that facilitate context length extrapolation in code, particularly in the context of long code completion tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 34 references that are in the map. First posted on arXiv 25 Feb 2026.

Builds on

23 of 34 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  8. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  9. 2018Self-Attention with Relative Position Representationscited by 1,112
  10. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  11. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  12. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
11 more
  1. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  2. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  3. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  4. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  5. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 114
  6. 2023LongCoder: A Long-Range Pre-trained Language Model for Code Completioncited by 110
  7. 2019Lightweight and Efficient End-to-End Speech Recognition Using Low-Rank Transformercited by 47
  8. 2021A Simple and Effective Positional Encoding for Transformerscited by 37
  9. 2021PermuteFormer: Efficient Relative Position Encoding for Long Sequencescited by 14
  10. 2023Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolationcited by 9
  11. 2024Selective Shot Learning for Code Explanationcited by 1

Built on by

Not cited yet.