mlsanityData to 7 Oct 2026

Contextual Position Encoding: Learning to Count What's Important

arXiv 2405.18719 ↗First posted 29 May 2024cs.CL cs.AIcited by 53

Not on the map. Why?

Abstract (arXiv)

The attention mechanism is a critical component of Large Language Models (LLMs) that allows tokens in a sequence to interact with each other, but is order-invariant. Incorporating position encoding (PE) makes it possible to address by position, such as attending to the i-th token. However, current PE methods use token counts to derive position, and thus cannot generalize to higher levels of abstraction, such as attending to the i-th sentence. In this paper, we propose a new position encoding method, Contextual Position Encoding (CoPE), that allows positions to be conditioned on context by incrementing position only on certain tokens determined by the model. This allows more general position addressing such as attending to the i-th particular word, noun, or sentence. We show that CoPE can solve the selective copy, counting and Flip-Flop tasks where popular position embeddings fail, and improves perplexity on language modeling and coding tasks.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Builds on 18 of its 23 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

18 of 23 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  11. 2018Self-Attention with Relative Position Representationscited by 1,112
  12. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
6 more
  1. 2017Convolutional Sequence to Sequence Learningcited by 550
  2. 2014Memory Networkscited by 354
  3. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 148
  4. 2021Position Information in Transformers: An Overviewcited by 103
  5. 2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 74
  6. 2019R-Transformer: Recurrent Neural Network Enhanced Transformercited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 26; 2026: 12.

53 papers.

Sort:

By area:

  1. 2025Cameras as Relative Positional Encodingcited by 94
  2. 2025Any2AnyTryon: Leveraging Adaptive Position Embeddings for Versatile Virtual Clothing Taskscited by 62
  3. 2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 61
  4. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
  5. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mambacited by 45
  6. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 43
  7. 2024Rethinking Visual Dependency in Long-Context Reasoning for Large Vision-Language Modelscited by 41
  8. 2025PaTH Attention: Position Encoding via Accumulating Householder Transformationscited by 32
  9. 2024LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Modelscited by 23
  10. 2024TULIP: Token-length Upgraded CLIPcited by 18
  11. 2024HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolationcited by 17
  12. 2024LieRE: Lie Rotational Positional Encodingscited by 16
38 more
  1. 2025Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Modelscited by 14
  2. 2025ParallelComp: Parallel Long-Context Compressor for Length Extrapolationcited by 13
  3. 2025Sample Complexity and Representation Ability of Test-time Scaling Paradigmscited by 12
  4. 2026Addressable Memory for Video World Modelscited by 11
  5. 2024DAPE V2: Process Attention Score as Feature Map for Length Extrapolationcited by 10
  6. 2025Multi-Token Attentioncited by 10
  7. 2025HoPE: Hyperbolic Rotary Positional Encoding for Stable Long-Range Dependency Modeling in Large Language Modelscited by 10
  8. 2024PositionID: LLMs can Control Lengths, Copy and Paste with Explicit Positional Awarenesscited by 9
  9. 2025RiemannFormer: A Framework for Attention in Curved Spacescited by 8
  10. 2025Long Context Scaling: Divide and Conquer via Multi-Agent Question-driven Collaborationcited by 7
  11. 2025Group Representational Position Encodingcited by 5
  12. 2025Rethinking Addressing in Language Models via Contexualized Equivariant Positional Encodingcited by 4
  13. 2025RAGSmith: A Framework for Finding the Optimal Composition of Retrieval-Augmented Generation Methods Across Datasetscited by 4
  14. 2024Signformer is all you need: Towards Edge AI for Sign Languagecited by 3
  15. 2025LEDiT: Your Length-Extrapolatable Diffusion Transformer without Positional Encodingcited by 3
  16. 2025A 2D Semantic-Aware Position Encoding for Vision Transformerscited by 3
  17. 2025RePo: Language Models with Context Re-Positioningcited by 3
  18. 2024Algorithmic Language Models with Neurally Compiled Librariescited by 2
  19. 2025Context-aware Biases for Length Extrapolationcited by 2
  20. 2025Latent Multi-Head Attention for Small Language Modelscited by 2
  21. 2025GeoPE:A Unified Geometric Positional Embedding for Structured Tensorscited by 2
  22. 2025Beyond Real: Imaginary Extension of Rotary Position Embeddings for Long-Context LLMscited by 2
  23. 2024Learning interpretable positional encodings in transformers depends on initializationcited by 1
  24. 2025TRA: Better Length Generalisation with Threshold Relative Attentioncited by 1
  25. 2026LeRoPE: Learnable RoPE Frequencies Improve Language Modelingcited by 1
  26. 2026Training Trajectories Determine Circuit Removability in Annealable Soft-Prior Transformerscited by 1
  27. 2024Optimizing Encoder-Only Transformers for Session-Based Recommendation Systemscited by 0
  28. 2024Disentanglement and Compositionality of Letter Identity and Letter Position in Variational Auto-Encoder Vision Modelscited by 0
  29. 2025Softplus Attention with Re-weighting Boosts Length Extrapolation in Large Language Modelscited by 0
  30. 2025LLMs are Bayesian, In Expectation, Not in Realizationcited by 0
  31. 2025NDLPNet: A Location-Aware Nighttime Deraining Network and a Real-World Benchmark Datasetcited by 0
  32. 2025MapFormer: Self-Supervised Learning of Cognitive Maps with Input-Dependent Positional Embeddingscited by 0
  33. 2026GRAB: An LLM-Inspired Sequence-First Click-Through Rate Prediction Modeling Paradigmcited by 0
  34. 2026Three-Phase Transformercited by 0
  35. 2026Dual Triangle Attention: Effective Bidirectional Attention Without Positional Embeddingscited by 0
  36. 2026Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoderscited by 0
  37. 2026Universal Transformers for Circuit Computations: Perfect Length Generalization in Tiny Transformerscited by 0
  38. 2026Content-Based Addressing for Long Contextcited by 0

50 most cited of 53 All 53 (JSON)