mlsanityData to 14 Oct 2026

SELF: Self-Extend the Context Length With Logistic Growth Function

arXiv 2505.17296 ↗First posted 22 May 2025cs.CL cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Large language models suffer issues when operated on long contexts that are larger than their training context length due to the standard position encoding for tokens in the attention layer. Tokens a long distance apart will rarely have an effect on each other and long prompts yield unexpected results. To solve this problem, we propose SELF (Self-Extend the Context Length With Logistic Growth Function): a solution of grouping consecutive tokens at varying group sizes using a logistic capacity equation combined with a constant group size at smaller relative distances. Our model had an increase in performance of up to 12% compared to the LongLM extension method in LEval (specifically on the Qwen model). On summarization related tasks in LongBench, our model performed up to 6.4% better than LongLM (specifically on the Llama-2-7b model). On reading comprehension tasks from LEval, our model performed up to 5.4% better than the LongLM. Our code is available at https://github.com/alexeipc/SELF-LLM.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 18 of its 27 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

18 of 27 references.

References · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  6. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  9. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  10. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  11. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  12. 2021Recent Advances in Adversarial Training for Adversarial Robustnesscited by 279
6 more
  1. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  2. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
  3. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  4. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 136
  5. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 111
  6. 2022When Neural Networks Fail to Generalize? A Model Sensitivity Perspectivecited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPEcited by 0