mlsanityData to 7 Oct 2026

Long Context Alignment with Short Instructions and Synthesized Positions

arXiv 2405.03939 ↗First posted 7 May 2024cs.CLcited by 22

Not on the map. Why?

Abstract (arXiv)

Effectively handling instructions with extremely long context remains a challenge for Large Language Models (LLMs), typically necessitating high-quality long data and substantial computational resources. This paper introduces Step-Skipping Alignment (SkipAlign), a new technique designed to enhance the long-context capabilities of LLMs in the phase of alignment without the need for additional efforts beyond training with original data length. SkipAlign is developed on the premise that long-range dependencies are fundamental to enhancing an LLM's capacity of long context. Departing from merely expanding the length of input samples, SkipAlign synthesizes long-range dependencies from the aspect of positions indices. This is achieved by the strategic insertion of skipped positions within instruction-following samples, which utilizes the semantic structure of the data to effectively expand the context. Through extensive experiments on base models with a variety of context window sizes, SkipAlign demonstrates its effectiveness across a spectrum of long-context tasks. Particularly noteworthy is that with a careful selection of the base model and alignment datasets, SkipAlign with only 6B parameters achieves it's best performance and comparable with strong baselines like GPT-3.5-Turbo-16K on LongBench.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 24 of its 28 references that are in the map. First posted on arXiv 7 May 2024.

Builds on

24 of 28 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  8. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  9. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  10. 2024Yi: Open Foundation Models by 01.AIcited by 706
  11. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  12. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
12 more
  1. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  2. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  3. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  4. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  5. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  6. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  7. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  8. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  9. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 114
  10. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 109
  11. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 102
  12. 2023Peek Across: Improving Multi-Document Modeling via Cross-Document Question-Answeringcited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 8; 2026: 4.

22 papers.

Sort:

By area:

  1. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  2. 2025Context Length Alone Hurts LLM Performance Despite Perfect Retrievalcited by 118
  3. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 60
  4. 2024GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Modelscited by 42
  5. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 30
  6. 2025Chain-of-Thought Matters: Improving Long-Context Language Models with Reasoning Path Supervisioncited by 20
  7. 2024GATEAU: Selecting Influential Samples for Long Context Alignmentcited by 18
  8. 2024An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encodingcited by 16
  9. 2025Token Weighting for Long-Range Language Modelingcited by 14
  10. 2024DAPE V2: Process Attention Score as Feature Map for Length Extrapolationcited by 10
  11. 2024MDCure: A Scalable Pipeline for Multi-Document Instruction-Followingcited by 10
  12. 2024LOGO -- Long cOntext aliGnment via efficient preference Optimizationcited by 9
10 more
  1. 2025CLIPPER: Compression enables long-context synthetic data generationcited by 9
  2. 2025WildLong: Synthesizing Realistic Long-Context Instruction Data at Scalecited by 9
  3. 2025LongAttn: Selecting Long-context Training Data via Token-level Attentioncited by 9
  4. 2024PECAN: LLM-Guided Dynamic Progress Control with Attention-Guided Hierarchical Weighted Graph for Long-Document QAcited by 0
  5. 2025LLM Reinforcement in Contextcited by 0
  6. 2025Invasive Context Engineering to Control Large Language Modelscited by 0
  7. 2026Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptationcited by 0
  8. 2026Protoreasoning in Tiny Transformerscited by 0
  9. 2026Arbitrary Cipher Attacks Against Large Language Models Do Not Require Fine-Tuningcited by 0
  10. 2026Learning When Not to Listen: Selective Anti-Interference Pretraining for Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Long-Short Alignment for Effective Long-Context Modeling in LLMs
  2. 2024Efficient Alignment of Large Language Models via Data Sampling
  3. 2025Generalizing From Short to Long: Effective Data Synthesis for Long-Context Instruction Tuning
  4. 2025A Survey on Training-free Alignment of Large Language Models
  5. 2025A Comprehensive Evaluation framework of Alignment Techniques for LLMs
  6. 2025RIDE: Enhancing Large Language Model Alignment through Restyled In-Context Learning Demonstration Exemplars
  7. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples
  8. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models