mlsanityData to 14 Oct 2026

GATEAU: Selecting Influential Samples for Long Context Alignment

arXiv 2410.15633 ↗First posted 21 Oct 2024cs.CL cs.AIcited by 20

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning large language models to handle instructions with extremely long contexts has yet to be fully investigated. Previous studies have attempted to scale up the available data volume by synthesizing long instruction-following samples, as constructing such a dataset tends to be challenging for annotators. However, a lack of a well-defined strategy for ensuring data quality may introduce low-quality samples and restrict the model's performance. Thus, we propose GATEAU, a novel framework to address the unique challenge of long context alignment by identifying the influential samples enriched with long-range dependency relations. Specifically, GATEAU measures the long-range dependencies from two essential aspects: the difficulty of generating target responses due to the long-range dependencies, and the difficulty of understanding long inputs due to such dependencies. Comprehensive experiments indicate that GATEAU effectively identifies influential samples, and the model trained on these selected samples exhibits better instruction-following and long-context understanding capabilities.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 23 of its 40 references that are in the map. First posted on arXiv 21 Oct 2024.

Builds on

23 of 40 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning25 Dec
  2. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning4 Dec
  3. 2023LIMA: Less Is More for Alignment18 May
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec

Other references · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  6. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  7. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  8. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  9. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  10. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  11. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 307
  12. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
7 more
  1. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  2. 2024Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilizationcited by 101
  3. 2023SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agentscited by 71
  4. 2024Clustering and Ranking: Diversity-preserved Instruction Selection through Expert-aligned Quality Estimationcited by 53
  5. 2024Long Context Alignment with Short Instructions and Synthesized Positionscited by 22
  6. 2024RECOST: External Knowledge Guided Data-efficient Instruction Tuningcited by 18
  7. 2023Mining Clues from Incomplete Utterance: A Query-enhanced Network for Incomplete Utterance Rewritingcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 12.

20 papers.

Sort:

By area:

  1. 2024Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidancecited by 32
  2. 2025A Goal Without a Plan Is Just a Wish: Efficient and Effective Global Planner Training for Long-Horizon Agent Taskscited by 19
  3. 2026From Context to Skills: Can Language Models Learn from Context Skillfully?cited by 19
  4. 2025LongAttn: Selecting Long-context Training Data via Token-level Attentioncited by 9
  5. 2026Crafter: A Multi-Agent Harness for Editable Scientific Figure Generation from Diverse Inputscited by 9
  6. 2025RhinoInsight: Improving Deep Research through Control Mechanisms for Model Behavior and Contextcited by 8
  7. 2025Long Context Scaling: Divide and Conquer via Multi-Agent Question-driven Collaborationcited by 7
  8. 2025LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructionscited by 6
  9. 2025From Context to EDUs: Faithful and Structured Context Compression via Elementary Discourse Unit Decompositioncited by 3
  10. 2025FaithLens: Detecting and Explaining Faithfulness Hallucinationcited by 3
  11. 2026Multi-Granularity Reasoning for Image Quality Assessment via Attribute-Aware Reinforcement Learning to Rankcited by 1
  12. 2026UniMark: Unified Adaptive Multi-bit Watermarking for Autoregressive Image Generatorscited by 0
8 more
  1. 2026Hierarchical Retrieval Augmented Generation for Adversarial Technique Annotation in Cyber Threat Intelligence Textcited by 0
  2. 2026Avoiding Overthinking and Underthinking: Curriculum-Aware Budget Scheduling for LLMscited by 0
  3. 2026MechELK: A Mechanistic Interpretability Framework for Eliciting Latent Knowledge in Large Language Modelscited by 0
  4. 2026Taming Visual Neglect: A Variational Information Bottleneck Framework for Adaptive Attention in Multimodal In-Context Learningcited by 0
  5. 2026Structure-Token Evidence-Anchored Reasoning for Scientific Chart Understandingcited by 0
  6. 2026Capability-Routed Visual Retrieval and Evidence Threading for Long-Context Document Question Answeringcited by 0
  7. 2026TraceICL: Thread-Resolved Demonstration Curation with Consistency Calibration for Robust Multimodal In-Context Learningcited by 0
  8. 2026TerraAgent: Demonstration-Grounded Multi-Agent Reasoning for Long-Context Remote Sensing Visual Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Models
  2. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models
  3. 2026LongCrafter: Towards Diverse Long-Context Understanding via Evidence-Graph-Guided Instruction Synthesis
  4. 2025Generalizing From Short to Long: Effective Data Synthesis for Long-Context Instruction Tuning
  5. 2024Enhancing and Assessing Instruction-Following with Fine-Grained Instruction Variants
  6. 2024Align2LLaVA: Cascaded Human and Large Language Model Preference Alignment for Multi-modal Instruction Curation
  7. 2024CodecLM: Aligning Language Models with Tailored Synthetic Data
  8. 2025MAIN: Mutual Alignment Is Necessary for instruction tuning