# GATEAU: Selecting Influential Samples for Long Context Alignment

arXiv: https://arxiv.org/abs/2410.15633 · HTML: https://mlsanity.com/paper/2410.15633/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 23 of its 40 references that are in the map.
- First posted on arXiv 21 Oct 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [LESS: Selecting Influential Data for Targeted Instruction Tuning](https://mlsanity.com/paper/2402.04333/) (2024)
- [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://mlsanity.com/paper/2306.04751/) (2023)
- [What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning](https://mlsanity.com/paper/2312.15685/) (2023)
- [From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning](https://mlsanity.com/paper/2308.12032/) (2023)
- [Effective Long-Context Scaling of Foundation Models](https://mlsanity.com/paper/2309.16039/) (2023)
- [The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning](https://mlsanity.com/paper/2312.01552/) (2023)
- [Focused Transformer: Contrastive Training for Context Scaling](https://mlsanity.com/paper/2307.03170/) (2023)
- [Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization](https://mlsanity.com/paper/2406.16008/) (2024)
- [SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agents](https://mlsanity.com/paper/2305.13040/) (2023)
- [Clustering and Ranking: Diversity-preserved Instruction Selection through Expert-aligned Quality Estimation](https://mlsanity.com/paper/2402.18191/) (2024)
- [Long Context Alignment with Short Instructions and Synthesized Positions](https://mlsanity.com/paper/2405.03939/) (2024)
- [RECOST: External Knowledge Guided Data-efficient Instruction Tuning](https://mlsanity.com/paper/2402.17355/) (2024)
- [Mining Clues from Incomplete Utterance: A Query-enhanced Network for Incomplete Utterance Rewriting](https://mlsanity.com/paper/2307.00866/) (2023)

## Built on by (most cited first, 20 of 20)

- [Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidance](https://mlsanity.com/paper/2411.14279/) (2024)
- [A Goal Without a Plan Is Just a Wish: Efficient and Effective Global Planner Training for Long-Horizon Agent Tasks](https://mlsanity.com/paper/2510.05608/) (2025)
- [From Context to Skills: Can Language Models Learn from Context Skillfully?](https://mlsanity.com/paper/2604.27660/) (2026)
- [LongAttn: Selecting Long-context Training Data via Token-level Attention](https://mlsanity.com/paper/2502.16860/) (2025)
- [Crafter: A Multi-Agent Harness for Editable Scientific Figure Generation from Diverse Inputs](https://mlsanity.com/paper/2605.30611/) (2026)
- [RhinoInsight: Improving Deep Research through Control Mechanisms for Model Behavior and Context](https://mlsanity.com/paper/2511.18743/) (2025)
- [Long Context Scaling: Divide and Conquer via Multi-Agent Question-driven Collaboration](https://mlsanity.com/paper/2505.20625/) (2025)
- [LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions](https://mlsanity.com/paper/2505.17134/) (2025)
- [From Context to EDUs: Faithful and Structured Context Compression via Elementary Discourse Unit Decomposition](https://mlsanity.com/paper/2512.14244/) (2025)
- [FaithLens: Detecting and Explaining Faithfulness Hallucination](https://mlsanity.com/paper/2512.20182/) (2025)
- [Multi-Granularity Reasoning for Image Quality Assessment via Attribute-Aware Reinforcement Learning to Rank](https://mlsanity.com/paper/2604.09704/) (2026)
- [UniMark: Unified Adaptive Multi-bit Watermarking for Autoregressive Image Generators](https://mlsanity.com/paper/2604.11843/) (2026)
- [Hierarchical Retrieval Augmented Generation for Adversarial Technique Annotation in Cyber Threat Intelligence Text](https://mlsanity.com/paper/2604.14166/) (2026)
- [Avoiding Overthinking and Underthinking: Curriculum-Aware Budget Scheduling for LLMs](https://mlsanity.com/paper/2604.19780/) (2026)
- [MechELK: A Mechanistic Interpretability Framework for Eliciting Latent Knowledge in Large Language Models](https://mlsanity.com/paper/2605.28825/) (2026)
- [Taming Visual Neglect: A Variational Information Bottleneck Framework for Adaptive Attention in Multimodal In-Context Learning](https://mlsanity.com/paper/2608.23570/) (2026)
- [Structure-Token Evidence-Anchored Reasoning for Scientific Chart Understanding](https://mlsanity.com/paper/2609.13267/) (2026)
- [Capability-Routed Visual Retrieval and Evidence Threading for Long-Context Document Question Answering](https://mlsanity.com/paper/2609.13268/) (2026)
- [TraceICL: Thread-Resolved Demonstration Curation with Consistency Calibration for Robust Multimodal In-Context Learning](https://mlsanity.com/paper/2610.12516/) (2026)
- [TerraAgent: Demonstration-Grounded Multi-Agent Reasoning for Long-Context Remote Sensing Visual Question Answering](https://mlsanity.com/paper/2610.12517/) (2026)

## Abstract (arXiv)

Aligning large language models to handle instructions with extremely long contexts has yet to be fully investigated. Previous studies have attempted to scale up the available data volume by synthesizing long instruction-following samples, as constructing such a dataset tends to be challenging for annotators. However, a lack of a well-defined strategy for ensuring data quality may introduce low-quality samples and restrict the model's performance. Thus, we propose GATEAU, a novel framework to address the unique challenge of long context alignment by identifying the influential samples enriched with long-range dependency relations. Specifically, GATEAU measures the long-range dependencies from two essential aspects: the difficulty of generating target responses due to the long-range dependencies, and the difficulty of understanding long inputs due to such dependencies. Comprehensive experiments indicate that GATEAU effectively identifies influential samples, and the model trained on these selected samples exhibits better instruction-following and long-context understanding capabilities.