# G-DIG: Towards Gradient-based Diverse and High-quality Instruction Data Selection for Machine Translation

arXiv: https://arxiv.org/abs/2405.12915 · HTML: https://mlsanity.com/paper/2405.12915/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 43 references that are in the map.
- First posted on arXiv 21 May 2024.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [LESS: Selecting Influential Data for Targeted Instruction Tuning](https://mlsanity.com/paper/2402.04333/) (2024)
- [From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning](https://mlsanity.com/paper/2308.12032/) (2023)
- [Few-shot Learning with Multilingual Language Models](https://mlsanity.com/paper/2112.10668/) (2021)
- [Prompting Large Language Model for Machine Translation: A Case Study](https://mlsanity.com/paper/2301.07069/) (2023)
- [In-context Examples Selection for Machine Translation](https://mlsanity.com/paper/2212.02437/) (2022)
- [Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions](https://mlsanity.com/paper/2005.06676/) (2020)
- [MoDS: Model-oriented Data Selection for Instruction Tuning](https://mlsanity.com/paper/2311.15653/) (2023)
- [Learning Compact Metrics for MT](https://mlsanity.com/paper/2110.06341/) (2021)
- [The unreasonable effectiveness of few-shot learning for machine translation](https://mlsanity.com/paper/2302.01398/) (2023)
- [Leveraging GPT-4 for Automatic Translation Post-Editing](https://mlsanity.com/paper/2305.14878/) (2023)
- [Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructions](https://mlsanity.com/paper/2305.15083/) (2023)
- [TIM: Teaching Large Language Models to Translate with Comparison](https://mlsanity.com/paper/2307.04408/) (2023)
- [Interactive Label Cleaning with Example-based Explanations](https://mlsanity.com/paper/2106.03922/) (2021)
- [Improving Translation Faithfulness of Large Language Models via Augmenting Instructions](https://mlsanity.com/paper/2308.12674/) (2023)
- [Improving Machine Translation with Large Language Models: A Preliminary Study with Cooperative Decoding](https://mlsanity.com/paper/2311.02851/) (2023)
- [MT-PATCHER: Selective and Extendable Knowledge Distillation from Large Language Models for Machine Translation](https://mlsanity.com/paper/2403.09522/) (2024)

## Built on by (most cited first, 30 of 30)

- [TAGCOS: Task-agnostic Gradient Clustered Coreset Selection for Instruction Tuning Data](https://mlsanity.com/paper/2407.15235/) (2024)
- [Improving Influence-based Instruction Tuning Data Selection for Balanced Learning of Diverse Capabilities](https://mlsanity.com/paper/2501.12147/) (2025)
- [Seed-X: Building Strong Multilingual Translation LLM with 7B Parameters](https://mlsanity.com/paper/2507.13618/) (2025)
- [Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generation](https://mlsanity.com/paper/2508.10858/) (2025)
- [Chasing Random: Instruction Selection Strategies Fail to Generalize](https://mlsanity.com/paper/2410.15225/) (2024)
- [Diversity-driven Data Selection for Language Model Tuning through Sparse Autoencoder](https://mlsanity.com/paper/2502.14050/) (2025)
- [Ensembles of Low-Rank Expert Adapters](https://mlsanity.com/paper/2502.00089/) (2025)
- [Seed LiveInterpret 2.0: End-to-end Simultaneous Speech-to-speech Translation with Your Voice](https://mlsanity.com/paper/2507.17527/) (2025)
- [Influential Language Data Selection via Gradient Trajectory Pursuit](https://mlsanity.com/paper/2410.16710/) (2024)
- [Select2Reason: Efficient Instruction-Tuning Data Selection for Long-CoT Reasoning](https://mlsanity.com/paper/2505.17266/) (2025)
- [DATE-LM: Benchmarking Data Attribution Evaluation for Large Language Models](https://mlsanity.com/paper/2507.09424/) (2025)
- [Teaching LLMs According to Their Aptitude: Adaptive Reasoning for Mathematical Problem Solving](https://mlsanity.com/paper/2502.12022/) (2025)
- [Mining Useful General Data for Low-Resource Domain Adaptation](https://mlsanity.com/paper/2511.07380/) (2025)
- [MDIT: A Model-free Data Interpolation Method for Diverse Instruction Tuning](https://mlsanity.com/paper/2504.07288/) (2025)
- [RICo: Refined In-Context Contribution for Automatic Instruction-Tuning Data Selection](https://mlsanity.com/paper/2505.05327/) (2025)
- [MIRA: Mid-training Rubric Anchoring for Source-Aware Data Selection](https://mlsanity.com/paper/2605.30288/) (2026)
- [RAISE: Reinforced Adaptive Instruction Selection For Large Language Models](https://mlsanity.com/paper/2504.07282/) (2025)
- [Rethinking Representativeness and Diversity in Dynamic Data Selection](https://mlsanity.com/paper/2603.04981/) (2026)
- [Interaction-Aware Influence Functions for Group Attribution](https://mlsanity.com/paper/2605.15675/) (2026)
- [Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories](https://mlsanity.com/paper/2606.03979/) (2026)
- [BLADE: Scalable Bi-level Adaptive Data Selection for LLM Training](https://mlsanity.com/paper/2606.18650/) (2026)
- [ContrastScore: Towards Higher Quality, Less Biased, More Efficient Evaluation Metrics with Contrastive Evaluation](https://mlsanity.com/paper/2504.02106/) (2025)
- [From Tens of Hours to Tens of Thousands: Scaling Back-Translation for Speech Recognition](https://mlsanity.com/paper/2505.16972/) (2025)
- [Daunce: Data Attribution through Uncertainty Estimation](https://mlsanity.com/paper/2505.23223/) (2025)
- [CC-Tuning: A Cross-Lingual Connection Mechanism for Improving Joint Multilingual Supervised Fine-Tuning](https://mlsanity.com/paper/2506.00875/) (2025)
- [RL from Teacher-Model Refinement: Gradual Imitation Learning for Machine Translation](https://mlsanity.com/paper/2507.22219/) (2025)
- [MADS: Model-Aware Diverse Core Set Selection for Instruction Tuning](https://mlsanity.com/paper/2605.30857/) (2026)
- [Multilingual Fine-Tuning via Localized Gradient Conflict Resolution](https://mlsanity.com/paper/2606.05613/) (2026)
- [Refreshing Less, Selecting Better: Reusing Stale Gradient Features for Efficient Influence-Based Data Selection](https://mlsanity.com/paper/2609.32888/) (2026)
- [Which and When to Admit: Gradient Admission for Data-Centric Small Language Model Finetuning](https://mlsanity.com/paper/2610.07553/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable abilities in general scenarios. Instruction finetuning empowers them to align with humans in various tasks. Nevertheless, the Diversity and Quality of the instruction data remain two main challenges for instruction finetuning. With regard to this, in this paper, we propose a novel gradient-based method to automatically select high-quality and diverse instruction finetuning data for machine translation. Our key innovation centers around analyzing how individual training examples influence the model during training. Specifically, we select training examples that exert beneficial influences on the model as high-quality ones by means of Influence Function plus a small high-quality seed dataset. Moreover, to enhance the diversity of the training data we maximize the variety of influences they have on the model by clustering on their gradients and resampling. Extensive experiments on WMT22 and FLORES translation tasks demonstrate the superiority of our methods, and in-depth analysis further validates their effectiveness and generalization.