mlsanityData to 14 Oct 2026

ResoFilter: Fine-grained Synthetic Data Filtering for Large Language Models through Data-Parameter Resonance Analysis

arXiv 2412.14809 ↗First posted 19 Dec 2024cs.CLcited by 4

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

Large language models (LLMs) have shown remarkable effectiveness across various domains, with data augmentation methods utilizing GPT for synthetic data generation becoming prevalent. However, the quality and utility of augmented data remain questionable, and current methods lack clear metrics for evaluating data characteristics. To address these challenges, we propose ResoFilter, a novel method that integrates models, data, and tasks to refine datasets. ResoFilter leverages the fine-tuning process to obtain Data-Parameter features for data selection, offering improved interpretability by representing data characteristics through model weights. Our experiments demonstrate that ResoFilter achieves comparable results to full-scale fine-tuning using only half the data in mathematical tasks and exhibits strong generalization across different models and domains. This method provides valuable insights for constructing synthetic datasets and evaluating high-quality data, offering a promising solution for enhancing data augmentation techniques and improving training dataset quality for LLMs. For reproducibility, we will release our code and data upon acceptance.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 39 of its 59 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

39 of 59 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  11. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
27 more
  1. 2018Deep contextualized word representationscited by 2,290
  2. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  4. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  5. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  6. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  7. 2023LIMA: Less Is More for Alignmentcited by 1,100
  8. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  9. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  10. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  11. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  12. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  13. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  14. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  15. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  16. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  17. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  18. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  19. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 364
  20. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 307
  21. 2024Generative Representational Instruction Tuningcited by 233
  22. 2024Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningcited by 138
  23. 2023AugGPT: Leveraging ChatGPT for Text Data Augmentationcited by 130
  24. 2023MM-LIMA: Less Is More for Alignment in Multi-Modal Datasetscited by 61
  25. 2023Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Taskscited by 55
  26. 2024Synthetic Dialogue Dataset Generation using LLM Agentscited by 39
  27. 2024Scalable Model Editing via Customized Expert Networkscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025SAE-V: Interpreting Multimodal Models for Enhanced Alignmentcited by 23
  2. 2025Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergencecited by 6
  3. 2026Task--Specificity Score: Measuring How Much Instructions Really Matter for Supervisioncited by 0
  4. 2026Know What You Don't Know: Agentic Data Synthesis by Evolving Generation Space for Closed-Loop Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Data Synthesis and Augmentation for Large Language Models
  2. 2025Scaling Laws of Synthetic Data for Language Models
  3. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Models
  4. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
  5. 2024Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Models
  6. 2023Synthetic Data Generation in Low-Resource Settings via Fine-Tuning of Large Language Models
  7. 2025Aleph-Alpha-GermanWeb: Improving German-language LLM pre-training with model-based data curation and synthetic data generation
  8. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey