mlsanityData to 7 Oct 2026

GRACE: Gradient-aligned Reasoning Data Curation for Efficient Post-training

arXiv 2605.13130 ↗First posted 13 May 2026cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Existing reasoning data curation pipelines score whole samples, treating every intermediate step as equally valuable. In reality, steps within a trace contribute very unevenly, and selecting reasoning data well requires assessing them individually. We present GRACE, a gradient-aligned curation method that views each reasoning trace as a sequence of optimization events and scores every step by two complementary signals: its alignment with the answer-oriented gradient direction, and its consistency with the preceding reasoning trajectory. Step-level scores are aggregated into a sample-level value for subset selection, using only the model's internal optimization signals and no external reward models or step annotations. To make this scalable, GRACE introduces a representation-level gradient proxy that estimates step-level alignment from token-level upstream signals in a single forward pass. Post-training Qwen3-VL-2B-Instruct on MMathCoT-1M, GRACE reaches 108.8% of the full-data performance with 20% of the data and retains 100.2% with only 5%, with subsets that transfer effectively across model backbones.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 35 of its 37 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

35 of 37 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Unlocking Multimodal Mathematical Reasoning via Process Reward Model8 Jan
  2. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 33

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2025Qwen3-VL Technical Reportcited by 2,534
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  9. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  11. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  12. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
21 more
  1. 2023LIMA: Less Is More for Alignmentcited by 1,080
  2. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 617
  3. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 602
  4. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  5. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 516
  6. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  7. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 446
  8. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  9. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
  10. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 345
  11. 2023TRAK: Attributing Model Behavior at Scalecited by 273
  12. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 192
  13. 2018Representer Point Selection for Explaining Deep Neural Networkscited by 187
  14. 2022Datamodels: Predicting Predictions from Training Datacited by 186
  15. 2023Instruction Mining: Instruction Data Selection for Tuning Large Language Modelscited by 57
  16. 2023Self-Evolved Diverse Data Sampling for Efficient Instruction Tuningcited by 52
  17. 2024ICONS: Influence Consensus for Vision-Language Data Selectioncited by 28
  18. 2024Concept-skill Transferability-based Data Selection for Large Vision-Language Modelscited by 24
  19. 2025Filter Images First, Generate Instructions Later: Pre-Instruction Data Selection for Visual Instruction Tuningcited by 12
  20. 2025Benchmarking Multimodal CoT Reward Model Stepwise by Visual Programcited by 10
  21. 2025Uncovering Intrinsic Capabilities: A Paradigm for Data Curation in Vision-Language Modelscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Reasoning Quality Emerges Early: Data Curation for Reasoning Models
  2. 2026Not All Preferences Deserve Gradients: Understanding Gradient Utility in Offline Reasoning Alignment
  3. 2026Data-Efficient Curation for Multimodal Reasoning under Fixed Training Protocols
  4. 2026Invariant Gradient Alignment for Robust Reasoning Distillation
  5. 2026GRACE: Step-Level Benchmark for Faithful Reasoning over Context
  6. 2026Stabilizing Efficient Reasoning with Step-Level Advantage Selection
  7. 2025InfiAlign: A Scalable and Sample-Efficient Framework for Aligning LLMs to Enhance Reasoning Capabilities
  8. 2026Skill-Aware Data Selection and Fine-Tuning for Data-Efficient Reasoning Distillation