Robust Trajectory Distillation: Hybrid Reweighting Meets Teacher-Inspired Targets
On the map: Knowledge distillation → On-policy and self-distillation for LLMs
Abstract (arXiv)
Dataset distillation (DD) condenses large corpora into compact, information-rich subsets for efficient training and reuse. However, under noisy supervision, DD risks condensing corrupted associations together with useful signals, degrading robustness. Conventional noisy-label remedies (sample selection, loss weighting, label correction) tightly couple noise estimation with model optimization, often require clean anchors, and can amplify confirmation bias-assumptions that are misaligned with DD's goal of compact, plug-and-play supervision. We therefore propose a trajectory-based DD framework that jointly suppresses noise and preserves transferable knowledge without relabeling or clean subsets. It comprises two complementary components: Selective Guidance Reweighting (SGR), which fuses global forgetting patterns (second-split forgetting) with local neighborhood consistency into a progressive reweighting scheme that prioritizes clean supervision along the teacher trajectory; and Teacher-Inspired Auxiliary Targets (TIAT), which inject auxiliary residual guidance distilled from intermediate teacher dynamics to reinforce informative signals while remaining internally consistent. Together, SGR and TIAT produce distilled datasets with cleaner and richer representations under noisy supervision. The framework is robust, label-preserving, computationally lightweight, and broadly applicable, yielding consistent gains over state-of-the-art DD baselines across symmetric, asymmetric, and real-world noise.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 37 of its 50 references that are in the map. First posted on arXiv 29 Jun 2026.
Builds on
37 of 50 references.
References · 37
- 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,059
- 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 997
- 2020DivideMix: Learning with Noisy Labels as Semi-supervised Learningcited by 692
- 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 653
- 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 582
- 2020Dataset Condensation with Gradient Matchingcited by 516
- 2022Dataset Distillation by Matching Training Trajectoriescited by 442
- 2020Early-Learning Regularization Prevents Memorization of Noisy Labelscited by 416
- 2014Training Deep Neural Networks on Noisy Labels with Bootstrappingcited by 322
- 2017WebVision Database: Visual Learning and Understanding from Web Datacited by 262
- 2021Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotationscited by 256
- 2017Decoupling "when to update" from "how to update"cited by 230
25 more
- 2018Dataset Distillationcited by 230
- 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 218
- 2022Dataset Distillation using Neural Feature Regressioncited by 177
- 2019Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Ratescited by 142
- 2022Minimizing the Accumulated Trajectory Error to Improve Dataset Distillationcited by 142
- 2023Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matchingcited by 118
- 2023On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigmcited by 109
- 2019Curriculum Loss: Robust Learning and Generalization against Label Corruptioncited by 88
- 2021Detecting Corrupted Labels Without Training a Model to Predictcited by 58
- 2020Deep k-NN for Noisy Labelscited by 55
- 2022Learning with Neighbor Consistency for Noisy Labelscited by 55
- 2023M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancycited by 49
- 2025Dataset Distillation with Neural Characteristic Function: A Minmax Perspectivecited by 45
- 2022Characterizing Datapoints via Second-Split Forgettingcited by 37
- 2024Exploiting Inter-sample and Inter-feature Relations in Dataset Distillationcited by 31
- 2018Data Dropout: Optimizing Training Data for Convolutional Neural Networkscited by 23
- 2022Neighborhood Collective Estimation for Noisy Label Identification and Correctioncited by 22
- 2024SelMatch: Effectively Scaling Up Dataset Distillation via Selection-Based Initialization and Partial Updates by Trajectory Matchingcited by 22
- 2023Learning from Noisy Labels with Decoupled Meta Label Purifiercited by 21
- 2023Data Distillation Can Be Like Vodka: Distilling More Times For Better Qualitycited by 17
- 2024DANCE: Dual-View Distribution Alignment for Dataset Condensationcited by 15
- 2022L2B: Learning to Bootstrap Robust Models for Combating Label Noisecited by 10
- 2023Generalization Matters: Loss Minima Flattening via Parameter Hybridization for Efficient Online Knowledge Distillationcited by 8
- 2024An Embedding is Worth a Thousand Noisy Labelscited by 5
- 2024Dataset Distillers Are Good Label Denoisers In the Wildcited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Trajectory Learnability for Offline On-Policy Distillation with Imperfect Teachers
- 2024Dynamic Guidance Adversarial Distillation with Enhanced Teacher Knowledge
- 2026Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillation
- 2026Multimodal Dataset Distillation via Phased Teacher Models
- 2023SLaM: Student-Label Mixing for Distillation with Unlabeled Examples
- 2026LARK: Learnability-Grounded Trajectory Selection for Efficient Reasoning Distillation
- 2026Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Models
- 2023Distilling Knowledge for Short-to-Long Term Trajectory Prediction