mlsanityData to 14 Oct 2026

RoCo-ACE: Rollout-Conditioned Online Distillation for Retention-Aware Knowledge Injection

arXiv 2607.24771 ↗First posted 10 Jun 2026cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Knowledge injection updates pretrained MLLMs with new factual or domain-specific knowledge, but fitting full authoritative answers can cause drift in non-updated behavior. Online distillation mitigates this drift by training on model-generated rollouts, yet uniform reference-conditioned distillation provides coarse supervision: it can under-emphasize reference-supported rollout tokens and supervise omitted facts only indirectly. We introduce RoCo-ACE, a rollout-conditioned online distillation objective for knowledge injection. RoCo uses same-rollout reference-free/reference-conditioned likelihood contrast to reallocate additional distillation weight to reference-supported rollout tokens, while ACE adds sparse reference-side anchored correction for authoritative anchors omitted from the rollout without full-answer imitation. Across three knowledge-injection settings, six retention benchmarks, multiple baselines, and multiple base models, RoCo-ACE achieves the best injected-knowledge accuracy among compared methods while keeping evaluated retention close to the base model.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 30 references that are in the map. First posted on arXiv 10 Jun 2026.

Builds on

20 of 30 references.

References · 20

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  6. 2016Learning without Forgettingcited by 2,496
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  8. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  9. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  10. 2026On-Policy Context Distillation for Language Modelscited by 186
  11. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 163
  12. 2021Training Networks in Null Space of Feature Covariance for Continual Learningcited by 143
8 more
  1. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performancecited by 117
  2. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routingcited by 76
  3. 2026CRISP: Compressed Reasoning via Iterative Self-Policy Distillationcited by 58
  4. 2024MIBench: Evaluating Multimodal Large Language Models over Multiple Imagescited by 39
  5. 2025SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuningcited by 33
  6. 2025Automating Steering for Safe Multimodal Large Language Modelscited by 17
  7. 2025Put the Space of LoRA Initialization to the Extreme to Preserve Pre-trained Knowledgecited by 16
  8. 2025LoRA in LoRA: Towards Parameter-Efficient Architecture Expansion for Continual Visual Instruction Tuningcited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026MixSD: Mixed Contextual Self-Distillation for Knowledge Injection
  2. 2025Comparing Knowledge Injection Methods for LLMs in a Low-Resource Regime
  3. 2024Efficient Knowledge Injection in LLMs via Self-Distillation
  4. 2025Memorization vs. Reasoning: Updating LLMs with New Knowledge
  5. 2026Reinforcement-aware Knowledge Distillation for LLM Reasoning
  6. 2026GUARD: Natural Forgetting in Large Reasoning Models via Guided Answer-Reasoning Distillation
  7. 2025Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncation
  8. 2024DDK: Distilling Domain Knowledge for Efficient Large Language Models