# CT-OPD: Counterfactual Trace On-Policy Distillation for Diffusion Vision-Language Models

arXiv: https://arxiv.org/abs/2609.32781 · HTML: https://mlsanity.com/paper/2609.32781/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 31 of its 32 references that are in the map.
- First posted on arXiv 26 Sep 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 30 of 31)

- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Qwen-Image Technical Report](https://mlsanity.com/paper/2508.02324/) (2025)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Large Language Diffusion Models](https://mlsanity.com/paper/2502.09992/) (2025)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation](https://mlsanity.com/paper/2310.05737/) (2023)
- [MoE-LLaVA: Mixture of Experts for Large Vision-Language Models](https://mlsanity.com/paper/2401.15947/) (2024)
- [MMaDA: Multimodal Large Diffusion Language Models](https://mlsanity.com/paper/2505.15809/) (2025)
- [LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning](https://mlsanity.com/paper/2505.16933/) (2025)
- [Self-Rewarding Vision-Language Model via Reasoning Decomposition](https://mlsanity.com/paper/2508.19652/) (2025)
- [Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models](https://mlsanity.com/paper/2509.06949/) (2025)
- [LaViDa: A Large Diffusion Language Model for Multimodal Understanding](https://mlsanity.com/paper/2505.16839/) (2025)
- [Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs](https://mlsanity.com/paper/2402.12030/) (2024)
- [LLaDA-MoE: A Sparse MoE Diffusion Language Model](https://mlsanity.com/paper/2509.24389/) (2025)
- [Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching](https://mlsanity.com/paper/2503.20083/) (2025)
- [d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillation](https://mlsanity.com/paper/2601.07568/) (2026)
- [MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Models](https://mlsanity.com/paper/2508.13148/) (2025)
- [Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimization](https://mlsanity.com/paper/2510.08233/) (2025)
- [Consolidating Reinforcement Learning for Multimodal Discrete Diffusion Models](https://mlsanity.com/paper/2510.02880/) (2025)
- [Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation](https://mlsanity.com/paper/2606.06712/) (2026)
- [DiffusionVL: Translating Any Autoregressive Models into Diffusion Vision Language Models](https://mlsanity.com/paper/2512.15713/) (2025)
- [Fast-dVLM: Efficient Block-Diffusion VLM via Direct Conversion from Autoregressive VLM](https://mlsanity.com/paper/2604.06832/) (2026)
- [A Dual-Space Framework for General Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2504.11426/) (2025)
- [Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization](https://mlsanity.com/paper/2607.22334/) (2026)
- [LaViDa-R1: Advancing Reasoning for Unified Multimodal Diffusion Language Models](https://mlsanity.com/paper/2602.14147/) (2026)
- [Trace-Based On-Policy Distillation for Masked Diffusion Language Models](https://mlsanity.com/paper/2607.16872/) (2026)
- [dOPSD: On-Policy Self-Distillation for Diffusion Language Models](https://mlsanity.com/paper/2607.04428/) (2026)
- [Diffusion-State Policy Optimization for Masked Diffusion Language Models](https://mlsanity.com/paper/2602.06462/) (2026)
- [Beyond Mode-Seeking RL: Trajectory-Balance Post-Training for Diffusion Language Models](https://mlsanity.com/paper/2605.13935/) (2026)
- [OPTD: On-Policy Transition Distillation with Consistency-Guided Adaptive Compression for Few-Step Diffusion Language Models](https://mlsanity.com/paper/2608.02942/) (2026)

## Abstract (arXiv)

Diffusion vision-language models generate answers by gradually resolving masked tokens, making accurate conditional prediction in partially resolved states central to post-training. Masking completed answers yields coherent contexts and targets, but prescribed masks do not reflect the model's reveal decisions. Its trajectories capture these decisions, yet their provisional visible tokens can conflict with the target response. Outcome-based reinforcement learning follows these trajectories but provides only response-level feedback, which loses contrast when sampled rewards tie. To align coherent token-level supervision with the model's reveal decisions, we introduce Counterfactual Trace On-Policy Distillation (CT-OPD), which combines completed teacher responses with trajectory masks from the current student. CT-OPD retokenizes each teacher response in the student's vocabulary and extracts unresolved-position masks at successive stages of the student's reverse process. For each mask, it discards provisional rollout values and reconstructs the partial state from the teacher endpoint, so the supervised positions follow the current trajectory while the visible context and targets remain consistent with the same response. The student is trained on these reconstructed states with its native categorical loss, and trajectories are refreshed as the model evolves. Across dense and sparse diffusion architectures, CT-OPD consistently enhances multimodal understanding and reasoning capabilities, with gains of up to 9.80 points on the nine-benchmark average. On the unified understanding-and-generation architecture, it also improves both visual understanding and image generation, showing that the same principle transfers across architectures and modalities. Ablations further attribute these gains to coherent reconstruction and current-model trajectory masks.