mlsanityData to 14 Oct 2026

ChemOPD: Multi-Teacher On-Policy Distillation for Multi-Task Chemical Reasoning

arXiv 2609.33838 ↗First posted 27 Sep 2026cs.LG cs.CLcited by 0

On the map: LLM reasoning → Prompting and chain-of-thought reasoning also fits: Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Large language models are increasingly expected to support diverse chemical reasoning capabilities within a unified model. One approach is to develop specialized capabilities separately and consolidate them through multi-teacher on-policy distillation, but this raises two questions: how should specialization be organized, and how should specialist guidance be integrated? We introduce ChemOPD, which addresses both. We estimate task affinities from supervised fine-tuning gradients and solve a constrained mixed-integer program(MIP) to construct partially overlapping specialist groups. During distillation, we retain a generalist teacher trained on all tasks so that specialist guidance supplements rather than replaces its supervision. Our anchor-residual objective gradually increases the routed specialist's contribution on student-generated responses. On ChemCoTBench, affinity-guided specialization produces task-dependent gains over the generalist teacher and improves several capabilities beyond semantic task grouping. Yet stronger teacher-side performance does not automatically yield stronger students: with the same specialists and routes, anchor-residual OPD improves most reported metrics over specialist-only distillation and realizes a larger share of the available teacher gains. These results highlight specialization and capability integration as connected but distinct design problems in chemical reasoning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 17 of its 39 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

17 of 39 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe14 Apr

Other references · 16

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  4. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 163
  5. 2024LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Datasetcited by 128
  6. 2024ChemLLM: A Chemical Large Language Modelcited by 106
  7. 2026TIP: Token Importance in On-Policy Distillationcited by 104
  8. 2018Adapting Auxiliary Losses Using Gradient Similaritycited by 100
  9. 2026MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Trainingcited by 72
  10. 2025ChemDFM-R: A Chemical Reasoning LLM Enhanced with Atomized Chemical Knowledgecited by 20
  11. 2026MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debatecited by 17
  12. 2026Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservationcited by 15
4 more
  1. 2026DanceOPD: On-Policy Generative Field Distillationcited by 14
  2. 2025Enhancing Chemical Reaction and Retrosynthesis Prediction with Large Language Model and Dual-task Learningcited by 9
  3. 2024Towards Principled Task Grouping for Multi-Task Learningcited by 3
  4. 2026MolEmb: Multimodal Large Language Models Can Be Strong Molecular Embedding Modelscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025MiST: Understanding the Role of Mid-Stage Scientific Training in Developing Chemical Reasoning Models
  2. 2025Training a Scientific Reasoning Model for Chemistry
  3. 2026ChemDIRT: A Diversified Instruction, Representation, and Task Benchmark for Robust Chemistry-LLM Evaluation
  4. 2025ChemOrch: Empowering LLMs with Chemical Intelligence via Synthetic Instructions
  5. 2026Masked Diffusion Language Agents for Tool-Integrated Chemical Reasoning
  6. 2025ChemATP: A Training-Free Chemical Reasoning Framework for Large Language Models
  7. 2025ChemAgent: Self-updating Library in Large Language Models Improves Chemical Reasoning
  8. 2025SUPERChem: A Multimodal Reasoning Benchmark in Chemistry