mlsanityData to 14 Oct 2026

ChemOrch: Empowering LLMs with Chemical Intelligence via Synthetic Instructions

arXiv 2509.16543 ↗First posted 20 Sep 2025cs.CLcited by 3

On the map: Synthetic data and self-improvement for LLMs → Code and instruction data synthesis

Abstract (arXiv)

Empowering large language models (LLMs) with chemical intelligence remains a challenge due to the scarcity of high-quality, domain-specific instruction-response datasets and the misalignment of existing synthetic data generation pipelines with the inherently hierarchical and rule-governed structure of chemical information. To address this, we propose ChemOrch, a framework that synthesizes chemically grounded instruction-response pairs through a two-stage process: task-controlled instruction generation and tool-aware response construction. ChemOrch enables controllable diversity and levels of difficulty for the generated tasks, and ensures response precision through tool planning and distillation, and tool-based self-repair mechanisms. The effectiveness of ChemOrch is evaluated based on: 1) the high quality of generated instruction data, demonstrating superior diversity and strong alignment with chemical constraints; 2) the reliable generation of evaluation tasks that more effectively reveal LLM weaknesses in chemistry; and 3) the significant improvement of LLM chemistry capabilities when the generated instruction data are used for fine-tuning. Our work thus represents a critical step toward scalable and verifiable chemical intelligence in LLMs.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Code and instruction data synthesis, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 46 of its 68 references that are in the map. First posted on arXiv 20 Sep 2025.

Builds on

46 of 68 references.

References · 46

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  4. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  5. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  6. 2023ChatDev: Communicative Agents for Software Developmentcited by 919
  7. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 769
  8. 2024Phi-4 Technical Reportcited by 711
  9. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 631
  10. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  11. 2024Understanding the planning of LLM agents: A surveycited by 421
  12. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
34 more
  1. 2021Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Developmentcited by 264
  2. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 256
  3. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 232
  4. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
  5. 2021Generating Datasets with Pretrained Language Modelscited by 193
  6. 2023Simple synthetic data reduces sycophancy in large language modelscited by 187
  7. 2023What can Large Language Models do in chemistry? A comprehensive benchmark on eight taskscited by 185
  8. 2023Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Modelscited by 133
  9. 2023AugGPT: Leveraging ChatGPT for Text Data Augmentationcited by 130
  10. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 125
  11. 2024ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidencecited by 113
  12. 2024ChemLLM: A Chemical Large Language Modelcited by 106
  13. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitationscited by 100
  14. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 99
  15. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  16. 202314 Examples of How LLMs Can Transform Materials Science and Chemistry: A Reflection on a Large Language Model Hackathoncited by 86
  17. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  18. 2024HoneyComb: A Flexible LLM-Based Agent System for Materials Sciencecited by 57
  19. 2024Dynamic Evaluation of Large Language Models by Meta Probing Agentscited by 54
  20. 2020Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answeringcited by 41
  21. 2025ChemAgent: Self-updating Library in Large Language Models Improves Chemical Reasoningcited by 41
  22. 2024Multimodal Large Language Models for Inverse Molecular Design with Retrosynthetic Planningcited by 29
  23. 2024MolecularGPT: Open Large Language Model (LLM) for Few-Shot Molecular Property Predictioncited by 24
  24. 2024DataGen: Unified Synthetic Dataset Generation via Large Language Modelscited by 23
  25. 2024AutoBench-V: Can Large Vision-Language Models Benchmark Themselves?cited by 21
  26. 2024SyntheT2C: Generating Synthetic Data for Fine-Tuning Large Language Models on the Text2Cypher Taskcited by 19
  27. 2025Oasis: One Image is All You Need for Multimodal Instruction Data Synthesiscited by 14
  28. 2024GP-MoLFormer: A Foundation Model For Molecular Generationcited by 11
  29. 2025MetaSynth: Meta-Prompting-Driven Agentic Scaffolds for Diverse Synthetic Data Generationcited by 10
  30. 2024Scikit-fingerprints: easy and efficient computation of molecular fingerprints in Pythoncited by 9
  31. 2024Can LLMs Generate Diverse Molecules? Towards Alignment with Structural Diversitycited by 9
  32. 2024MetaScientist: A Human-AI Synergistic Framework for Automated Mechanical Metamaterial Designcited by 8
  33. 2025ASKCOS: an open source software suite for synthesis planningcited by 5
  34. 2025ELTEX: A Framework for Domain-Driven Synthetic Data Generationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026SkillGen: Verified Inference-Time Agent Skill Synthesiscited by 36
  2. 2025Building a Foundational Guardrail for General Agentic Systems via Synthetic Datacited by 29
  3. 2026Synthetic Interaction Data for Scalable Personalization in Large Language Modelscited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2026ChemDIRT: A Diversified Instruction, Representation, and Task Benchmark for Robust Chemistry-LLM Evaluation
  2. 2025CheMatAgent: Enhancing LLMs for Chemistry and Materials Science through Tree-Search Based Tool Learning
  3. 2025Chemical reasoning in LLMs unlocks strategy-aware synthesis planning and reaction mechanism elucidation
  4. 2024Are large language models superhuman chemists?
  5. 2024ChemDFM-X: Towards Large Multimodal Model for Chemistry
  6. 2025Atom-anchored LLMs speak Chemistry: A Retrosynthesis Demonstration
  7. 2024ChemSafetyBench: Benchmarking LLM Safety on Chemistry Domain
  8. 2026Rethinking Molecular Text Representations for LLMs: An Empirical Study