# oMeBench: Towards Robust Benchmarking of LLMs in Organic Mechanism Elucidation and Reasoning

arXiv: https://arxiv.org/abs/2510.07731 · HTML: https://mlsanity.com/paper/2510.07731/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 19 of its 59 references that are in the map.
- First posted on arXiv 9 Oct 2025.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)

## Builds on (most cited first, 19 of 19)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [2 OLMo 2 Furious](https://mlsanity.com/paper/2501.00656/) (2024)
- [ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction](https://mlsanity.com/paper/2010.09885/) (2020)
- [SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models](https://mlsanity.com/paper/2307.10635/) (2023)
- [EquiBind: Geometric Deep Learning for Drug Binding Structure Prediction](https://mlsanity.com/paper/2202.05146/) (2022)
- [ChemBERTa-2: Towards Chemical Foundation Models](https://mlsanity.com/paper/2209.01712/) (2022)
- [Training a Scientific Reasoning Model for Chemistry](https://mlsanity.com/paper/2506.17238/) (2025)
- [Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operations](https://mlsanity.com/paper/2505.21318/) (2025)
- [Large Language Models for Scientific Synthesis, Inference and Explanation](https://mlsanity.com/paper/2310.07984/) (2023)
- [Chemical reasoning in LLMs unlocks strategy-aware synthesis planning and reaction mechanism elucidation](https://mlsanity.com/paper/2503.08537/) (2025)
- [LLM-Augmented Chemical Synthesis and Design Decision Programs](https://mlsanity.com/paper/2505.07027/) (2025)
- [mCLM: A Modular Chemical Language Model that Generates Functional and Makeable Molecules](https://mlsanity.com/paper/2505.12565/) (2025)

## Built on by (most cited first, 4 of 4)

- [TREX: Automating LLM Fine-tuning via Agent-Driven Tree-based Exploration](https://mlsanity.com/paper/2604.14116/) (2026)
- [Learning Mechanistic Reasoning for Chemical Reactions with Large Language Models](https://mlsanity.com/paper/2607.12771/) (2026)
- [ChemVA: Advancing Large Language Models on Chemical Reaction Diagrams Understanding](https://mlsanity.com/paper/2605.17214/) (2026)
- [ChemExplain: A Multimodal Benchmark for Explanatory Chemical Reasoning](https://mlsanity.com/paper/2610.17183/) (2026)

## Abstract (arXiv)

Organic reaction mechanisms describe the step-wise elementary processes by which reactants transform into intermediates and products, and are fundamental to understanding chemical reactivity and guiding molecular and reaction de-sign. While large language models (LLMs) have shown promise on chemical tasks such as synthesis design, it remains unclear to what extent this reflects genuine chemical reasoning capabilities: the ability to generate chemically valid intermediates, maintain consistency across reaction steps, and follow logically coherent multi-step pathways. To investigate this, we introduce oMeBench, the first large-scale, expert-curated benchmark for organic mechanism reasoning, comprising over 10,000 annotated mechanistic steps with reaction type labels, intermediate structures, and difficulty ratings. To enable fine-grained evaluation, we further propose oMeS, a dynamic scoring framework that jointly assesses step-level logical consistency and chemical structural similarity. Systematic evaluation of state-of-the-art LLMs reveals that while current models exhibit promising chemical intuition, they struggle to produce correct and consistent reasoning across multi-step mechanisms. Notably, combining prompting strategies with fine-tuning enables smaller-scale models to achieve performance comparable to closed-source frontier models. We hope oMeBench will serve as a rigorous foundation for advancing AI systems toward genuine chemical reasoning.