# Unraveling Molecular Structure: A Multimodal Spectroscopic Dataset for Chemistry

arXiv: https://arxiv.org/abs/2407.17492 · HTML: https://mlsanity.com/paper/2407.17492/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 75 references that are in the map.
- First posted on arXiv 4 Jul 2024.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [Junction Tree Variational Autoencoder for Molecular Graph Generation](https://mlsanity.com/paper/1802.04364/) (2018)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans](https://mlsanity.com/paper/2110.04994/) (2021)
- [OpenNMT: Open-source Toolkit for Neural Machine Translation](https://mlsanity.com/paper/1709.03815/) (2017)
- [Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Prediction](https://mlsanity.com/paper/1811.02633/) (2018)
- [Learning to Plan Chemical Syntheses](https://mlsanity.com/paper/1708.04202/) (2017)
- [4M: Massively Multimodal Masked Modeling](https://mlsanity.com/paper/2312.06647/) (2023)
- [Accelerating Material Design with the Generative Toolkit for Scientific Discovery](https://mlsanity.com/paper/2207.03928/) (2022)
- [Prefix-Tree Decoding for Predicting Mass Spectra from Molecules](https://mlsanity.com/paper/2303.06470/) (2023)
- [Fragmentation trees reloaded](https://mlsanity.com/paper/1412.1929/) (2014)
- [Unassisted Noise Reduction of Chemical Reaction Data Sets](https://mlsanity.com/paper/2102.01399/) (2021)

## Built on by (most cited first, 30 of 31)

- [From Generalist to Specialist: A Survey of Large Language Models for Chemistry](https://mlsanity.com/paper/2412.19994/) (2024)
- [NMR-Solver: Automated Structure Elucidation via Large-Scale Spectral Matching and Physics-Guided Fragment Optimization](https://mlsanity.com/paper/2509.00640/) (2025)
- [MolSpectra: Pre-training 3D Molecular Representation with Multi-modal Energy Spectra](https://mlsanity.com/paper/2502.16284/) (2025)
- [Artificial Intelligence in Spectroscopy: Advancing Chemistry from Prediction to Generation and Beyond](https://mlsanity.com/paper/2502.09897/) (2025)
- [Quantum Doubly Stochastic Transformers](https://mlsanity.com/paper/2504.16275/) (2025)
- [DiffSpectra: Molecular Structure Elucidation from Spectra using Diffusion Models](https://mlsanity.com/paper/2507.06853/) (2025)
- [DiffNMR: Diffusion Models for Nuclear Magnetic Resonance Spectra Elucidation](https://mlsanity.com/paper/2507.08854/) (2025)
- [IR-Agent: Expert-Inspired LLM Agents for Structure Elucidation from Infrared Spectra](https://mlsanity.com/paper/2508.16112/) (2025)
- [Test-Time Tuned Language Models Enable End-to-end De Novo Molecular Structure Generation from MS/MS Spectra](https://mlsanity.com/paper/2510.23746/) (2025)
- [Atomic Diffusion Models for Small Molecule Structure Elucidation from NMR Spectra](https://mlsanity.com/paper/2512.03127/) (2025)
- [FRIGID: Scaling Diffusion-Based Molecular Generation from Mass Spectra at Training and Inference Time](https://mlsanity.com/paper/2604.16648/) (2026)
- [SpectraLLM: Uncovering the Ability of LLMs for Molecular Structure Elucidation from Multi-Spectral Data](https://mlsanity.com/paper/2508.08441/) (2025)
- [MassSpecGym in the Wild: Uncovering and Correcting Evaluation Pitfalls in AI-Driven Molecule Discovery](https://mlsanity.com/paper/2606.19624/) (2026)
- [SpectrumWorld: Artificial Intelligence Foundation for Spectroscopy](https://mlsanity.com/paper/2508.01188/) (2025)
- [We Need Improved Data Curation and Attribution in AI for Scientific Discovery](https://mlsanity.com/paper/2504.02486/) (2025)
- [Large Language Models Transform Organic Synthesis From Reaction Prediction to Automation](https://mlsanity.com/paper/2508.05427/) (2025)
- [CITE: A Comprehensive Benchmark for Heterogeneous Text-Attributed Graphs on Catalytic Materials](https://mlsanity.com/paper/2508.15392/) (2025)
- [SpecMol: A Spectroscopy-Grounded Foundation Model for Multi-Task Molecular Learning](https://mlsanity.com/paper/2509.21861/) (2025)
- [NMIRacle: Multi-modal Generative Molecular Elucidation from IR and NMR Spectra](https://mlsanity.com/paper/2512.19733/) (2025)
- [NMRTrans: Structure Elucidation from Experimental NMR Spectra via Set Transformers](https://mlsanity.com/paper/2602.10158/) (2026)
- [SpecX: A Large-Scale Benchmark for Multi-Modal Spectroscopy and Cross-Paradigm Evaluation](https://mlsanity.com/paper/2605.18791/) (2026)
- [Latent Diffusion-Based 3D Molecular Recovery from Vibrational Spectra](https://mlsanity.com/paper/2603.06113/) (2026)
- [Molecules Meet Language: Confound-Aware Representation Learning and Chemical Property Steering in Transformer-VAE Latent Spaces](https://mlsanity.com/paper/2605.06303/) (2026)
- [CoRe-Gen: Robust Spectrum-to-Structure Generation under Imperfect Fingerprint Conditions](https://mlsanity.com/paper/2605.12980/) (2026)
- [Hypothesis-and-Refinement Learning of Organic Structures from Multimodal Spectroscopic Data](https://mlsanity.com/paper/2607.19816/) (2026)
- [Data Fusion and Contrastive Alignment for Unconstrained IR Molecular Structure Elucidation](https://mlsanity.com/paper/2607.26164/) (2026)
- [Simulation-to-real transfer learning for infrared spectroscopic chemical sensing and analysis from molecules to complex samples](https://mlsanity.com/paper/2608.13341/) (2026)
- [Multi-Agent Closed-Loop Reasoning for Organic Structure Elucidation from Multimodal Spectra](https://mlsanity.com/paper/2608.14720/) (2026)
- [Towards Reasonable Molecular Structure Elucidation from Infrared Spectroscopy with Chemical Feedback](https://mlsanity.com/paper/2608.16082/) (2026)
- [S3C-LLM: Skill-Code Guided Agentic Language Models for Spectrum-to-Structure Elucidation](https://mlsanity.com/paper/2608.30910/) (2026)

## Abstract (arXiv)

Spectroscopic techniques are essential tools for determining the structure of molecules. Different spectroscopic techniques, such as Nuclear magnetic resonance (NMR), Infrared spectroscopy, and Mass Spectrometry, provide insight into the molecular structure, including the presence or absence of functional groups. Chemists leverage the complementary nature of the different methods to their advantage. However, the lack of a comprehensive multimodal dataset, containing spectra from a variety of spectroscopic techniques, has limited machine-learning approaches mostly to single-modality tasks for predicting molecular structures from spectra. Here we introduce a dataset comprising simulated 1H-NMR, 13C-NMR, HSQC-NMR, Infrared, and Mass spectra (positive and negative ion modes) for 790k molecules extracted from chemical reactions in patent data. This dataset enables the development of foundation models for integrating information from multiple spectroscopic modalities, emulating the approach employed by human experts. Additionally, we provide benchmarks for evaluating single-modality tasks such as structure elucidation, predicting the spectra for a target molecule, and functional group predictions. This dataset has the potential automate structure elucidation, streamlining the molecular discovery pipeline from synthesis to structure determination. The dataset and code for the benchmarks can be found at https://rxn4chemistry.github.io/multimodal-spectroscopic-dataset.