# Benchmark for Assessing Olfactory Perception of Large Language Models

arXiv: https://arxiv.org/abs/2604.00002 · HTML: https://mlsanity.com/paper/2604.00002/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 48 references that are in the map.
- First posted on arXiv 8 Mar 2026.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [OpenAI GPT-5 System Card](https://mlsanity.com/paper/2601.03267/) (2025)
- [What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks](https://mlsanity.com/paper/2305.18365/) (2023)
- [LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Dataset](https://mlsanity.com/paper/2402.09391/) (2024)
- [Large language models predict human sensory judgments across six modalities](https://mlsanity.com/paper/2302.01308/) (2023)
- [Prompt Repetition Improves Non-Reasoning LLMs](https://mlsanity.com/paper/2512.14982/) (2025)
- [Assessing the Chemical Intelligence of Large Language Models](https://mlsanity.com/paper/2505.07735/) (2025)
- [Sniff AI: Is My 'Spicy' Your 'Spicy'? Exploring LLM's Perceptual Alignment with Human Smell Experiences](https://mlsanity.com/paper/2411.06950/) (2024)
- [From Molecules to Mixtures: Learning Representations of Olfactory Mixture Similarity using Inductive Biases](https://mlsanity.com/paper/2501.16271/) (2025)
- [Advancements and limitations of LLMs in replicating human color-word associations](https://mlsanity.com/paper/2411.02116/) (2024)

## Built on by (most cited first, 4 of 4)

- [AromaGen: Interactive Generation of Rich Olfactory Experiences with Multimodal Language Models](https://mlsanity.com/paper/2604.01650/) (2026)
- [What Images Cannot Say: Language-Guided Olfactory Representation Learning](https://mlsanity.com/paper/2607.06402/) (2026)
- [COLIP-2: Olfaction-Vision-Language Embeddings](https://mlsanity.com/paper/2607.17559/) (2026)
- [Beyond Benchmark Accuracy: A Diagnostic Test for LLM Chemical Prediction](https://mlsanity.com/paper/2610.19906/) (2026)

## Abstract (arXiv)

Here we introduce the Olfactory Perception (OP) benchmark, designed to assess the capability of large language models (LLMs) to reason about smell. The benchmark contains 1,010 questions across eight task categories spanning odor classification, odor primary descriptor identification, intensity and pleasantness judgments, multi-descriptor prediction, mixture similarity, olfactory receptor activation, and smell identification from real-world odor sources. Each question is presented in two prompt formats, compound names and isomeric SMILES, to evaluate the effect of molecular representations. Evaluating 21 model configurations across major model families, we find that compound-name prompts consistently outperform isomeric SMILES, with gains ranging from +2.4 to +18.9 percentage points (mean approx +7 points), suggesting current LLMs access olfactory knowledge primarily through lexical associations rather than structural molecular reasoning. The best-performing model reaches 64.4% overall accuracy, which highlights both emerging capabilities and substantial remaining gaps in olfactory reasoning. We further evaluate a subset of the OP across 21 languages and find that aggregating predictions across languages improves olfactory prediction, with AUROC = 0.86 for the best performing language ensemble model. LLMs should be able to handle olfactory and not just visual or aural information.