# ExpStar: Towards Automatic Commentary Generation for Multi-discipline Scientific Experiments

arXiv: https://arxiv.org/abs/2507.09693 · HTML: https://mlsanity.com/paper/2507.09693/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 57 references that are in the map.
- First posted on arXiv 13 Jul 2025.

## Builds on (most cited first, 30 of 33)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://mlsanity.com/paper/2403.05525/) (2024)
- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning](https://mlsanity.com/paper/2302.14115/) (2023)
- [Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos](https://mlsanity.com/paper/2501.13826/) (2025)
- [CogVLM2: Visual Language Models for Image and Video Understanding](https://mlsanity.com/paper/2408.16500/) (2024)
- [MMVU: Measuring Expert-Level Multi-Discipline Video Understanding](https://mlsanity.com/paper/2501.12380/) (2025)
- [Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories](https://mlsanity.com/paper/2306.09224/) (2023)
- [EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameters](https://mlsanity.com/paper/2402.04252/) (2024)
- [Streaming Dense Video Captioning](https://mlsanity.com/paper/2404.01297/) (2024)
- [AutoAD: Movie Description in Context](https://mlsanity.com/paper/2303.16899/) (2023)
- [AutoAD II: The Sequel -- Who, When, and What in Movie Audio Description](https://mlsanity.com/paper/2310.06838/) (2023)
- [MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videos](https://mlsanity.com/paper/2406.08407/) (2024)
- [MatchTime: Towards Automatic Soccer Game Commentary Generation](https://mlsanity.com/paper/2406.18530/) (2024)
- [AutoAD III: The Prequel -- Back to the Pixels](https://mlsanity.com/paper/2404.14412/) (2024)
- [GOAL: A Challenging Knowledge-grounded Video Captioning Benchmark for Real-time Soccer Commentary Generation](https://mlsanity.com/paper/2303.14655/) (2023)
- [Synchronized Video Storytelling: Generating Video Narrations with Structured Storyline](https://mlsanity.com/paper/2405.14040/) (2024)

## Built on by (most cited first, 2 of 2)

- [SAMoRA: Semantic-Aware Mixture of LoRA Experts for Task-Adaptive Learning](https://mlsanity.com/paper/2604.19048/) (2026)
- [DiffCap-Bench: A Comprehensive, Challenging, Robust Benchmark for Image Difference Captioning](https://mlsanity.com/paper/2605.04503/) (2026)

## Abstract (arXiv)

Experiment commentary is crucial in describing the experimental procedures, delving into underlying scientific principles, and incorporating content-related safety guidelines. In practice, human teachers rely heavily on subject-specific expertise and invest significant time preparing such commentary. To address this challenge, we introduce the task of automatic commentary generation across multi-discipline scientific experiments. While recent progress in large multimodal models (LMMs) has demonstrated promising capabilities in video understanding and reasoning, their ability to generate fine-grained and insightful experiment commentary remains largely underexplored. In this paper, we make the following contributions: (i) We construct textitExpInstruct, the first dataset tailored for experiment commentary generation, featuring over 7textitK step-level commentaries across 21 scientific subjects from 3 core disciplines (ie, science, healthcare and engineering). Each sample includes procedural descriptions along with potential scientific principles (eg, chemical equations and physical laws) and safety guidelines. (ii) We propose ExpStar, an automatic experiment commentary generation model that leverages a retrieval-augmented mechanism to adaptively access, evaluate, and utilize external knowledge. (iii) Extensive experiments show that our ExpStar substantially outperforms 14 leading LMMs, which highlights the superiority of our dataset and model. We believe that ExpStar holds great potential for advancing AI-assisted scientific experiment instruction.