Lightweight Structured Multimodal Reasoning for Clinical Scene Understanding in Robotics
On the map: LLM reasoning → Clinical and biomedical applications
Abstract (arXiv)
Healthcare robotics requires robust multimodal perception and reasoning to ensure safety in dynamic clinical environments. Current Vision-Language Models (VLMs) demonstrate strong general-purpose capabilities but remain limited in temporal reasoning, uncertainty estimation, and structured outputs needed for robotic planning. We present a lightweight agentic multimodal framework for video-based scene understanding. Combining the Qwen2.5-VL-3B-Instruct model with a SmolAgent-based orchestration layer, it supports chain-of-thought reasoning, speech-vision fusion, and dynamic tool invocation. The framework generates structured scene graphs and leverages a hybrid retrieval module for interpretable and adaptive reasoning. Evaluations on the Video-MME benchmark and a custom clinical dataset show competitive accuracy and improved robustness compared to state-of-the-art VLMs, demonstrating its potential for applications in robot-assisted surgery, patient monitoring, and decision support.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 20 of its 34 references that are in the map. First posted on arXiv 26 Sep 2025.
Builds on
20 of 34 references.
Closest ancestors · 1
Same direction, newest first
Other references · 19
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2023GPT-4 Technical Reportcited by 18,132
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
- 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
- 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
- 2024LightRAG: Simple and Fast Retrieval-Augmented Generationcited by 310
7 more
- 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
- 2024Enhancing Video-LLM Reasoning via Agent-of-Thoughts Distillationcited by 36
- 2023Improving the Reliability of Large Language Models by Leveraging Uncertainty-Aware In-Context Learningcited by 29
- 2024From Efficient Multimodal Models to World Models: A Surveycited by 13
- 2025ViQAgent: Zero-Shot Video Question Answering via Agent with Open-Vocabulary Grounding Validationcited by 5
- 2025From Standalone LLMs to Integrated Intelligence: A Survey of Compound Al Systemscited by 4
- 2025Enhancing Surgical Documentation through Multimodal Visual-Temporal Transformers and Generative AIcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025SurgRAW: Multi-Agent Workflow with Chain of Thought Reasoning for Robotic Surgical Video Analysis
- 2026MedVLA: A Hierarchical Vision-Language-Action Framework for Closed-Loop Precision Medical Robot Manipulation
- 2026MEDVISTAGYM: A Scalable Training Environment for Thinking with Medical Images via Tool-Integrated Reinforcement Learning
- 2025Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes
- 2026Lightweight Visual Reasoning for Socially-Aware Robots
- 2026MedMASLab: A Unified Orchestration Framework for Benchmarking Multimodal Medical Multi-Agent Systems
- 2025Seeing and Reasoning with Confidence: Supercharging Multimodal LLMs with an Uncertainty-Aware Agentic Framework
- 2025Systematic Evaluation of Large Vision-Language Models for Surgical Artificial Intelligence