# LAMMI-Pathology: A Tool-Centric Bottom-Up LVLM-Agent Framework for Molecularly Informed Medical Intelligence in Pathology arXiv: https://arxiv.org/abs/2602.18773 ยท HTML: https://mlsanity.com/paper/2602.18773/ ## Key facts - Cited by 3 papers in the map (data through 14 Oct 2026). - Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use. - Builds on 36 of its 62 references that are in the map. - First posted on arXiv 21 Feb 2026. ## Placed in - LLM agents and tool use / [Medical and scientific domain agents](https://mlsanity.com/agents/medical-and-scientific-domain-agents/) ## Builds on (most cited first, 30 of 36) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021) - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022) - [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020) - [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022) - [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025) - [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022) - [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025) - [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019) - [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023) - [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023) - [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024) - [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022) - [ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases](https://mlsanity.com/paper/2306.05301/) (2023) - [ToolACE: Winning the Points of LLM Function Calling](https://mlsanity.com/paper/2409.00920/) (2024) - [Efficient Guided Generation for Large Language Models](https://mlsanity.com/paper/2307.09702/) (2023) - [TALM: Tool Augmented Language Models](https://mlsanity.com/paper/2205.12255/) (2022) - [The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Survey](https://mlsanity.com/paper/2404.11584/) (2024) - [Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images](https://mlsanity.com/paper/2306.07831/) (2023) - [HEST-1k: A Dataset for Spatial Transcriptomics and Histology Image Analysis](https://mlsanity.com/paper/2406.16192/) (2024) - [Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videos](https://mlsanity.com/paper/2312.04746/) (2023) - [GTA: A Benchmark for General Tool Agents](https://mlsanity.com/paper/2407.08713/) (2024) - [A Review of Prominent Paradigms for LLM-Based Agents: Tool Use (Including RAG), Planning, and Feedback Learning](https://mlsanity.com/paper/2406.05804/) (2024) - [DeepAgent: A General Reasoning Agent with Scalable Toolsets](https://mlsanity.com/paper/2510.21618/) (2025) - [MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learning](https://mlsanity.com/paper/2401.10727/) (2024) - [PathAsst: A Generative Foundation AI Assistant Towards Artificial General Intelligence of Pathology](https://mlsanity.com/paper/2305.15072/) (2023) - [Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage](https://mlsanity.com/paper/2412.15606/) (2024) - [LLM Agents Making Agent Tools](https://mlsanity.com/paper/2502.11705/) (2025) - [Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations](https://mlsanity.com/paper/2109.14989/) (2021) - [PathMMU: A Massive Multimodal Expert-Level Benchmark for Understanding and Reasoning in Pathology](https://mlsanity.com/paper/2401.16355/) (2024) ## Built on by (most cited first, 3 of 3) - [Learning CLI Agents with Structured Action Credit under Selective Observation](https://mlsanity.com/paper/2605.08013/) (2026) - [Democratizing and accelerating AI-driven pathology research through agentic intelligence](https://mlsanity.com/paper/2606.20677/) (2026) - [Evaluating Agentic Bioinformatics through Function, Evidence, and Validation](https://mlsanity.com/paper/2607.27556/) (2026) ## Abstract (arXiv) The emergence of tool-calling-based agent systems introduces a more evidence-driven paradigm for pathology image analysis in contrast to the coarse-grained text-image diagnostic approaches. With the recent large-scale experimental adoption of spatial transcriptomics technologies, molecularly validated pathological diagnosis is becoming increasingly open and accessible. In this work, we propose LAMMI-Pathology (LVLM-Agent System for Molecularly Informed Medical Intelligence in Pathology), a scalable agent framework for domain-specific agent tool-calling. LAMMI-Pathology adopts a tool-centric, bottom-up architecture in which customized domain-adaptive tools serve as the foundation. These tools are clustered by domain style to form component agents, which are then coordinated through a top-level planner hierarchically, avoiding excessively long context lengths that could induce task drift. Based on that, we introduce a novel trajectory construction mechanism based on Atomic Execution Nodes (AENs), which serve as reliable and composable units for building semi-simulated reasoning trajectories that capture credible agent-tool interactions. Building on this foundation, we develop a trajectory-aware fine-tuning strategy that aligns the planner's decision-making process with these multi-step reasoning trajectories, thereby enhancing inference robustness in pathology understanding and its adaptive use of the customized toolset.