mlsanityData to 7 Oct 2026

Picking the Right Specialist: Attentive Neural Process-based Selection of Task-Specialized Models as Tools for Agentic Healthcare Systems

arXiv 2602.14901 ↗First posted 16 Feb 2026cs.LG cs.AI cs.CVcited by 2

On the map: LLM agents and tool use → Medical and scientific domain agents

Abstract (arXiv)

Task-specialized models form the backbone of agentic healthcare systems, enabling the agents to answer clinical queries across tasks such as disease diagnosis, localization, and report generation. Yet, for a given task, a single "best" model rarely exists. In practice, each task is better served by multiple competing specialist models where different models excel on different data samples. As a result, for any given query, agents must reliably select the right specialist model from a heterogeneous pool of tool candidates. To this end, we introduce ToolSelect, which adaptively learns model selection for tools by minimizing a population risk over sampled specialist tool candidates using a consistent surrogate of the task-conditional selection loss. Concretely, we propose an Attentive Neural Process-based selector conditioned on the query and per-model behavioral summaries to choose among the specialist models. Motivated by the absence of any established testbed, we, for the first time, introduce an agentic Chest X-ray environment equipped with a diverse suite of task-specialized models (17 disease detection, 19 report generation, 6 visual grounding, and 13 VQA) and develop ToolSelectBench, a benchmark of 1448 queries. Our results demonstrate that ToolSelect consistently outperforms 10 SOTA methods across four different task families.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use. Builds on 56 of its 70 references that are in the map. First posted on arXiv 16 Feb 2026.

Builds on

56 of 70 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025EndoCogniAgent: Closed-Loop Agentic Reasoning with Self-Consistency Validation for Endoscopic Diagnosis10 Aug
  2. 2025Surgical AI Copilot: Energy-Based Fourier Gradient Low-Rank Adaptation for Surgical LLM Agent Reasoning and Planning12 Mar
  3. 2025A Survey of LLM-based Agents in Medicine: How far are we from Baymax?16 Feb
  4. 2024KG4Diagnosis: A Hierarchical Multi-Agent LLM Framework with Knowledge Graph Enhancement for Medical Diagnosis22 Dec
  5. 2024CXR-Agent: Vision-language models for chest X-ray interpretation with uncertainty aware radiology reporting11 Jul
  6. 2024MMedAgent: Learning to Use Medical Tools with Multi-modal Agent2 Jul

Other references · 50

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2023A Survey of Large Language Modelscited by 2,537
  7. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  8. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  9. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  10. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  11. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  12. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
38 more
  1. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  2. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  3. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  4. 2023A Comprehensive Overview of Large Language Modelscited by 639
  5. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  6. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  7. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  8. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  9. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 311
  10. 2019Attentive Neural Processescited by 280
  11. 2020VinDr-CXR: An open dataset of chest X-rays with radiologist's annotationscited by 236
  12. 2023Cross-Entropy Loss Functions: Theoretical Analysis and Applicationscited by 206
  13. 2024ToolACE: Winning the Points of LLM Function Callingcited by 204
  14. 2022TALM: Tool Augmented Language Modelscited by 197
  15. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 188
  16. 2023Large Language Model Routing with Benchmark Datasetscited by 166
  17. 2024MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Makingcited by 162
  18. 2024GraphRouter: A Graph-based Router for LLM Selectionscited by 154
  19. 2024EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instructioncited by 153
  20. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 140
  21. 2021TorchXRayVision: A library of chest X-ray datasets and modelscited by 131
  22. 2023AutoMix: Automatically Mixing Language Modelscited by 116
  23. 2023ToolChain*: Efficient Action Space Navigation in Large Language Models with A* Searchcited by 103
  24. 2023Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modelingcited by 94
  25. 2024A Vision-Language Foundation Model to Enhance Efficiency of Chest X-ray Interpretationcited by 79
  26. 2024A Survey of Large Language Models for Graphscited by 71
  27. 2025MedRAX: Medical Reasoning Agent for Chest X-raycited by 68
  28. 2024TensorOpera Router: A Multi-Model Router for Efficient LLM Inferencecited by 59
  29. 2024Learning to Use Tools via Cooperative and Interactive Agentscited by 54
  30. 2024Harnessing the Power of Multiple Minds: Lessons Learned from LLM Routingcited by 43
  31. 2024Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Treescited by 38
  32. 2024ToolPlanner: A Tool Augmented LLM for Multi Granularity Instructions with Path Planning and Feedbackcited by 28
  33. 2024EVA-X: A Foundation Model for General Chest X-ray Analysis with Self-supervised Learningcited by 22
  34. 2025ReXVQA: A Large-scale Visual Question Answering Benchmark for Generalist Chest X-ray Understandingcited by 20
  35. 2023OrchestraLLM: Efficient Orchestration of Language Models for Dialogue State Trackingcited by 17
  36. 2022Generalizability of Machine Learning Models: Quantitative Evaluation of Three Methodological Pitfallscited by 12
  37. 2021How Reliable Are Out-of-Distribution Generalization Methods for Medical Image Segmentation?cited by 10
  38. 2025FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Datacited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Volumetric Radiology AI in the Era of Multimodal Large Language Modelscited by 0
  2. 2026What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Why Specialist Models Still Matter: A Heterogeneous Multi-Agent Paradigm for Medical Artificial Intelligence
  2. 2026MedCTA: A Benchmark for Clinical Tool Agents
  3. 2025Learning to Be A Doctor: Searching for Effective Medical Agent Architectures
  4. 2026From Given to Gathered Evidence: Agentic Learning for Longitudinal Medical Reasoning
  5. 2026AutoMedBench: Towards Medical AutoResearch with Agentic AI Models
  6. 2026Which Tool Response Should I Trust? Tool-Expertise-Aware Chest X-ray Agent with Multimodal Agentic Learning
  7. 2026HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agents
  8. 2025PASS: Probabilistic Agentic Supernet Sampling for Interpretable and Adaptive Chest X-Ray Reasoning