# BIABench: Evaluating AI agents on real-world bioimage analysis tasks

arXiv: https://arxiv.org/abs/2609.34274 · HTML: https://mlsanity.com/paper/2609.34274/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use.
- Builds on 30 of its 70 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Placed in

- LLM agents and tool use / [Medical and scientific domain agents](https://mlsanity.com/agents/medical-and-scientific-domain-agents/)

## Builds on (most cited first, 30 of 30)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025)
- [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://mlsanity.com/paper/2410.07095/) (2024)
- [Accelerating scientific discovery with Co-Scientist](https://mlsanity.com/paper/2502.18864/) (2025)
- [Cell Detection with Star-convex Polygons](https://mlsanity.com/paper/1806.03535/) (2018)
- [LAB-Bench: Measuring Capabilities of Language Models for Biology Research](https://mlsanity.com/paper/2407.10362/) (2024)
- [BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology](https://mlsanity.com/paper/2503.00096/) (2025)
- [Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows](https://mlsanity.com/paper/2605.27922/) (2026)
- [MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Research](https://mlsanity.com/paper/2503.13399/) (2025)
- [NuInsSeg: A Fully Annotated Dataset for Nuclei Instance Segmentation in H&E-Stained Histological Images](https://mlsanity.com/paper/2308.01760/) (2023)
- [BioAgent Bench: An AI Agent Evaluation Suite for Bioinformatics](https://mlsanity.com/paper/2601.21800/) (2026)
- [LABBench2: An Improved Benchmark for AI Systems Performing Biology Research](https://mlsanity.com/paper/2604.09554/) (2026)
- [Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks](https://mlsanity.com/paper/2606.12344/) (2026)
- [BioImage.IO Chatbot: A Community-Driven AI Assistant for Integrative Computational Bioimaging](https://mlsanity.com/paper/2310.18351/) (2023)
- [Multimodal Large Language Models for Bioimage Analysis](https://mlsanity.com/paper/2407.19778/) (2024)
- [Tracking one-in-a-million: Large-scale benchmark for microbial single-cell tracking with experiment-aware robustness metrics](https://mlsanity.com/paper/2411.00552/) (2024)
- [BixBench3: Benchmarking AI agents on research-study-scale computational biology tasks](https://mlsanity.com/paper/2608.25286/) (2026)
- [GenCellAgent: Generalizable, Training-Free Cellular Image Segmentation via Large Language Model Agents](https://mlsanity.com/paper/2510.13896/) (2025)
- [ReX-MLE: The Autonomous Agent Benchmark for Medical Imaging Challenges](https://mlsanity.com/paper/2512.17838/) (2025)
- [BioXArena: Benchmarking LLM Agents on Multi-Modal Biomedical Machine Learning Tasks](https://mlsanity.com/paper/2605.15766/) (2026)
- [Agentic-J: An AI Agent for Biological Microscopy Image Analysis](https://mlsanity.com/paper/2606.02080/) (2026)
- [A novel attention mechanism for noise-adaptive and robust segmentation of microtubules in microscopy images](https://mlsanity.com/paper/2507.07800/) (2025)
- [CellFMCount: A Fluorescence Microscopy Dataset, Benchmark, and Methods for Cell Counting](https://mlsanity.com/paper/2511.19351/) (2025)
- [StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows](https://mlsanity.com/paper/2608.17800/) (2026)

## Abstract (arXiv)

Artificial-intelligence (AI) agents hold promise for automating bioimage analysis, yet no benchmark evaluates whether they can carry out real-world analyses end to end. Such analyses are hard for agents because 2D images, 3D volumes and time-lapse sequences are often too large to read as context, so an agent must choose and run an analysis through code, specialized software and rendered views. Published studies make this capability testable, because each pairs raw images with a peer-reviewed result. We introduce BIABench, a benchmark of 16 tasks reconstructed from published biological studies that retain their scientific questions, imaging data and ground truth. The tasks span eleven analysis subtasks and modalities from H&E histology to single-molecule localization microscopy. Each submission receives an outcome score, which compares the output files with the ground truth using field-standard metrics, and a process score, in which a vision-language model judges method choice and quality control against an expert-written rubric. We evaluated general-purpose and biology-specific agents across several language models, with repeated runs of every task. Routine two-dimensional tasks were solved well, but on some tasks that added a third dimension or a time axis no agent scored above 0.19. Neither biological specialization, stronger models nor detailed expert instructions closed this gap. The agents were also unreliable, with scores varying more between repeated runs of one agent than between different agents, and without ground truth a correct run could not be told from a wrong one by its process score or by the time spent. Released openly with its data and code, BIABench provides a verifiable framework for evaluating, and eventually training, agents for reliable long-horizon bioimage analysis.