# SentinelBench: A Benchmark for Long-Running Monitoring Agents

arXiv: https://arxiv.org/abs/2606.05342 · HTML: https://mlsanity.com/paper/2606.05342/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 46 references that are in the map.
- First posted on arXiv 3 Jun 2026.

## Builds on (most cited first, 30 of 34)

- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models](https://mlsanity.com/paper/2401.13919/) (2024)
- [AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents](https://mlsanity.com/paper/2407.18901/) (2024)
- [WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?](https://mlsanity.com/paper/2403.07718/) (2024)
- [TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks](https://mlsanity.com/paper/2412.14161/) (2024)
- [RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts](https://mlsanity.com/paper/2411.15114/) (2024)
- [The BrowserGym Ecosystem for Web Agent Research](https://mlsanity.com/paper/2412.05467/) (2024)
- [AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?](https://mlsanity.com/paper/2407.15711/) (2024)
- [Measuring AI Ability to Complete Long Software Tasks](https://mlsanity.com/paper/2503.14499/) (2025)
- [ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents](https://mlsanity.com/paper/2410.06703/) (2024)
- [The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs](https://mlsanity.com/paper/2509.09677/) (2025)
- [OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application Workflows](https://mlsanity.com/paper/2508.09124/) (2025)
- [ACE-Step: A Step Towards Music Generation Foundation Model](https://mlsanity.com/paper/2506.00045/) (2025)
- [AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications](https://mlsanity.com/paper/2602.22769/) (2026)
- [Magentic-UI: Towards Human-in-the-loop Agentic Systems](https://mlsanity.com/paper/2507.22358/) (2025)
- [Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environments](https://mlsanity.com/paper/2602.11964/) (2026)
- [ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces](https://mlsanity.com/paper/2604.05172/) (2026)
- [ARE: Scaling Up Agent Environments and Evaluations](https://mlsanity.com/paper/2509.17158/) (2025)
- [The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break](https://mlsanity.com/paper/2604.11978/) (2026)
- [HCAST: Human-Calibrated Autonomy Software Tasks](https://mlsanity.com/paper/2503.17354/) (2025)
- [Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks](https://mlsanity.com/paper/2604.24964/) (2026)
- [Proactive Agent Research Environment: Simulating Active Users to Evaluate Proactive Assistants](https://mlsanity.com/paper/2604.00842/) (2026)
- [ASTRA-bench: Evaluating Tool-Use Agent Reasoning and Action Planning with Personal User Context](https://mlsanity.com/paper/2603.01357/) (2026)
- [WebGames: Challenging General-Purpose Web-Browsing AI Agents](https://mlsanity.com/paper/2502.18356/) (2025)
- [LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning](https://mlsanity.com/paper/2604.14140/) (2026)

## Built on by (most cited first, 3 of 3)

- [Fara-1.5: Scalable Learning Environments for Computer Use Agents](https://mlsanity.com/paper/2606.20785/) (2026)
- [Agents in the Large: Perception-Centered Architecture for Persistent Agents](https://mlsanity.com/paper/2608.30478/) (2026)
- [ReLiveGym: Evaluating Long-Lived Agents over Weeks of Replayed Reality](https://mlsanity.com/paper/2610.00710/) (2026)

## Abstract (arXiv)

AI agents are increasingly asked to carry out work that spans minutes, hours, or longer. Yet the default model of agent behavior is continuous action: issuing tool calls, refreshing pages, searching for alternatives, or otherwise trying to force progress. This is the wrong approach for many long-running tasks, which are better served by a strategy of sustained attention. Instead, agents should monitor an environment, notice when an external event makes progress possible, then respond promptly without wasting resources while waiting. To measure progress on this class of tasks, we introduce SentinelBench, an open-source benchmark for time-evolving monitoring tasks. SentinelBench contains 100 tasks across 10 synthetic web environments, including email, calendars, finance, professional networking, and entertainment. Each environment exposes a live web interface and replays a scripted sequence of events, requiring agents to navigate and reason about web pages whose state shifts underfoot. SentinelBench measures task completion, reaction time, and resource use, exposing the tradeoff between responsiveness and cost. We report results across three models and two browser-agent harnesses, establishing performance baselines for future comparison and demonstrating how agent design choices can dramatically impact key metrics. Together, these results show that SentinelBench distinguishes meaningful differences in agent behavior.