# DevicesWorld: Benchmarking Cross-Device Agents in Heterogeneous Environments

arXiv: https://arxiv.org/abs/2607.13465 · HTML: https://mlsanity.com/paper/2607.13465/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 27 references that are in the map.
- First posted on arXiv 15 Jul 2026.

## Placed in

- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 20 of 20)

- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [Why Do Multi-Agent LLM Systems Fail?](https://mlsanity.com/paper/2503.13657/) (2025)
- [AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents](https://mlsanity.com/paper/2405.14573/) (2024)
- [AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents](https://mlsanity.com/paper/2407.18901/) (2024)
- [WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?](https://mlsanity.com/paper/2403.07718/) (2024)
- [Android in the Wild: A Large-Scale Dataset for Android Device Control](https://mlsanity.com/paper/2307.10088/) (2023)
- [The BrowserGym Ecosystem for Web Agent Research](https://mlsanity.com/paper/2412.05467/) (2024)
- [Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents](https://mlsanity.com/paper/2504.00906/) (2025)
- [WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks](https://mlsanity.com/paper/2407.05291/) (2024)
- [Benchmarking Mobile Device Control Agents across Diverse Configurations](https://mlsanity.com/paper/2404.16660/) (2024)
- [CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents](https://mlsanity.com/paper/2407.01511/) (2024)
- [HomeBench: Evaluating LLMs in Smart Homes with Valid and Invalid Instructions Across Single and Multiple Devices](https://mlsanity.com/paper/2505.19628/) (2025)
- [SimuHome: A Temporal- and Environment-Aware Benchmark for Smart Home LLM Agents](https://mlsanity.com/paper/2509.24282/) (2025)
- [OpenComputer: Verifiable Software Worlds for Computer-Use Agents](https://mlsanity.com/paper/2605.19769/) (2026)
- [WindowsWorld: A Process-Centric Benchmark of Autonomous GUI Agents in Professional Cross-Application Environments](https://mlsanity.com/paper/2604.27776/) (2026)
- [SMH-Bench: Benchmarking LLM Agents for Environment-Grounded Reasoning and Action in Smart Homes](https://mlsanity.com/paper/2606.01912/) (2026)
- [HomeFlow: A Data Flywheel for Smart Home Agent Training with Verifiable Simulation](https://mlsanity.com/paper/2606.01230/) (2026)
- [Challenges in Mobile Multi-Device Ecosystems](https://mlsanity.com/paper/1605.07760/) (2016)
- [AdaM: Adapting Multi-User Interfaces for Collaborative Environments in Real-Time](https://mlsanity.com/paper/1803.01166/) (2018)

## Built on by (most cited first, 1 of 1)

- [Words to World: Benchmarking LLMs in Interactive Physical Environments](https://mlsanity.com/paper/2610.15228/) (2026)

## Abstract (arXiv)

LLM-based agents have rapidly improved at operating individual digital environments such as mobile applications, desktop systems, and smart homes. However, real-world user goals often span multiple devices: information may come from a phone, be processed on a desktop, and the result may need to appear on another device. Most existing benchmarks center on a single dominant execution environment, making it difficult to evaluate whether agents can acquire and integrate information across heterogeneous devices and complete end-to-end tasks with cross-device dependencies. We introduce DevicesWorld, a large-scale executable benchmark for cross-device collaborative operation. DevicesWorld contains 6,140 tasks and integrates three classes of device environments -- mobile, desktop, and IoT -- into a unified cross-device interaction and evaluation framework. Each task defines a natural-language user goal, participating devices and initial states, executable actions, rule-based verifiers, and a cleanup procedure. A multi-stage construction and quality-control pipeline keeps tasks close to realistic user needs while allowing final outcomes to be automatically verified from device states and generated files. We evaluate five frontier LLM-agent systems on a fixed evaluation set. All methods achieve low success rates, with the best reaching only 12.5%. Among failed runs, about 28.7% satisfy at least one scoring condition yet still fail the full task. Trajectories show that agents become stuck acquiring information or manipulating interfaces, confuse source and output devices, or terminate before all conditions are jointly satisfied. DevicesWorld turns cross-device collaborative operation into an executable, reproducible, and diagnostically useful evaluation problem for research on reliable cross-device agents.