mlsanityData to 7 Oct 2026

Remote Labor Index: Measuring AI Automation of Remote Work

arXiv 2510.26787 ↗First posted 30 Oct 2025cs.LG cs.AI cs.CLcited by 18

Not on the map. Why?

Abstract (arXiv)

AIs have made rapid progress on research-oriented benchmarks of knowledge and reasoning, but it remains unclear how these gains translate into economic value and automation. To measure this, we introduce the Remote Labor Index (RLI), a broadly multi-sector benchmark comprising real-world, economically valuable projects designed to evaluate end-to-end agent performance in practical settings. AI agents perform near the floor on RLI, with the highest-performing agent achieving an automation rate of 2.5%. These results help ground discussions of AI automation in empirical evidence, setting a common basis for tracking AI impacts and enabling stakeholders to proactively navigate AI-driven labor automation.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 21 of its 34 references that are in the map. First posted on arXiv 30 Oct 2025.

Builds on

21 of 34 references.

References · 21

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  4. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  5. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  6. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  7. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  8. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  9. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  10. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  11. 2025Humanity's Last Examcited by 467
  12. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
9 more
  1. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 286
  2. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIcited by 227
  3. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 153
  4. 2025GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Taskscited by 142
  5. 2025SWE-Lancer: Can Frontier LLMs Earn 1 Million from Real-World Freelance Software Engineering?cited by 99
  6. 2025HCAST: Human-Calibrated Autonomy Software Taskscited by 19
  7. 2025RExBench: Can coding agents autonomously implement AI research extensions?cited by 17
  8. 2024Scenarios for the Transition to AGIcited by 13
  9. 2025The AI Productivity Index (APEX)cited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 15.

18 papers.

Sort:

By area:

  1. 2025PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoningcited by 29
  2. 2026The 2025 AI Agent Index: Documenting Technical and Safety Features of Deployed Agentic AI Systemscited by 27
  3. 2026SafePro: Evaluating the Safety of Professional-Level AI Agentscited by 10
  4. 2026How Well Does Agent Development Reflect Real-World Work?cited by 10
  5. 2026LLMs Corrupt Your Documents When You Delegatecited by 9
  6. 2026Measuring Progress Toward AGI: A Cognitive Frameworkcited by 6
  7. 2026BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agentscited by 6
  8. 2025UpBench: A Dynamically Evolving Real-World Labor-Market Agentic Benchmark Framework Built for Human-Centric AIcited by 2
  9. 2025Reliable agent engineering should integrate machine-compatible organizational principlescited by 2
  10. 2026EcoGym: Evaluating LLMs for Long-Horizon Plan-and-Execute in Interactive Economiescited by 2
  11. 2026Self-Sovereign Agentcited by 2
  12. 2026SpreadsheetArena: Decomposing Preference in LLM Generation of Spreadsheet Workbookscited by 1
6 more
  1. 2026CollabSkill: Evaluating Human-Agent Collaboration On Real-World Taskscited by 1
  2. 2026OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Groundingcited by 1
  3. 2026The Open Source Economic Index of AI Adoption and Capabilitycited by 0
  4. 2026Economic Evaluations of Language Modelscited by 0
  5. 2026Walking on the DARKSIDEcited by 0
  6. 2026DAYJOB: A Benchmark for Long-Horizon Professional Workcited by 0