mlsanityData to 7 Oct 2026

EdgeBench: Unveiling Scaling Laws of Learning from Real-World Environments

arXiv 2607.05155 ↗First posted 6 Jul 2026cs.CL cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Pretraining scaling laws reveal that model capability improves predictably with data and compute. But learning from real world environments after deployment remains far less understood. Analyzing roughly 38,000 hours of agent interaction with the environment across 134 real world tasks, we find, to the best of our knowledge, the first evidence that overall performance during environment learning follows a log-sigmoid scaling law with remarkably high precision, reaching R^2 = 0.998. Across model generations, we also find that agent learning speed roughly doubles every three months. This discovery stems from EdgeBench, a suite of 134 real world tasks with ultra-long horizons, spanning scientific discovery, software engineering, combinatorial optimization, professional knowledge work, formal mathematics, and interactive games. Each task sustains at least 12 hours of continuous agent operation under rich, multilevel feedback, and is built through substantial expert effort. We publicly release 51 tasks and our full evaluation framework to accelerate the study of how agents learn from real world experience.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 57 of its 138 references that are in the map. First posted on arXiv 6 Jul 2026.

Builds on

57 of 138 references.

References · 57

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Invariant Risk Minimizationcited by 1,832
  3. 2016Group Equivariant Convolutional Networkscited by 1,154
  4. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  5. 2013Spectral Networks and Locally Connected Networks on Graphscited by 1,097
  6. 2015Causal inference using invariant prediction: identification and confidence intervalscited by 533
  7. 2017Meta-learners for Estimating Heterogeneous Treatment Effects using Machine Learningcited by 475
  8. 2022MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fieldscited by 467
  9. 2018Pyro: Deep Universal Probabilistic Programmingcited by 379
  10. 2017Quasi-Oracle Estimation of Heterogeneous Treatment Effectscited by 328
  11. 2018On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groupscited by 296
  12. 2020Equivariant Flows: Exact Likelihood Generative Learning for Symmetric Densitiescited by 248
38 more
  1. 2021Towards Causal Representation Learningcited by 207
  2. 2019Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyrocited by 187
  3. 2016Automatic Differentiation Variational Inferencecited by 159
  4. 2023Equivariant flow matchingcited by 131
  5. 2022Semiparametric doubly robust targeted double machine learning: a reviewcited by 129
  6. 2023A Survey on Causal Discovery: Theory and Practicecited by 63
  7. 2017Frequentist Consistency of Variational Bayescited by 60
  8. 2021Causal Matrix Completioncited by 45
  9. 2019Equivariant Hamiltonian Flowscited by 39
  10. 2023SE(3) Equivariant Augmented Coupling Flowscited by 37
  11. 2017Causal inference for social network datacited by 30
  12. 2022Symmetry Group Equivariant Architectures for Physicscited by 23
  13. 2013Bayesian Models of Graphs, Arrays and Other Exchangeable Random Structurescited by 22
  14. 2023A Unified Framework to Enforce, Discover, and Promote Symmetry in Machine Learningcited by 22
  15. 2017Time series experiments and causal estimands: exact randomization tests and tradingcited by 21
  16. 2017Group invariance principles for causal generative modelscited by 20
  17. 2020Causal Inference with Spatio-temporal Data: Estimating the Effects of Airstrikes on Insurgent Violence in Iraqcited by 14
  18. 2016Regularization and confounding in linear regression for treatment effect estimationcited by 12
  19. 2024Do Finetti: On Causal Effects for Exchangeable Datacited by 11
  20. 2024Simplifying debiased inference via automatic differentiation and probabilistic programmingcited by 10
  21. 2022Using Embeddings for Causal Estimation of Peer Influence in Social Networkscited by 9
  22. 2023Spatial causal inference in the presence of unmeasured confounding and interferencecited by 9
  23. 2025Kinetic Langevin Diffusion for Crystalline Materials Generationcited by 9
  24. 2023SymmPI: Predictive Inference for Data with Group Symmetriescited by 8
  25. 2016Adjusting for Unmeasured Spatial Confounding with Distance Adjusted Propensity Score Matchingcited by 7
  26. 2020Generalized propensity score approach to causal inference with spatial interferencecited by 7
  27. 2022Phenomenological Causalitycited by 7
  28. 2024Hierarchical Causal Modelscited by 7
  29. 2025Towards Identifiability of Hierarchical Temporal Causal Representation Learningcited by 7
  30. 2017Limitations of design-based causal inference and A/B testing under arbitrary and network interferencecited by 6
  31. 2017Better together? Statistical learning in models made of modulescited by 6
  32. 2022Estimating Social Influence from Observational Datacited by 5
  33. 2023Representing and Learning Functions Invariant Under Crystallographic Groupscited by 5
  34. 2023Randomization Inference When N Equals Onecited by 5
  35. 2022Graphical criteria for the identification of marginal causal effects in continuous-time survival and event-history analysescited by 4
  36. 2024Automated Efficient Estimation using Monte Carlo Efficient Influence Functionscited by 4
  37. 2021Instrumental variables, spatial confounding and interferencecited by 3
  38. 2021A causal inference framework for spatial confoundingcited by 3

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 12.

12 papers.

  1. 2026Rethinking the Evaluation of Harness Evolution for Agentscited by 39
  2. 2026Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AIcited by 6
  3. 2026The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillationcited by 3
  4. 2026PATH-Bench: Path-Dependent Evaluation of Lifelong Agentscited by 1
  5. 2026GDPevo: Evaluating Agent Self-Evolution on Real Business Taskscited by 1
  6. 2026E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operationcited by 1
  7. 2026ExplorationBench: Measuring AI Systems' Exploration in Verifiable Alien Worldscited by 0
  8. 2026Long-Horizon Scaling: How Model Capabilities Shape the Returns to Computationcited by 0
  9. 2026EasyPPO: Stabilizing the Critic Is Keycited by 0
  10. 2026Context Language Modelscited by 0
  11. 2026ReLiveGym: Evaluating Long-Lived Agents over Weeks of Replayed Realitycited by 0
  12. 2026SciExam for ENSO: Can AI Agents Build Climate Models?cited by 0