Frontier Autolab: Organizational Memory, Adversarial Dissent and Temporal Leakage in Multi-Agent LLM Firms Across Fifty Years of Technological Change
On the map: LLM agents and tool use → Financial trading and forecasting agents also fits: Social simulation and multi-agent debate
Abstract (arXiv)
Multi-agent LLM systems are increasingly structured like organizations, with roles, critics and shared memory, yet they are evaluated on tasks that last minutes. We ask how such an organization behaves when the ground it stands on keeps moving. Frontier Autolab is a long-horizon testbed in which one simulated firm, voiced by sixteen role personas and a dedicated Red Team, must re-found itself in nine technology eras from 1990 to 2040. Each era is temporally gated: the firm decides from a dated briefing, a historian-judge then reveals what happened and scores the decision on a five-dimension rubric, and lessons enter a persistent Playbook. Six eras are scored against history, one against the live market and two are open forecasts. Across four trajectories (36 era decisions, 180 subscores) we find a consistent foresight-commitment gap: in all 24 historically scored eras the judge rated the firm's recognition of the coming shift above its choice of where to build (mean gap 1.9 points on a 10-point scale), because boards chose the layer their existing assets could reach. Organizational design shaped long-run character. A Red Team armed with numeric kill gates produced fifty years of gated pilots and no product, and the rubric rated this firm highest; firms whose memory stored market-structure lessons pivoted every era, while a firm whose memory stored only validation procedure kept one method throughout. We also show why such results are hard to trust. Scores rise across eras in every run while the judge's own hindsight subscore falls (within-run r = -0.58), so apparent learning is confounded with recall of history, and we trace further distortions to self-judging, briefing selection and score aggregation. We release all records and an API harness, and specify fictional and post-cutoff eras that would turn the testbed into a benchmark.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use. Builds on 21 of its 37 references that are in the map. First posted on arXiv 29 Sep 2026.
Builds on
21 of 37 references.
References · 21
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
- 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
- 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
- 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
- 2023Large Language Models are not Fair Evaluatorscited by 1,050
- 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
- 2023ChatDev: Communicative Agents for Software Developmentcited by 893
- 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
- 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
9 more
- 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
- 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 580
- 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
- 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 175
- 2024ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilitiescited by 104
- 2024Approaching Human-Level Forecasting with Language Modelscited by 102
- 2024Dated Data: Tracing Knowledge Cutoffs in Large Language Modelscited by 48
- 2023Assessing Look-Ahead Bias in Stock Return Predictions Generated By GPT Sentiment Analysiscited by 29
- 2025Pitfalls in Evaluating Language Model Forecasterscited by 23
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Can LLM Agents Sustain Long-Horizon Organizational Dynamics?
- 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agents
- 2026Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents
- 2026Loop-Back Authority in LLM Agent Teams: A Paired Experiment on Flat and Hierarchical Coordination
- 2026Business Arena: Benchmarking LLM Agents in a Realistic Marketplace
- 2026How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making
- 2026Organizational Control Layer: Governance Infrastructure at the Execution Boundary of LLM Agent Systems
- 2026Agent Memory: Characterization and System Implications of Stateful Long-Horizon Workloads