# ArrivalBench: Agent-Generated Data Pipelines Are Correct Once and Wrong Under Time

arXiv: https://arxiv.org/abs/2610.02363 · HTML: https://mlsanity.com/paper/2610.02363/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 36 references that are in the map.
- First posted on arXiv 1 Oct 2026.

## Builds on (most cited first, 18 of 18)

- [Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows](https://mlsanity.com/paper/2411.07763/) (2024)
- [DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models](https://mlsanity.com/paper/2410.07331/) (2024)
- [BEAVER: An Enterprise Benchmark for Text-to-SQL](https://mlsanity.com/paper/2409.02038/) (2024)
- [KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes](https://mlsanity.com/paper/2506.06541/) (2025)
- [Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environments](https://mlsanity.com/paper/2602.11964/) (2026)
- [ELT-Bench: An End-to-End Benchmark for Evaluating AI Agents on ELT Pipelines](https://mlsanity.com/paper/2504.04808/) (2025)
- [Look-Ahead-Bench: a Standardized Benchmark of Look-ahead Bias in Point-in-Time LLMs for Finance](https://mlsanity.com/paper/2601.13770/) (2026)
- [AgenticDataBench: A Comprehensive Benchmark for Data Agents](https://mlsanity.com/paper/2607.01647/) (2026)
- [Elle: Inferring Isolation Anomalies from Experimental Observations](https://mlsanity.com/paper/2003.10554/) (2020)
- [How Many Tasks Are Enough for Agent Benchmark Decisions? A Replay Analysis of Public LLM Agent Benchmarks](https://mlsanity.com/paper/2607.12338/) (2026)
- [DBSP: Automatic Incremental View Maintenance for Rich Query Languages](https://mlsanity.com/paper/2203.16684/) (2022)
- [ELT-Bench-Verified: Benchmark Quality Issues Underestimate AI Agent Capabilities](https://mlsanity.com/paper/2603.29399/) (2026)
- [AutoStreamPipe: LLM Assisted Automatic Generation of Data Stream Processing Pipelines](https://mlsanity.com/paper/2510.23408/) (2025)
- [The System Prompt Is the Attack Surface: How LLM Agent Configuration Shapes Security and Creates Exploitable Vulnerabilities](https://mlsanity.com/paper/2603.25056/) (2026)
- [Minimal Prompt Perturbations Lead to Code Vulnerabilities: Prompt Fragility and Hidden-State Signals in Coding LLMs](https://mlsanity.com/paper/2605.29737/) (2026)
- [SecureForge: Finding and Preventing Vulnerabilities in LLM-Generated Code via Prompt Optimization](https://mlsanity.com/paper/2605.08382/) (2026)
- [CDR-Bench: Evaluating Faithful Execution of Compositional, Order-Sensitive Data Refinement Recipes](https://mlsanity.com/paper/2606.31435/) (2026)
- [Look-Ahead-Freedom as Temporal Non-Interference: A Verifiable Correctness Property for Backtesting and Agentic Trading Pipelines](https://mlsanity.com/paper/2607.04958/) (2026)

## Abstract (arXiv)

Benchmarks for agent-generated data work grade a pipeline by running it once against a fixed snapshot. ArrivalBench instead re-executes the pipeline an agent leaves behind under adversarial but replayable delivery schedules (late, duplicated, out-of-order and retried records) and requires its final state to equal a batch recomputation of the complete log. Because the oracle recomputes rather than classifies, a wrong table and a crash are distinct verdicts: a crash is visible to monitoring a team already runs, and a wrong table is not. On 40 tasks we built, our reimplementation of single-execution grading certifies 86-100% of the pipelines eleven models produce; re-executing the same artifacts finds 7.0-79.2% of the certified ones silently wrong. The gap is not produced by the repair loop: within the same model and task, pipelines repaired against the snapshot test fail replay about as often as those that passed it first time. In every model, idempotency hazards fail more often than ordering hazards. Separating a wrong answer from a crash also changes how interventions read: a hazard warning cuts one model's silent failure from 48.2% to 10.5% while raising its crash rate from 9.0% to 37.0%, so all-in failure moves only from 51.0% to 44.0%. All eleven arms were independently re-run, and rates moved by at most 5.9 points.