# PilotBench: A Benchmark for General Aviation Agents with Safety Constraints

arXiv: https://arxiv.org/abs/2604.08987 · HTML: https://mlsanity.com/paper/2604.08987/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 4 of its 21 references that are in the map.
- First posted on arXiv 10 Apr 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 4 of 4)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [TartanAviation: Image, Speech, and ADS-B Trajectory Datasets for Terminal Airspace Operations](https://mlsanity.com/paper/2403.03372/) (2024)
- [CHATATC: Large Language Model-Driven Conversational Agents for Supporting Strategic Air Traffic Flow Management](https://mlsanity.com/paper/2402.14850/) (2024)
- [Research on Flight Accidents Prediction based Back Propagation Neural Network](https://mlsanity.com/paper/2406.13954/) (2024)

## Built on by (most cited first, 3 of 3)

- [AeroCopilotBench: A Two-Tier Benchmark for Evaluating LLM Agents as Aviation Copilots in an Interactive Virtual Cockpit Environment](https://mlsanity.com/paper/2608.16349/) (2026)
- [Beyond Semantic Accuracy: Consequence-Aware Evaluation for Safety-Critical Language Understanding](https://mlsanity.com/paper/2608.24621/) (2026)
- [FLY-EVAL++: An Evidence-Driven Evaluation Protocol for Safety-Constrained Flight Prediction with Large Language Models](https://mlsanity.com/paper/2609.04021/) (2026)

## Abstract (arXiv)

As Large Language Models (LLMs) advance toward embodied AI agents operating in physical environments, a fundamental question emerges: can models trained on text corpora reliably reason about complex physics while adhering to safety constraints? We address this through PilotBench, a benchmark evaluating LLMs on safety-critical flight trajectory and attitude prediction. Built from 708 real-world general aviation trajectories spanning nine operationally distinct flight phases with synchronized 34-channel telemetry, PilotBench systematically probes the intersection of semantic understanding and physics-governed prediction through comparative analysis of LLMs and traditional forecasters. We introduce Pilot-Score, a composite metric balancing 60% regression accuracy with 40% instruction adherence and safety compliance. Comparative evaluation across 41 models uncovers a Precision-Controllability Dichotomy: traditional forecasters achieve superior MAE of 7.01 but lack semantic reasoning capabilities, while LLMs gain controllability with 86--89% instruction-following at the cost of 11--14 MAE precision. Phase-stratified analysis further exposes a Dynamic Complexity Gap-LLM performance degrades sharply in high-workload phases such as Climb and Approach, suggesting brittle implicit physics models. These empirical discoveries motivate hybrid architectures combining LLMs' symbolic reasoning with specialized forecasters' numerical precision. PilotBench provides a rigorous foundation for advancing embodied AI in safety-constrained domains.