mlsanityData to 14 Oct 2026

MA-Bench: Measuring Operational Planning in Language Models

arXiv 2610.19138 ↗First posted 13 Oct 2026cs.LG cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

How well can a language model turn a large, incomplete and partly contradictory body of documents into a plan when no single correct answer exists? We introduce Mission Analysis (MA)-Bench, built on mission analysis, the first step of military operational planning, in which a staff reads the full situation and states the problem before anyone proposes a solution. Its 10 expert-built scenarios pair synthetic packets of ten to twelve documents with 1,257 expert-written pass/fail requirements, each containing information on their source, question type, and importance level. A reasoning-model judge grades each brief against the requirements without seeing the documents; it grades like a fifth credentialed planner, leaving a four-planner panel's agreement unchanged (Fleiss kappa 0.42 to 0.41), and a judge from a second provider reproduces the model ranking (Spearman 0.94). Across twenty frontier and open-weights models the best gets 62% of the rubric items right. Models are strongest on facts stated in the documents (58%) and weakest on judgments that must be inferred (33%), and six failures in ten are omissions rather than errors. MA-Bench grades the determinations, not the prose. We release the packets, rubrics, prompts, scorer and every graded brief with item-level verdicts.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 47 of its 61 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

47 of 61 references.

References · 47

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  7. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  8. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  9. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  10. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  11. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  12. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
35 more
  1. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  2. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  3. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  4. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  5. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
  6. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  7. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  8. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 469
  9. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  10. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 392
  11. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 390
  12. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 382
  13. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  14. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
  15. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 321
  16. 2025GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Taskscited by 158
  17. 2023War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Warscited by 145
  18. 2024NATURAL PLAN: Benchmarking LLMs on Natural Language Planningcited by 128
  19. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 128
  20. 2025Reinforcement Learning with Rubric Anchorscited by 113
  21. 2025SWE-Lancer: Can Frontier LLMs Earn 1 Million from Real-World Freelance Software Engineering?cited by 105
  22. 2024Chain of Thoughtlessness? An Analysis of CoT in Planningcited by 103
  23. 2023Clarify When Necessary: Resolving Ambiguity Through Interaction with LMscited by 98
  24. 2025UserBench: An Interactive Gym Environment for User-Centric Agentscited by 76
  25. 2024TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generationcited by 68
  26. 2024Escalation Risks from Language Models in Military and Diplomatic Decision-Makingcited by 63
  27. 2025QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?cited by 63
  28. 2020ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ)cited by 58
  29. 2024Determinants of LLM-assisted Decision-Makingcited by 56
  30. 2025PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoningcited by 32
  31. 2025ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judgecited by 24
  32. 2024Human vs. Machine: Behavioral Differences Between Expert Humans and Language Models in Wargame Simulationscited by 21
  33. 2025Remote Labor Index: Measuring AI Automation of Remote Workcited by 20
  34. 2024COA-GPT: Generative Pre-trained Transformers for Accelerated Course of Action Development in Military Operationscited by 12
  35. 2025The AI Productivity Index (APEX)cited by 10

Built on by

Not cited yet.