mlsanityData to 7 Oct 2026

Relay-Bench: Evaluating LLMs on Multi-Domain Reasoning Chains

arXiv 2607.18438 ↗First posted 20 Jul 2026cs.CL cs.AI cs.LGcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Introducing Relay-Bench, an unsaturated, holistic, text-only benchmark that measures LLMs' ability to complete an assortment of tasks from distinct domains in a single prompt. The leading model, GPT-5.5 (xHigh), scores 43.3%. The test set entirely consists of composite problems: groups of single-domain subproblems that are strung together into challenges that require reasoning across multiple domains in combination. Many of these problems then have layers of complexity added through prompt encoding and deliberate context bloat. Domains tested include visual reasoning, coding, math, information extraction (with a focus on web search), problem-solving, general knowledge, and data analysis. No restrictions are imposed outside of the model harness, and models are explicitly encouraged to leverage code-execution, web searches, and all available tools. All problems are composed of two to thirteen subproblems and do not require multi-modal input or output.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 26 of its 39 references that are in the map. First posted on arXiv 20 Jul 2026.

Builds on

26 of 39 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Are Your LLMs Capable of Stable Reasoning?17 Dec

Other references · 25

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  5. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  6. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  7. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  8. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  9. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  10. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  11. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  12. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
13 more
  1. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  2. 2025Humanity's Last Examcited by 467
  3. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIcited by 227
  4. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  5. 2025Why Language Models Hallucinatecited by 205
  6. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 136
  7. 2025Context Length Alone Hurts LLM Performance Despite Perfect Retrievalcited by 118
  8. 2024SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulationcited by 100
  9. 2025Measuring Agents in Productioncited by 35
  10. 2026Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 32
  11. 2026APEX-Agentscited by 29
  12. 2025AgentFrontier: Expanding the Capability Frontier of LLM Agents with ZPD-Guided Data Synthesiscited by 12
  13. 2025AA-Omniscience: Evaluating Cross-Domain Knowledge Reliability in Large Language Modelscited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025RUST-BENCH: Benchmarking LLM Reasoning on Unstructured Text within Structured Tables
  2. 2025ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoning
  3. 2023Are LLMs the Master of All Trades? : Exploring Domain-Agnostic Reasoning Skills of LLMs
  4. 2026IsoSci: A Benchmark of Isomorphic Cross-Domain Science Problems for Evaluating Reasoning versus Knowledge Retrieval in LLMs
  5. 2025Mapping the Minds of LLMs: A Graph-Based Analysis of Reasoning LLM
  6. 2025Short-Path Prompting in LLMs: Analyzing Reasoning Instability and Solutions for Robust Performance
  7. 2025LLMSR@XLLM25: An Empirical Study of LLM for Structural Reasoning
  8. 2026General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks