mlsanityData to 14 Oct 2026

SealQA: Raising the Bar for Reasoning in Search-Augmented Language Models

arXiv 2506.01062 ↗First posted 1 Jun 2025cs.CL cs.AI cs.LGcited by 75

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

We introduce SealQA, a new challenge benchmark for evaluating SEarch-Augmented Language models on fact-seeking questions where web search yields conflicting, noisy, or unhelpful results. SealQA comes in three flavors: (1) Seal-0 (main) and (2) Seal-Hard, which assess factual accuracy and reasoning capabilities, with Seal-0 focusing on the most challenging questions where chat models (e.g., GPT-4.1) typically achieve near-zero accuracy; and (3) LongSeal, which extends SealQA to test long-context, multi-document reasoning in "needle-in-a-haystack" settings. Our evaluation reveals critical limitations in current models: Even frontier LLMs perform poorly across all SealQA flavors. On Seal-0, frontier agentic models equipped with tools like o3 and o4-mini achieve only 17.1% and 6.3% accuracy, respectively, at their best reasoning efforts. We find that advanced reasoning models such as DeepSeek-R1-671B and o3-mini are highly vulnerable to noisy search results. Notably, increasing test-time compute does not yield reliable gains across o3-mini, o4-mini, and o3, with performance often plateauing or even declining early. Additionally, while recent models are less affected by the "lost-in-the-middle" issue, they still fail to reliably identify relevant documents in LongSeal when faced with numerous distractors. To facilitate future work, we release SealQA at huggingface.co/datasets/vtllms/sealqa.

Key facts. Cited by 75 papers in the map (data through 14 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 41 of its 63 references that are in the map. First posted on arXiv 1 Jun 2025.

Builds on

41 of 63 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning12 Mar
  2. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation19 Sep

Other references · 39

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  9. 2025s1: Simple test-time scalingcited by 1,341
  10. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  11. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  12. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
27 more
  1. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
  2. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
  3. 2024Measuring short-form factuality in large language modelscited by 373
  4. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
  5. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 318
  6. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
  7. 2024Knowledge Conflicts for LLMs: A Surveycited by 283
  8. 2022RealTime QA: What's the Answer Right Now?cited by 244
  9. 2021A Dataset for Answering Time-Sensitive Questionscited by 154
  10. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  11. 2022Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidencecited by 144
  12. 2022StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Modelscited by 126
  13. 2024Tug-of-War Between Knowledge: Exploring and Resolving Knowledge Conflicts in Retrieval-Augmented Language Modelscited by 93
  14. 2024FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"cited by 86
  15. 2024What Evidence Do Language Models Find Convincing?cited by 75
  16. 2024ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLMcited by 73
  17. 2025Retrieval-Augmented Generation with Conflicting Evidencecited by 63
  18. 2023Resolving Knowledge Conflicts in Large Language Modelscited by 50
  19. 2023BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Informationcited by 44
  20. 2024WikiContradict: A Benchmark for Evaluating LLMs on Real-World Knowledge Conflicts from Wikipediacited by 44
  21. 2021Attacking Open-domain Question Answering by Injecting Misinformationcited by 41
  22. 2024AmbigDocs: Reasoning across Documents on Different Entities under the Same Namecited by 21
  23. 2024ECon: On the Detection and Resolution of Evidence Conflictscited by 18
  24. 2024Open Domain Question Answering with Conflicting Contextscited by 18
  25. 2024Adaptive Question Answering: Enhancing Language Model Proficiency for Addressing Knowledge Conflicts with Source Citationscited by 15
  26. 2025FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documentscited by 15
  27. 2024DebateQA: Evaluating Question Answering on Debatable Knowledgecited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 60.

75 papers.

Sort:

By area:

  1. 2026Kimi K2.5: Visual Agentic Intelligencecited by 501
  2. 2026EvoSkill: Automated Skill Discovery for Multi-Agent Systemscited by 107
  3. 2025MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scalingcited by 81
  4. 2025A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challengescited by 62
  5. 2025DeepDive: Advancing Deep Search Agents with Knowledge Graphs and Multi-Turn RLcited by 55
  6. 2025Scaling Agents via Continual Pre-trainingcited by 40
  7. 2026MiroThinker-1.7 & H1: Towards Heavy-Duty Research Agents via Verificationcited by 32
  8. 2025IterResearch: Rethinking Long-Horizon Agents with Interaction Scalingcited by 23
  9. 2026ExpSeek: Self-Triggered Experience Seeking for Web Agentscited by 23
  10. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applicationscited by 22
  11. 2026From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skillscited by 22
  12. 2026Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agentcited by 22
38 more
  1. 2026MetaSkill-Evolve: Recursive Self-Improvement of LLM Agents via Two-Timescale Meta-Skill Evolutioncited by 22
  2. 2025WebLeaper: Empowering Efficiency and Efficacy in WebAgent via Enabling Info-Rich Seekingcited by 18
  3. 2026CausalFlow: Causal Attribution and Counterfactual Repair for LLM Agent Failurescited by 18
  4. 2026APEX-MEM: Agentic Semi-Structured Memory with Temporal Reasoning for Long-Term Conversational AIcited by 17
  5. 2026Toward Efficient Agents: Memory, Tool learning, and Planningcited by 16
  6. 2025DeepSieve: Information Sieving via LLM-as-a-Knowledge-Routercited by 13
  7. 2026WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolutioncited by 12
  8. 2026Are Tools All We Need? Unveiling the Tool-Use Tax in LLM Agentscited by 11
  9. 2026LiteResearcher: A Scalable Agentic RL Training Framework for Deep Research Agentcited by 10
  10. 2026Harness-1: Reinforcement Learning for Search Agents with State-Externalizing Harnessescited by 10
  11. 2025LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Servicescited by 8
  12. 2025EcomBench: Towards Holistic Evaluation of Foundation Agents in E-commercecited by 8
  13. 2026Lost in the Noise: How Reasoning Models Fail with Contextual Distractorscited by 8
  14. 2026DualSpec: Accelerating Deep Research Agents via Dual-Process Action Speculationcited by 8
  15. 2026Evidence-Augmented Policy Optimization with Reward Co-Evolution for Long-Context Reasoningcited by 7
  16. 2026Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMscited by 7
  17. 2026FORT-Searcher: Synthesizing Shortcut-Resistant Search Tasks for Training Deep Search Agentscited by 7
  18. 2026GISA: A Benchmark for General Information-Seeking Assistantcited by 6
  19. 2026Unlocking Reasoning Capability on Machine Translation in Large Language Modelscited by 6
  20. 2026Argus: Evidence Assembly for Scalable Deep Research Agentscited by 6
  21. 2026SkillSmith: Co-Evolving Skills and Tools for Self-Improving Agent Systemscited by 6
  22. 2025Fathom-DeepResearch: Unlocking Long Horizon Information Retrieval and Synthesis for SLMscited by 5
  23. 2026SRR-Judge: Step-Level Rating and Refinement for Enhancing Search-Integrated Reasoning in Search Agentscited by 5
  24. 2026Search More, Think Less: Rethinking Long-Horizon Agentic Search for Efficiency and Generalizationcited by 5
  25. 2026ICA: Information-Aware Credit Assignment for Visually Grounded Long-Horizon Information-Seeking Agentscited by 4
  26. 2026Agent Skills Matter: Inferring Proprietary Skills from Execution Trajectoriescited by 4
  27. 2025Dr. Bench: A Multidimensional Evaluation for Deep Research Agents, from Answers to Reportscited by 3
  28. 2026ROMA: Recursive Open Meta-Agent Framework for Long-Horizon Multi-Agent Systemscited by 3
  29. 2026MERRIN: A Benchmark for Multimodal Evidence Retrieval and Reasoning in Noisy Web Environmentscited by 3
  30. 2026EvoBrowseComp: Benchmarking Search Agents on Evolving Knowledgecited by 3
  31. 2026SearchMaster: Grounded and Regulated Self-Play for Search Agentscited by 3
  32. 2025BRIEF-Pro: Universal Context Compression with Short-to-Long Synthesis for Fast and Accurate Multi-Hop Reasoningcited by 2
  33. 2026Search, Do not Guess: Teaching Small Language Models to Be Effective Search Agentscited by 2
  34. 2026Beyond Monolingual Deep Research: Evaluating Agents and Retrievers with Cross-Lingual BrowseComp-Pluscited by 2
  35. 2026Risk-Constrained Freshness-Aware Semantic Caching for Open-Web Retrieval-Augmented LLMscited by 2
  36. 2026SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agentscited by 2
  37. 2026When History Is Multimodal: Rethinking Context Management for Long-Horizon Agentscited by 2
  38. 2025If I Could Turn Back Time: Temporal Reframing as a Historical Reasoning Task for LLMscited by 1

50 most cited of 75 All 75 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Breaking the Evaluation Paradox: Evaluating High-Entropy Search with Computationally Irreducible Constraints
  2. 2026Over-Searching in Search-Augmented Large Language Models
  3. 2026LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake
  4. 2026DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agents
  5. 2025Improving Context Fidelity via Native Retrieval-Augmented Reasoning
  6. 2026RAC: Retrieval-Augmented Clarification for Faithful Conversational Search
  7. 2024Long-form factuality in large language models
  8. 2025BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language Models in Chinese