Ko-WideSearch: A Korean Breadth-Search Benchmark for Exhaustive Set Enumeration by Web Agents
On the map: LLM agents and tool use → Web, search, and research agents
Abstract (arXiv)
Web-agent benchmarks overwhelmingly measure depth -- pinning one obscure answer behind a chain of constraints -- while breadth, exhaustively enumerating a closed set and filling each item's attributes, is barely evaluated, especially outside English. Breadth is also hard to build: certifying that a gold set is complete and every cell correct is far costlier than checking a single answer. I introduce textscKo-WideSearch, a Korean breadth-search benchmark built by an automated synthesize-and-verify pipeline. Each task names a set-parent entity -- a TV season, a dynasty, a league, an administrative region, an election -- and asks for its full membership plus a per-item attribute table, graded by Item-, Column-, and Row-F1. It spans 228 tables over 190 entities and sixteen categories across three difficulty tiers, set by two structural knobs I dial independently -- table width and a 2-D composite key -- so cross-product membership climbs from 0% to 100% across the tiers. A single normalization-aware comparator is shared between gold construction and grading, so stable date and count columns are not over-dropped on formatting alone. Across twenty web agents, the failure is consistent: agents recover the set but not the rows (e.g. Item-F1 92.8 against Row-F1 53.7), accuracy falls steadily as the knobs harden, and neither more search nor more spend closes the gap. Broken down by cell, the hard part is finding the right value, not formatting it: open-ended free-text cells fail most, while cells with a standard answer such as a date or a name usually come out right.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 42 of its 55 references that are in the map. First posted on arXiv 25 Jun 2026.
Builds on
42 of 55 references.
Closest ancestors · 6
Same direction, newest first
- 2026K-BrowseComp: A Web Browsing Agent Benchmark Grounded in Korean Contexts1 Jun
- 2025WideSearch: Benchmarking Agentic Broad Info-Seeking11 Aug
- 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?22 Jul
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agents25 Jul
- 2023Mind2Web: Towards a Generalist Agent for the Web9 Jun
- 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents4 Jul
Other references · 36
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
- 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
- 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
- 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
- 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
24 more
- 2023GAIA: a benchmark for General AI Assistantscited by 1,128
- 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
- 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
- 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
- 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
- 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 515
- 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 428
- 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
- 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
- 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
- 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 318
- 2020HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Datacited by 264
- 2023Proving Test Set Contamination in Black Box Language Modelscited by 230
- 2021FeTaQA: Free-form Table Question Answeringcited by 175
- 2021KLUE: Korean Language Understanding Evaluationcited by 114
- 2024KMMLU: Measuring Massive Multitask Language Understanding in Koreancited by 91
- 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 86
- 2021What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformerscited by 76
- 2024CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Koreancited by 66
- 2023HAE-RAE Bench: Evaluation of Korean Knowledge in Language Modelscited by 47
- 2022KOBEST: Korean Balanced Evaluation of Significant Taskscited by 37
- 2024EXAONE 3.0 7.8B Instruction Tuned Language Modelcited by 21
- 2025Kanana: Compute-efficient Bilingual Language Modelscited by 20
- 2026Soft Contamination Means Benchmarks Test Shallow Generalizationcited by 8
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023EFOk-CQA: Towards Knowledge Graph Complex Query Answering beyond Set Operation
- 2025Improving Table Understanding with LLMs and Entity-Oriented Search
- 2026BlitzRank: Principled Zero-shot Ranking Agents with Tournament Graphs
- 2026SemJoin: Semantic Join Optimization
- 2026ChartWalker: Benchmarking the Cross-Chart RAG Task with Hierarchical Knowledge Graphs
- 2024BEAVER: An Enterprise Benchmark for Text-to-SQL
- 2026Compositional Chain-of-Relations for Faithful Knowledge Graph Question Answering with Large Language Models
- 2026Atlas: Efficient Verifiable Semantic Search