# HSCodeComp: A Realistic and Expert-level Benchmark for Deep Search Agents in Hierarchical Rule Application

arXiv: https://arxiv.org/abs/2510.19631 · HTML: https://mlsanity.com/paper/2510.19631/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 34 of its 51 references that are in the map.
- First posted on arXiv 22 Oct 2025.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 30 of 34)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025)
- [Inference-Time Scaling for Generalist Reward Modeling](https://mlsanity.com/paper/2504.02495/) (2025)
- [WebSailor: Navigating Super-human Reasoning for Web Agent](https://mlsanity.com/paper/2507.02592/) (2025)
- [ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems](https://mlsanity.com/paper/2505.11831/) (2025)
- [WebWalker: Benchmarking LLMs in Web Traversal](https://mlsanity.com/paper/2501.07572/) (2025)
- [Llama-Nemotron: Efficient Reasoning Models](https://mlsanity.com/paper/2505.00949/) (2025)
- [BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language Models in Chinese](https://mlsanity.com/paper/2504.19314/) (2025)
- [OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automation](https://mlsanity.com/paper/2505.23885/) (2025)
- [WebShaper: Agentically Data Synthesizing via Information-Seeking Formalization](https://mlsanity.com/paper/2507.15061/) (2025)
- [xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluations](https://mlsanity.com/paper/2506.13651/) (2025)
- [Reward Reasoning Model](https://mlsanity.com/paper/2505.14674/) (2025)
- [DeepShop: A Benchmark for Deep Research Shopping Agents](https://mlsanity.com/paper/2506.02839/) (2025)
- [Cognitive Kernel-Pro: A Framework for Deep Research Agents and Agent Foundation Models Training](https://mlsanity.com/paper/2508.00414/) (2025)
- [A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk](https://mlsanity.com/paper/1712.05796/) (2017)
- [AgentOrchestra: Orchestrating Multi-Agent Intelligence with the Tool-Environment-Agent(TEA) Protocol](https://mlsanity.com/paper/2506.12508/) (2025)
- [MedBrowseComp: Benchmarking Medical Deep Research and Computer Use](https://mlsanity.com/paper/2505.14963/) (2025)
- [FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning](https://mlsanity.com/paper/2509.13160/) (2025)
- [MedResearcher-R1: Expert-Level Medical Deep Researcher via A Knowledge-Informed Trajectory Synthesis Framework](https://mlsanity.com/paper/2508.14880/) (2025)
- [WebMall -- A Multi-Shop Benchmark for Evaluating Web Agents](https://mlsanity.com/paper/2508.13024/) (2025)
- [AWorld: Orchestrating the Training Recipe for Agentic AI](https://mlsanity.com/paper/2508.20404/) (2025)
- [Explainable Rule Application via Structured Prompting: A Neural-Symbolic Approach](https://mlsanity.com/paper/2506.16335/) (2025)
- [DAgent: A Relational Database-Driven Data Analysis Report Generation Agent](https://mlsanity.com/paper/2503.13269/) (2025)
- [WebGames: Challenging General-Purpose Web-Browsing AI Agents](https://mlsanity.com/paper/2502.18356/) (2025)

## Built on by (most cited first, 5 of 5)

- [Marco DeepResearch: Unlocking Efficient Deep Research Agents via Verification-Centric Design](https://mlsanity.com/paper/2603.28376/) (2026)
- [CHIME: Credit-Aware Hierarchical Memory Evolution for Long-Horizon Agentic Planning](https://mlsanity.com/paper/2609.02074/) (2026)
- [A Deterministic Agentic Workflow for HS Tariff Classification: Multi-Dimensional Rule Reasoning with Interpretable Decisions](https://mlsanity.com/paper/2605.14857/) (2026)
- [Knowing When to Ask: Self-Gated Clarification for Hierarchical Language Agents](https://mlsanity.com/paper/2606.11349/) (2026)
- [Consensus-based Agentic Large Language Model Framework for Harmonized Tariff Schedule Code Classification](https://mlsanity.com/paper/2606.16987/) (2026)

## Abstract (arXiv)

Effective deep search agents must not only access open-domain and domain-specific knowledge but also apply complex rules-such as legal clauses, medical manuals and tariff rules. These rules often feature vague boundaries and implicit logic relationships, making precise application challenging for agents. However, this critical capability is largely overlooked by current agent benchmarks. To fill this gap, we introduce HSCodeComp, the first realistic, expert-level e-commerce benchmark designed to evaluate deep search agents in hierarchical rule application. In this task, the deep reasoning process of agents is guided by these rules to predict 10-digit Harmonized System Code (HSCode) of products with noisy but realistic descriptions. These codes, established by the World Customs Organization, are vital for global supply chain efficiency. Built from real-world data collected from large-scale e-commerce platforms, our proposed HSCodeComp comprises 632 product entries spanning diverse product categories, with these HSCodes annotated by several human experts. Extensive experimental results on several state-of-the-art LLMs, open-source, and closed-source agents reveal a huge performance gap: best agent achieves only 46.8% 10-digit accuracy, far below human experts at 95.0%. Besides, detailed analysis demonstrates the challenges of hierarchical rule application, and test-time scaling fails to improve performance further.