HIERA: Hierarchical Multi-Agent Relevance Assessment for Content Discovery Systems
On the map: LLM agents and tool use → Web, search, and research agents
Abstract (arXiv)
Content discovery systems depend on relevance judgment for search quality evaluation, but human annotation faces inter-annotator disagreement and scaling costs. While Large Language Models show promise as automated assessors, current approaches rely on flat aggregation strategies: single-step prompting, voting ensembles, or uncoordinated multi-agent pipelines that aggregate independent outputs without integration. We propose HIERA, a hierarchical multi-agent relevance assessment framework with four specialized agents: a Relevance Judge, Query Analyzer, Item Analyzer, and Relation Analyzer. The Judge determines when specialist analysis is needed; the Relation Analyzer then coordinates query and item analyses with external knowledge to establish relevance relationships for final judgment. Ablation studies show that the same agents and external knowledge without hierarchical coordination degrade performance, confirming that the coordination structure itself accounts for the improvement. Evaluation across five datasets (EVS, MSRD, ESCI, WANDS, Home Depot) shows improvements over 11 baselines: 10.2% on Home Depot, 4.8% on ESCI, and up to 38% on EVS (p < 0.05). Hierarchical coordination yields 12.7% improvement over uncoordinated collaboration using identical agents.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 10 of its 29 references that are in the map. First posted on arXiv 1 Aug 2026.
Builds on
10 of 29 references.
References · 10
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 598
- 2023Large language models can accurately predict searcher preferencescited by 131
- 2023Perspectives on Large Language Models for Relevance Judgmentcited by 85
- 2024QueryAgent: A Reliable and Efficient Reasoning Framework with Environmental Feedback-based Self-Correctioncited by 27
- 2024A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Lookcited by 18
- 2024Retrieve, Annotate, Evaluate, Repeat: Leveraging Multimodal LLMs for Large-Scale Product Retrieval Evaluationcited by 12
- 2025Multi-stage Large Language Model Pipelines Can Outperform GPT-4o in Relevance Assessmentcited by 7
- 2025Automated Query-Product Relevance Labeling using Large Language Models for E-commerce Searchcited by 5
- 2024Best in Tau@LLMJudge: Criteria-Based Relevance Evaluation with Llama3cited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Prism-Reranker: Beyond Relevance Scoring -- Jointly Producing Contributions and Evidence for Agentic Retrieval
- 2026Completing Missing Annotation: Multi-Agent Debate for Accurate and Scalable Relevant Assessment for IR Benchmarks
- 2026A New Role for Relevance: Guiding Corpus Interaction in Agentic Search
- 2026HMARS: A Hierarchical Multi-Agent Memory System for Long-Context Reasoning
- 2026Rethinking Reasoning-Intensive Retrieval: Evaluating and Advancing Retrievers in Agentic Search Systems
- 2025HM-RAG: Hierarchical Multi-Agent Multimodal Retrieval Augmented Generation
- 2026Learning to Retrieve from Agent Trajectories
- 2025A Multi-Agent Perspective on Modern Information Retrieval