# CHIME: LLM-Assisted Hierarchical Organization of Scientific Studies for Literature Review Support

arXiv: https://arxiv.org/abs/2407.16148 · HTML: https://mlsanity.com/paper/2407.16148/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 41 references that are in the map.
- First posted on arXiv 23 Jul 2024.

## Builds on (most cited first, 17 of 17)

- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing](https://mlsanity.com/paper/1902.07669/) (2019)
- [SciRepEval: A Multi-Format Benchmark for Scientific Document Representations](https://mlsanity.com/paper/2211.13308/) (2022)
- [MARG: Multi-Agent Review Generation for Scientific Papers](https://mlsanity.com/paper/2401.04259/) (2024)
- [MS2: Multi-Document Summarization of Medical Studies](https://mlsanity.com/paper/2104.06486/) (2021)
- [Large Language Models Enable Few-Shot Clustering](https://mlsanity.com/paper/2307.00524/) (2023)
- [ClusterLLM: Large Language Models as a Guide for Text Clustering](https://mlsanity.com/paper/2305.14871/) (2023)
- [Goal-Driven Explainable Clustering via Language Descriptions](https://mlsanity.com/paper/2305.13749/) (2023)
- [Generating (Factual?) Narrative Summaries of RCTs: Experiments with Neural Multi-Document Summarization](https://mlsanity.com/paper/2008.11293/) (2020)
- [Explaining Relationships Between Scientific Documents](https://mlsanity.com/paper/2002.00317/) (2020)
- [Appraising the Potential Uses and Harms of LLMs for Medical Systematic Reviews](https://mlsanity.com/paper/2305.11828/) (2023)
- [Generating a Structured Summary of Numerous Academic Papers: Dataset and Method](https://mlsanity.com/paper/2302.04580/) (2023)
- [Multi-Document Scientific Summarization from a Knowledge Graph-Centric View](https://mlsanity.com/paper/2209.04319/) (2022)
- [Hierarchical Catalogue Generation for Literature Review: A Benchmark](https://mlsanity.com/paper/2304.03512/) (2023)
- [Towards Generating Citation Sentences for Multiple References with Intent Control](https://mlsanity.com/paper/2112.01332/) (2021)

## Built on by (most cited first, 29 of 29)

- [Many Heads Are Better Than One: Improved Scientific Idea Generation by A LLM-Based Multi-Agent System](https://mlsanity.com/paper/2410.09403/) (2024)
- [LLMs as Research Tools: A Large Scale Survey of Researchers' Usage and Perceptions](https://mlsanity.com/paper/2411.05025/) (2024)
- [Large Language Models for Automated Literature Review: An Evaluation of Reference Generation, Abstract Writing, and Review Composition](https://mlsanity.com/paper/2412.13612/) (2024)
- [How Far Are AI Scientists from Changing the World?](https://mlsanity.com/paper/2507.23276/) (2025)
- [AI for Auto-Research: Roadmap & User Guide](https://mlsanity.com/paper/2605.18661/) (2026)
- [Context-Aware Hierarchical Taxonomy Generation for Scientific Papers via LLM-Guided Multi-Aspect Clustering](https://mlsanity.com/paper/2509.19125/) (2025)
- [Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluator](https://mlsanity.com/paper/2507.11810/) (2025)
- [From Hypothesis to Publication: A Comprehensive Survey of AI-Driven Research Support Systems](https://mlsanity.com/paper/2503.01424/) (2025)
- [Deep Ideation: Designing LLM Agents to Generate Novel Research Ideas on Scientific Concept Network](https://mlsanity.com/paper/2511.02238/) (2025)
- [Whose Name Comes Up? Auditing LLM-Based Scholar Recommendations](https://mlsanity.com/paper/2506.00074/) (2025)
- [Intent-Aware Schema Generation And Refinement For Literature Review Tables](https://mlsanity.com/paper/2507.19521/) (2025)
- [ScienceMeter: Tracking Scientific Knowledge Updates in Language Models](https://mlsanity.com/paper/2505.24302/) (2025)
- [Repository Intelligence Graph: Deterministic Architectural Map for LLM Code Assistants](https://mlsanity.com/paper/2601.10112/) (2026)
- [Attribution Gradients: Incrementally Unfolding Citations for Critical Examination of Attributed AI Answers](https://mlsanity.com/paper/2510.00361/) (2025)
- [Agent Benchmarks Fail Public Sector Requirements](https://mlsanity.com/paper/2601.20617/) (2026)
- [ResearchPulse: Building Method-Experiment Chains through Multi-Document Scientific Inference](https://mlsanity.com/paper/2509.03565/) (2025)
- [EvoTaxo: Building and Evolving Taxonomy from Social Media Streams](https://mlsanity.com/paper/2603.19711/) (2026)
- [AgentFold: Closed-Loop Agentic Search for Protein Folding Model Design](https://mlsanity.com/paper/2608.26747/) (2026)
- [Accelerating Antibiotic Discovery with Large Language Models and Knowledge Graphs](https://mlsanity.com/paper/2503.16655/) (2025)
- [LLM as Attention-Informed NTM and Topic Modeling as long-input Generation: Interpretability and long-Context Capability](https://mlsanity.com/paper/2510.03174/) (2025)
- [Leveraging Hierarchical Organization for Medical Multi-document Summarization](https://mlsanity.com/paper/2510.23104/) (2025)
- [MVSS: A Unified Framework for Multi-View Structured Survey Generation](https://mlsanity.com/paper/2601.09504/) (2026)
- [PROMPT2BOX:Improving LLM Weakness Discovery and Specificity Estimation by Uncovering Entailment Structure among Prompts](https://mlsanity.com/paper/2603.21438/) (2026)
- [CoAuthorAI: A Human in the Loop System For Scientific Book Writing](https://mlsanity.com/paper/2604.19772/) (2026)
- [SC-Taxo: Hierarchical Taxonomy Generation under Semantic Consistency Constraints using Large Language Models](https://mlsanity.com/paper/2605.00620/) (2026)
- [LLM Jaggedness Unlocks Scientific Creativity](https://mlsanity.com/paper/2605.10574/) (2026)
- [How Much Structure Do LLMs Need? Evaluating LLMs for Bibliometric Cluster Description](https://mlsanity.com/paper/2605.24351/) (2026)
- [Security and Privacy Taxonomy Generation from Mobile App Reviews](https://mlsanity.com/paper/2608.09049/) (2026)
- [LLMs for Academic Workflows: An Evaluation of Literature Reviews Generated with Short and Long Context Windows of LLMs](https://mlsanity.com/paper/2608.26145/) (2026)

## Abstract (arXiv)

Literature review requires researchers to synthesize a large amount of information and is increasingly challenging as the scientific literature expands. In this work, we investigate the potential of LLMs for producing hierarchical organizations of scientific studies to assist researchers with literature review. We define hierarchical organizations as tree structures where nodes refer to topical categories and every node is linked to the studies assigned to that category. Our naive LLM-based pipeline for hierarchy generation from a set of studies produces promising yet imperfect hierarchies, motivating us to collect CHIME, an expert-curated dataset for this task focused on biomedicine. Given the challenging and time-consuming nature of building hierarchies from scratch, we use a human-in-the-loop process in which experts correct errors (both links between categories and study assignment) in LLM-generated hierarchies. CHIME contains 2,174 LLM-generated hierarchies covering 472 topics, and expert-corrected hierarchies for a subset of 100 topics. Expert corrections allow us to quantify LLM performance, and we find that while they are quite good at generating and organizing categories, their assignment of studies to categories could be improved. We attempt to train a corrector model with human feedback which improves study assignment by 12.6 F1 points. We release our dataset and models to encourage research on developing better assistive tools for literature review.