# How Much Structure Do LLMs Need? Evaluating LLMs for Bibliometric Cluster Description

arXiv: https://arxiv.org/abs/2605.24351 · HTML: https://mlsanity.com/paper/2605.24351/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 41 references that are in the map.
- First posted on arXiv 23 May 2026.

## Builds on (most cited first, 11 of 11)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Large Language Models: A Survey](https://mlsanity.com/paper/2402.06196/) (2024)
- [Multi-XScience: A Large-scale Dataset for Extreme Multi-document Summarization of Scientific Articles](https://mlsanity.com/paper/2010.14235/) (2020)
- [CHIME: LLM-Assisted Hierarchical Organization of Scientific Studies for Literature Review Support](https://mlsanity.com/paper/2407.16148/) (2024)
- [SciReviewGen: A Large-scale Dataset for Automatic Literature Review Generation](https://mlsanity.com/paper/2305.15186/) (2023)
- [Large Language Models for Automated Literature Review: An Evaluation of Reference Generation, Abstract Writing, and Review Composition](https://mlsanity.com/paper/2412.13612/) (2024)
- [Automatic Labelling of Topics with Neural Embeddings](https://mlsanity.com/paper/1612.05340/) (2016)
- [Neural Topic Modeling with Large Language Models in the Loop](https://mlsanity.com/paper/2411.08534/) (2024)
- [Reassessing Large Language Model Boolean Query Generation for Systematic Reviews](https://mlsanity.com/paper/2505.07155/) (2025)
- [ExtracTable: Human-in-the-Loop Transformation of Scientific Corpora into Structured Knowledge](https://mlsanity.com/paper/2506.03221/) (2025)

## Abstract (arXiv)

Large language models (LLMs) can support scientific literature synthesis, but remain prone to hallucinated references, uneven coverage, and weakly grounded thematic organization. We evaluate whether bibliometric structure improves LLM-assisted synthesis by comparing six pipelines for generating cluster descriptions under different levels of evidence and structure. Using 100 published bibliometric analyses, we reconstruct Scopus corpora, extract human-written cluster descriptions, and assess outputs by human alignment, semantic coverage, clustering quality, graph quality, and reference grounding. Results show that LLMs produce descriptions semantically close to human-written ones, but are unreliable when asked to infer bibliometric structure from scratch. Performance improves when bibliometric algorithms define the clusters and the LLM interprets them. Overall, LLM-assisted bibliometric synthesis is most promising as a hybrid workflow in which algorithms provide auditable structure and LLMs generate readable descriptions.