# LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QA

arXiv: https://arxiv.org/abs/2409.02897 · HTML: https://mlsanity.com/paper/2409.02897/

## Key facts

- Cited by 81 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 4 Sep 2024.

## Builds on (most cited first, 22 of 22)

- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [Enabling Large Language Models to Generate Text with Citations](https://mlsanity.com/paper/2305.14627/) (2023)
- [Efficient Attentions for Long Document Summarization](https://mlsanity.com/paper/2104.02112/) (2021)
- [Evaluating Verifiability in Generative Search Engines](https://mlsanity.com/paper/2304.09848/) (2023)
- [Effective Long-Context Scaling of Foundation Models](https://mlsanity.com/paper/2309.16039/) (2023)
- [Teaching language models to support answers with verified quotes](https://mlsanity.com/paper/2203.11147/) (2022)
- [RARR: Researching and Revising What Language Models Say, Using Language Models](https://mlsanity.com/paper/2210.08726/) (2022)
- [DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applications](https://mlsanity.com/paper/1711.05073/) (2017)
- [LongAlign: A Recipe for Long Context Alignment of Large Language Models](https://mlsanity.com/paper/2401.18058/) (2024)
- [Attribute First, then Generate: Locally-attributable Grounded Text Generation](https://mlsanity.com/paper/2403.17104/) (2024)
- [Learning to Plan and Generate Text with Citations](https://mlsanity.com/paper/2404.03381/) (2024)
- [Learning Fine-Grained Grounded Citations for Attributed Large Language Models](https://mlsanity.com/paper/2408.04568/) (2024)
- [Towards Faithful and Robust LLM Specialists for Evidence-Based Question-Answering](https://mlsanity.com/paper/2402.08277/) (2024)
- [Attribute or Abstain: Large Language Models as Long Document Assistants](https://mlsanity.com/paper/2407.07799/) (2024)

## Built on by (most cited first, 30 of 81)

- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks](https://mlsanity.com/paper/2412.15204/) (2024)
- [Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracy](https://mlsanity.com/paper/2502.05177/) (2025)
- [A Survey on Knowledge-Oriented Retrieval-Augmented Generation](https://mlsanity.com/paper/2503.10677/) (2025)
- [LongReward: Improving Long-context Large Language Models with AI Feedback](https://mlsanity.com/paper/2410.21252/) (2024)
- [Search Arena: Analyzing Search-Augmented LLMs](https://mlsanity.com/paper/2506.05334/) (2025)
- [DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis](https://mlsanity.com/paper/2508.20033/) (2025)
- [VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model](https://mlsanity.com/paper/2505.03739/) (2025)
- [SelfCite: Self-Supervised Alignment for Context Attribution in Large Language Models](https://mlsanity.com/paper/2502.09604/) (2025)
- [CiteEval: Principle-Driven Citation Evaluation for Source Attribution](https://mlsanity.com/paper/2506.01829/) (2025)
- [Docopilot: Improving Multimodal Models for Document-Level Understanding](https://mlsanity.com/paper/2507.14675/) (2025)
- [LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Data](https://mlsanity.com/paper/2502.12583/) (2025)
- [When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training](https://mlsanity.com/paper/2411.13476/) (2024)
- [LongEval: A Comprehensive Analysis of Long-Text Generation Through a Plan-based Paradigm](https://mlsanity.com/paper/2502.19103/) (2025)
- [SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimization](https://mlsanity.com/paper/2505.11166/) (2025)
- [Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?](https://mlsanity.com/paper/2605.19196/) (2026)
- [LOGO -- Long cOntext aliGnment via efficient preference Optimization](https://mlsanity.com/paper/2410.18533/) (2024)
- [HoT: Highlighted Chain of Thought for Referencing Supporting Facts from Inputs](https://mlsanity.com/paper/2503.02003/) (2025)
- [CL-bench Life: Can Language Models Learn from Real-Life Context?](https://mlsanity.com/paper/2604.27043/) (2026)
- [Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap](https://mlsanity.com/paper/2608.05179/) (2026)
- [MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Models](https://mlsanity.com/paper/2505.19959/) (2025)
- [VeriCite: Towards Reliable Citations in Retrieval-Augmented Generation via Rigorous Verification](https://mlsanity.com/paper/2510.11394/) (2025)
- [LooGLE v2: Are LLMs Ready for Real World Long Dependency Challenges?](https://mlsanity.com/paper/2510.22548/) (2025)
- [Expect the Unexpected: FailSafe Long Context QA for Finance](https://mlsanity.com/paper/2502.06329/) (2025)
- [Cite Before You Speak: Enhancing Context-Response Grounding in E-commerce Conversational LLM-Agents](https://mlsanity.com/paper/2503.04830/) (2025)
- [Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Find](https://mlsanity.com/paper/2505.18148/) (2025)
- [Attribution, Citation, and Quotation: A Survey of Evidence-based Text Generation with Large Language Models](https://mlsanity.com/paper/2508.15396/) (2025)
- [Linguistic Nepotism: Trading-off Quality for Language Preference in Multilingual RAG](https://mlsanity.com/paper/2509.13930/) (2025)
- [Generation-Time vs. Post-hoc Citation: A Holistic Evaluation of LLM Attribution](https://mlsanity.com/paper/2509.21557/) (2025)
- [Detecting and Correcting Reference Hallucinations in Commercial LLMs and Deep Research Agents](https://mlsanity.com/paper/2604.03173/) (2026)

## Abstract (arXiv)

Though current long-context large language models (LLMs) have demonstrated impressive capacities in answering user questions based on extensive text, the lack of citations in their responses makes user verification difficult, leading to concerns about their trustworthiness due to their potential hallucinations. In this work, we aim to enable long-context LLMs to generate responses with fine-grained sentence-level citations, improving their faithfulness and verifiability. We first introduce LongBench-Cite, an automated benchmark for assessing current LLMs' performance in Long-Context Question Answering with Citations (LQAC), revealing considerable room for improvement. To this end, we propose CoF (Coarse to Fine), a novel pipeline that utilizes off-the-shelf LLMs to automatically generate long-context QA instances with precise sentence-level citations, and leverage this pipeline to construct LongCite-45k, a large-scale SFT dataset for LQAC. Finally, we train LongCite-8B and LongCite-9B using the LongCite-45k dataset, successfully enabling their generation of accurate responses and fine-grained sentence-level citations in a single output. The evaluation results on LongBench-Cite show that our trained models achieve state-of-the-art citation quality, surpassing advanced proprietary models including GPT-4o.