# Towards AI-Assisted Research Writing: Benchmarking LLMs for AI/ML Introduction Generation

arXiv: https://arxiv.org/abs/2508.14273 · HTML: https://mlsanity.com/paper/2508.14273/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 43 references that are in the map.
- First posted on arXiv 19 Aug 2025.

## Builds on (most cited first, 19 of 19)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021)
- [TLDR: Extreme Summarization of Scientific Documents](https://mlsanity.com/paper/2004.15011/) (2020)
- [Multi-XScience: A Large-scale Dataset for Extreme Multi-document Summarization of Scientific Articles](https://mlsanity.com/paper/2010.14235/) (2020)
- [When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data](https://mlsanity.com/paper/2102.02201/) (2021)
- [PaperRobot: Incremental Draft Generation of Scientific Ideas](https://mlsanity.com/paper/1905.07870/) (2019)
- [Can LLMs Generate Architectural Design Decisions? -An Exploratory Empirical study](https://mlsanity.com/paper/2403.01709/) (2024)
- [Generating a Structured Summary of Numerous Academic Papers: Dataset and Method](https://mlsanity.com/paper/2302.04580/) (2023)
- [LDKP: A Dataset for Identifying Keyphrases from Long Scientific Documents](https://mlsanity.com/paper/2203.15349/) (2022)
- [Overview and Results: CL-SciSumm Shared Task 2019](https://mlsanity.com/paper/1907.09854/) (2019)
- [LCFO: Long Context and Long Form Output Dataset and Benchmarking](https://mlsanity.com/paper/2412.08268/) (2024)

## Built on by (most cited first, 6 of 6)

- [DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis](https://mlsanity.com/paper/2508.20033/) (2025)
- [AI for Auto-Research: Roadmap & User Guide](https://mlsanity.com/paper/2605.18661/) (2026)
- [Eliminating Agentic Workflow for Introduction Generation with Parametric Stage Tokens](https://mlsanity.com/paper/2601.09728/) (2025)
- [LECTOR: Joint Optimization of Scientific Reasoning Graphs and Introduction Generation](https://mlsanity.com/paper/2605.25964/) (2026)
- [Internalizing Academic Writing Workflows for Introduction Generation via Struct-Aware Policy Learning](https://mlsanity.com/paper/2608.03138/) (2026)
- [How Is Automated Research Evaluated? A Survey of Benchmarks and Evaluation Practices](https://mlsanity.com/paper/2610.11877/) (2026)

## Abstract (arXiv)

As researchers increasingly adopt LLMs as writing assistants, generating high-quality research paper introductions remains both challenging and essential. We introduce Scientific Introduction Generation (SciIG), a task that evaluates LLMs' ability to produce coherent introductions from titles, abstracts, and related works. Curating new datasets from NAACL 2025 and ICLR 2025 papers, we assess five state-of-the-art models, including both open-source (DeepSeek-v3, Gemma-3-12B, LLaMA 4-Maverick, MistralAI Small 3.1) and closed-source GPT-4o systems, across multiple dimensions: lexical overlap, semantic similarity, content coverage, faithfulness, consistency, citation correctness, and narrative quality. Our comprehensive framework combines automated metrics with LLM-as-a-judge evaluations. Results demonstrate LLaMA-4 Maverick's superior performance on most metrics, particularly in semantic similarity and faithfulness. Moreover, three-shot prompting consistently outperforms fewer-shot approaches. These findings provide practical insights into developing effective research writing assistants and set realistic expectations for LLM-assisted academic writing. To foster re- producibility and future research, we publicly release all code and datasets.