mlsanityData to 14 Oct 2026

CASTILLO: Characterizing Response Length Distributions of Large Language Models

arXiv 2505.16881 ↗First posted 22 May 2025cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Efficiently managing compute resources for Large Language Model (LLM) inference remains challenging due to the inherently stochastic and variable lengths of autoregressive text generation. Accurately estimating response lengths in advance enables proactive resource allocation, yet existing approaches either bias text generation towards certain lengths or rely on assumptions that ignore model- and prompt-specific variability. We introduce CASTILLO, a dataset characterizing response length distributions across 13 widely-used open-source LLMs evaluated on seven distinct instruction-following corpora. For each langleprompt, modelrangle sample pair, we generate 10 independent completions using fixed decoding hyper-parameters, record the token length of each response, and publish summary statistics (mean, std-dev, percentiles), along with the shortest and longest completions, and the exact generation settings. Our analysis reveals significant inter- and intra-model variability in response lengths (even under identical generation settings), as well as model-specific behaviors and occurrences of partial text degeneration in only subsets of responses. CASTILLO enables the development of predictive models for proactive scheduling and provides a systematic framework for analyzing model-specific generation behaviors. We publicly release the dataset and code to foster research at the intersection of generative language modeling and systems.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 27 of its 32 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

27 of 32 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  9. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  11. 2021Program Synthesis with Large Language Modelscited by 3,637
  12. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
15 more
  1. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  2. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  3. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  4. 2021Measuring Coding Challenge Competence With APPScited by 902
  5. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  6. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  7. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  8. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  9. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
  10. 2024xLSTM: Extended Long Short-Term Memorycited by 327
  11. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  12. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipelinecited by 86
  13. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  14. 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Servingcited by 52
  15. 2024Don't Stop Me Now: Embedding Based Scheduling for LLMscited by 35

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Conflicts Make Large Reasoning Models Vulnerable to Attackscited by 1