mlsanityData to 7 Oct 2026

Gacha Decoding: Eliciting Diverse Generations Through Instruction Following

arXiv 2610.01382 ↗First posted 1 Oct 2026cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce Gacha Decoding, an inference-time method for eliciting diverse language model generations that scales with model capability. Across open-ended domains (in-the-wild chat, creative writing, planning for image generation, and protein design), Gacha Decoding significantly outperforms existing generation diversity approaches at equal quality (up to 2.4x Vendi over the next-best prior approach), reaching the same number of high-quality modes with over an order of magnitude fewer samples (11.0x) and discovering novel modes that no other approach surfaces. Our key insight is to treat diversity as an instruction-following problem: rather than relying on the LM's token entropy, we combine its instruction-following capability with randomness from an external RNG tool to scalably identify and realize distinct modes of the response space. This approach of "planning with dice" enables Gacha to invert the long-observed tension between diversity and model capability. As the underlying LM becomes a better instruction follower, diversity under Gacha Decoding consistently improves--even as its token entropy and diversity under prior approaches decline. Together, our results highlight that instruction following, rather than token entropy alone, can drive generation diversity.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 43 of its 61 references that are in the map. First posted on arXiv 1 Oct 2026.

Builds on

43 of 61 references.

References · 43

  1. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  2. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  3. 2018Hierarchical Neural Story Generationcited by 1,182
  4. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  5. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  6. 2023Whose Opinions Do Language Models Reflect?cited by 759
  7. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  8. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  9. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 397
  10. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  11. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 348
  12. 2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 322
31 more
  1. 2022The Vendi Score: A Diversity Evaluation Metric for Machine Learningcited by 299
  2. 2018Plan-And-Write: Towards Better Automatic Storytellingcited by 232
  3. 2023Does Writing with Language Models Reduce Content Diversity?cited by 190
  4. 2026Gemma 4 Technical Reportcited by 187
  5. 2025Generalizing Verifiable Instruction Followingcited by 178
  6. 2025Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)cited by 137
  7. 2025Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversitycited by 100
  8. 2025OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generationcited by 81
  9. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 79
  10. 2023EQ-Bench: An Emotional Intelligence Benchmark for Large Language Modelscited by 75
  11. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 68
  12. 2025NoveltyBench: Evaluating Language Models for Humanlike Diversitycited by 64
  13. 2024One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversitycited by 57
  14. 2025Evaluating the Diversity and Quality of LLM Generated Contentcited by 51
  15. 2024Forcing Diffuse Distributions out of Language Modelscited by 48
  16. 2025Base Models Beat Aligned Models at Randomness and Creativitycited by 48
  17. 2024Echoes in AI: Quantifying lack of plot diversity in LLM outputscited by 47
  18. 2018Toward Diverse Text Generation with Inverse Reinforcement Learningcited by 41
  19. 2025Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoningcited by 38
  20. 2024Empirical evidence of Large Language Model's influence on human spoken communicationcited by 27
  21. 2020Diversifying Dialogue Generation with Non-Conversational Textcited by 21
  22. 2024Generative Monoculture in Large Language Modelscited by 21
  23. 2026How LLMs Distort Our Written Languagecited by 20
  24. 2025Spectrum Tuning: Post-Training for Distributional Coverage and In-Context Steerabilitycited by 17
  25. 2025String Seed of Thought: Prompting LLMs for Distribution-Faithful and Diverse Generationcited by 11
  26. 2026Where does output diversity collapse in post-training?cited by 10
  27. 2025Mode-Conditioning Unlocks Superior Test-Time Scalingcited by 8
  28. 2025Intent Factored Generation: Unleashing the Diversity in Your Language Modelcited by 6
  29. 2025The Basic B*** Effect: The Use of LLM-based Agents Reduces the Distinctiveness and Diversity of People's Choicescited by 5
  30. 2026Annotations Mitigate Post-Training Mode Collapsecited by 4
  31. 2026Where You Inject Diversity Matters: A Unified Framework for Diverse Generationcited by 2

Built on by

Not cited yet.