mlsanityData to 7 Oct 2026

COS(M+O)S: Curiosity and RL-Enhanced MCTS for Exploring Story Space via Language Models

arXiv 2501.17104 ↗First posted 28 Jan 2025cs.CL cs.AIcited by 3

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

We present COS(M+O)S, a System 2-inspired framework for open-ended plot development that systematically explores the vast space of possible story expansions, enabling a 3B-parameter language model to approach the plot quality of a 70B model on select short-story tasks. The method accomplishes this by combining Monte Carlo Tree Search (MCTS), guided by a step-level value model that rewards moderate surprisal (curiosity) while penalizing incoherence, and Odds Ratio Preference Optimization (ORPO) to fine-tune the policy on high-value plot expansions. This iterative reinforcement learning loop systematically explores multiple candidate plot branches, backpropagates quality signals, and adapts the policy for faster convergence, notably shifting the policy from puzzle-based Chain-of-Thought to more character-driven storytelling. In small-scale tests with short-story prompts, 67%-77% of participants favored COS(M+O)S's highest-rated expansions over lower-rated ones, suggesting that our learned value function aligns. GPT-4o ratings further show that COS(M+O)S surpasses naive single-pass decoding from Llama 3.2 3B by 0.59 SD, coming within 0.06 SD of Llama 3.1 70B (no significant difference, p=0.93). Pairwise comparisons with o1 place COS(M+O)S 1.5 SD above the 3B baseline and find no statistically significant gap from 70B. Nevertheless, absolute story quality remains modest, constrained by the small model's capacity and limited training data.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 22 of its 44 references that are in the map. First posted on arXiv 28 Jan 2025.

Builds on

22 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solvers12 Aug

Other references · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  7. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  8. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  9. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  10. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  11. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  12. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
9 more
  1. 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
  2. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
  3. 2018Plan-And-Write: Towards Better Automatic Storytellingcited by 232
  4. 2024Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systemscited by 162
  5. 2024AlphaMath Almost Zero: Process Supervision without Processcited by 160
  6. 2024Marco-o1: Towards Open Reasoning Models for Open-Ended Solutionscited by 142
  7. 2019Online Replanning in Belief Space for Partially Observable Task and Motion Problemscited by 60
  8. 2023Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decodingcited by 59
  9. 2018Controllable Neural Story Plot Generation via Reward Shapingcited by 49

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025Avoidance Decoding for Diverse Multi-Branch Story Generationcited by 6
  3. 2026BiT-MCTS: A Theme-based Bidirectional MCTS Approach to Chinese Fiction Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026PlotTwist: A Creative Plot Generation Framework with Small Language Models
  2. 2026Rewarding Creativity: A Human-Aligned Generative Reward Model for Reinforcement Learning in Storytelling
  3. 2026StoryLens: Preference-Aligned Story Rewriting via Context-Aware Narrative Enrichment
  4. 2026Retell, Reward, Repeat: Reinforcement Learning for Narrative Theory-Informed Story Retelling
  5. 2026StoryAlign: Evaluating and Training Reward Models for Story Generation
  6. 2023GROVE: A Retrieval-augmented Complex Story Generation Framework with A Forest of Evidence
  7. 2025Style over Story: Measuring LLM Narrative Preferences via Structured Selection
  8. 2022Robust Preference Learning for Storytelling via Contrastive Reinforcement Learning