mlsanityData to 14 Oct 2026

EvolveScaler: Synthesizing Information-Evolution Contexts via Executable State Machines and Natural-Language Rendering

arXiv 2609.08435 ↗First posted 8 Sep 2026cs.AI cs.SEcited by 0

Not on the map. Why?

Abstract (arXiv)

In persistent interactions, long contexts may encode an evolving process rather than a fixed record: later events can revise or revoke earlier information, changing what remains valid and what conclusions follow. We call this setting information evolution (IE). Solving IE requires identifying valid records, applying updates in order, and reconstructing the query-relevant state from the event history. Existing text-first synthesis pipelines make such data difficult to verify because state transitions and answer logic remain implicit. We introduce EvolveScaler, a code-driven framework that defines information evolution before rendering it as natural language. Human-authored operational specifications define state transitions, record validity, difficulty controls, and executable answer logic; a strong LLM then synthesizes a self-contained simulator from each specification. Executing validated simulators produces natural-language multi-turn event histories, while deterministic replay computes reference answers and atomic checklists. We instantiate EvolveScaler with 117 task prototypes and 159 final-question operators across five difficulty levels spanning approximately 7 to 1,200 events per instance, yielding about 35,100 training examples and 585 validated evaluation instances. On the very_long tier, the strongest model reaches 59.3% avg@5, while six models score below 10%. Training an internal A3B model on 6,000 EvolveScaler examples improves performance over its base checkpoint on all eight independently constructed out-of-distribution benchmarks, with a 5.25-point average gain. These results show that code-driven IE synthesis provides both challenging evaluation and transferable training supervision.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 49 of its 62 references that are in the map. First posted on arXiv 8 Sep 2026.

Builds on

49 of 62 references.

References · 49

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  7. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  8. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  10. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  11. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  12. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
37 more
  1. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  2. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  3. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  4. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  5. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  6. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  7. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 766
  8. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  9. 2023Textbooks Are All You Needcited by 569
  10. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  11. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 434
  12. 2025LLMs Get Lost In Multi-Turn Conversationcited by 429
  13. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  14. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  15. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  16. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  17. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 337
  18. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 324
  19. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 314
  20. 2018TextWorld: A Learning Environment for Text-based Gamescited by 310
  21. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 250
  22. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  23. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  24. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  25. 2025Generalizing Verifiable Instruction Followingcited by 194
  26. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  27. 2025MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMscited by 180
  28. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 176
  29. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  30. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 106
  31. 2024Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Followingcited by 102
  32. 2024Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queriescited by 88
  33. 2026CL-bench: A Benchmark for Context Learningcited by 47
  34. 2023Entity Tracking in Language Modelscited by 44
  35. 2025Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow Real Instructions?cited by 10
  36. 2026CL-bench Life: Can Language Models Learn from Real-Life Context?cited by 9
  37. 2025MARS-Bench: A Multi-turn Athletic Real-world Scenario Benchmark for Dialogue Evaluationcited by 4

Built on by

Not cited yet.