The two clocks and the innovation window: When and how generative models learn rules

Generative models trained on finite data face a fundamental tension: their score-matching or next-token objective converges to the empirical training distribution rather than the population distribution we seek to learn. Using rule-valid synthetic tasks, we trace this tension across two training timescales: τrule, the step at which generations first become rule-valid, and τmem, the step at which models begin reproducing training samples. Focusing on parity and extending to other binary rules and combinatorial puzzles, we characterize how these two clocks, τrule and τmem, depend on key aspects of the learning setup. Specifically, we show that τrule increases with rule complexity and decreases with model capacity, while τmem is approximately invariant to the rule and scales nearly linearly with dataset size N. We define the innovation window as the interval [τrule, τmem]. This window widens with increasing N and narrows with rule complexity, and may vanish entirely when τrule ≥ τmem. The same two-clock structure arises in both diffusion (DiT) and autoregressive (GPT) models, with architecture-dependent offsets. Dissecting the learned score of DiT models reveals a corresponding evolution of the optimization landscapes, where rule-valid samples' basins expand substantially around τrule, while training samples' basins begin to dominate around τmem. Together, these results yield a unified and predictive account of when and how generative models exhibit genuine innovation.