mlsanityData to 14 Oct 2026

Tape: A Cellular Automata Benchmark for Evaluating Rule-Shift Generalization in Reinforcement Learning

arXiv 2601.04695 ↗First posted 8 Jan 2026cs.AI cs.LGcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Out-of-distribution generalization in reinforcement learning is hard to diagnose when benchmark shifts mix dynamics, observations, goals, and rewards. We address this with Tape, a controlled benchmark that isolates latent rule-shift in dynamics while keeping the observation-action interface fixed. The protocol combines deterministic splits, 20-seed replication, bootstrap uncertainty reporting, and continuous metrics for sparse-success regimes. Across baseline families, we find a consistent ID-to-OOD drop and strong heterogeneity across stable/periodic/chaotic rules. Importantly, this fragility appears even in an intentionally simple 1D deterministic setting, suggesting that many current RL algorithms remain brittle to latent-law changes under minimal confounds. To calibrate strict success, we report a protocol-matched true-dynamics random-shooting reference (p_oracle is almost 0.187) and oracle-normalized scores ON(p) = 100 p / p_oracle; this is a budgeted operational reference, not a global-optimality bound. A smaller feasibility regime (L = H = 16) with 100% rule-wise solvability helps separate reachability limits from policy failure. These results position Tape as a mechanism-oriented diagnostic for robust adaptation and latent-mechanism inference, and as a controlled benchmark relevant to broader AGI-oriented evaluation without making strong AGI sufficiency claims.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 30 of its 44 references that are in the map. First posted on arXiv 8 Jan 2026.

Builds on

30 of 44 references.

References · 30

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  3. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  5. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  6. 2023Mastering Diverse Domains through World Modelscited by 1,253
  7. 2020Mastering Atari with Discrete World Modelscited by 1,007
  8. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  9. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  10. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  11. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  12. 2016Learning to learn by gradient descent by gradient descentcited by 620
18 more
  1. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  2. 2020Reinforcement Learning with Augmented Datacited by 468
  3. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
  4. 2019Challenges of Real-World Reinforcement Learningcited by 291
  5. 2018Online Learning: A Comprehensive Surveycited by 205
  6. 2018Assessing Generalization in Deep Reinforcement Learningcited by 133
  7. 2021Non-stationary Reinforcement Learning without Prior Knowledge: An Optimal Black-box Approachcited by 98
  8. 2017Towards Generalization and Simplicity in Continuous Controlcited by 68
  9. 2024GenRL: Multimodal-foundation world models for generalization in embodied agentscited by 31
  10. 2024CARMO: Dynamic Criteria Generation for Context-Aware Reward Modellingcited by 22
  11. 2024Few-Shot Recognition via Stage-Wise Retrieval-Augmented Finetuningcited by 15
  12. 2022Context-Tuning: Learning Contextualized Prompts for Natural Language Generationcited by 12
  13. 2025Direct Retrieval-augmented Optimization: Synergizing Knowledge Selection and Language Modelscited by 9
  14. 2023Context Tuning for Retrieval Augmented Generationcited by 6
  15. 2025LLM Program Optimization via Retrieval Augmented Searchcited by 4
  16. 2025PRESTO: Preimage-Informed Instruction Optimization for Prompting Black-Box LLMscited by 4
  17. 2025Reinforced Informativeness Optimization for Long-Form Retrieval-Augmented Generationcited by 2
  18. 2025CellARC: Measuring Intelligence with Cellular Automatacited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Benchmarking Reinforcement Learning via Stochastic Converse Optimality: Generating Systems with Known Optimal Policies
  2. 2025Rethinking RL Evaluation: Can Benchmarks Truly Reveal Failures of RL Methods?
  3. 2024ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learning
  4. 2023Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning
  5. 2026Smaller Abstract State Spaces Enable Cross-Scale Generalization in Reinforcement Learning
  6. 2024ODRL: A Benchmark for Off-Dynamics Reinforcement Learning
  7. 2025Gym4ReaL: A Suite for Benchmarking Real-World Reinforcement Learning
  8. 2026Trajectory-Aware Policy Optimization: Closing the State Distribution Gap in Long-Horizon LLM RL