mlsanityData to 7 Oct 2026

Project Auto-World: Towards Automated Benchmarking of Neural Relational Reasoners

arXiv 2606.24965 ↗First posted 23 Jun 2026cs.AI cs.LGcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Reasoning about relational structures remains a significant challenge for neural models, particularly when they must systematically apply learned knowledge to problem instances that are harder than those seen in training. Progress is hampered by the difficulty of evaluating such generalization, since a priori, it is rarely clear what makes an instance hard. We study how this issue can be addressed by using large language models (LLMs) to automate benchmark generation, learning to produce increasingly challenging instances in an end-to-end manner. Concretely, given a world parametrized by Datalog rules, and an Edge Transformer as the reasoning evaluator, we use LLM-driven evolutionary search (based on FunSearch) and autonomous agentic search to discover sampling functions that yield hard problem instances. We also show that the Edge Transformer can be improved using this data such that it generalizes well to further data perturbations. Finally, we show that the same machinery can be applied to novel worlds proposed by LLMs, opening the door to autonomous research on neural relational reasoning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 38 of its 57 references that are in the map. First posted on arXiv 23 Jun 2026.

Builds on

38 of 57 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024On Memorization of Large Language Models in Logical Reasoning30 Oct

Other references · 37

  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  2. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  3. 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 763
  4. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
  5. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  6. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  7. 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 449
  8. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  9. 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexitycited by 298
  10. 2021Neural Bellman-Ford Networks: A General Graph Neural Network Framework for Link Predictioncited by 273
  11. 2021Generating Datasets with Pretrained Language Modelscited by 192
  12. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
25 more
  1. 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 155
  2. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 139
  3. 2022Discovered Policy Optimisationcited by 103
  4. 2023LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimizationcited by 86
  5. 2020Learning Reasoning Strategies in End-to-End Differentiable Provingcited by 60
  6. 2025Self-Challenging Language Model Agentscited by 56
  7. 2021Inductive logic programming at 30cited by 46
  8. 2025TaskCraft: Automated Generation of Agentic Taskscited by 46
  9. 2022Model-Free Opponent Shapingcited by 43
  10. 2021Systematic Generalization with Edge Transformerscited by 38
  11. 2024Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generationcited by 25
  12. 2024REvolve: Reward Evolution with Large Language Models using Human Feedbackcited by 22
  13. 2024FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunSearchcited by 22
  14. 2023Neural Compositional Rule Learning for Knowledge Graph Reasoningcited by 20
  15. 2024Discovering Temporally-Aware Reinforcement Learning Algorithmscited by 18
  16. 2024Towards Principled Graph Transformerscited by 11
  17. 2013Bounded Recursive Self-Improvementcited by 10
  18. 2025Generative Modeling for Mathematical Discoverycited by 9
  19. 2025GRAIL: Graph Edit Distance and Node Alignment Using LLM-Generated Codecited by 8
  20. 2022R5: Rule Discovery with Reinforced and Recurrent Relational Reasoningcited by 7
  21. 2024EvIL: Evolution Strategies for Generalisable Imitation Learningcited by 7
  22. 2024Systematic Relational Reasoning With Epistemic Graph Neural Networkscited by 4
  23. 2026Reasoning-Driven Synthetic Data Generation and Evaluationcited by 4
  24. 2025When No Paths Lead to Rome: Benchmarking Systematic Neural Relational Reasoningcited by 2
  25. 2025On Discovering Algorithms for Adversarial Imitation Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Are Large Language Models Capable of Deep Relational Reasoning? Insights from DeepSeek-R1 and Benchmark Comparisons
  2. 2026Evaluating Relational Reasoning in LLMs with REL
  3. 2025Agentic Reasoning: A Streamlined Framework for Enhancing LLM Reasoning with Agentic Tools
  4. 2026OPT-BENCH: Evaluating the Iterative Self-Optimization of LLM Agents in Large-Scale Search Spaces
  5. 2023WorldSense: A Synthetic Benchmark for Grounded Reasoning in Large Language Models
  6. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems
  7. 2024LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models
  8. 2024Navigating the Labyrinth: Evaluating LLMs' Ability to Reason About Search Problems