mlsanityData to 7 Oct 2026

Exploration with Principles for Diverse AI Supervision

arXiv 2310.08899 ↗First posted 13 Oct 2023cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Training large transformers using next-token prediction has given rise to groundbreaking advancements in AI. While this generative AI approach has produced impressive results, it heavily leans on human supervision. Even state-of-the-art AI models like ChatGPT depend on fine-tuning through human demonstrations, demanding extensive human input and domain expertise. This strong reliance on human oversight poses a significant hurdle to the advancement of AI innovation. To address this limitation, we propose a novel paradigm termed Exploratory AI (EAI) aimed at autonomously generating high-quality training data. Drawing inspiration from unsupervised reinforcement learning (RL) pretraining, EAI achieves exploration within the natural language space. We accomplish this by harnessing large language models to assess the novelty of generated content. Our approach employs two key components: an actor that generates novel content following exploration principles and a critic that evaluates the generated content, offering critiques to guide the actor. Empirical evaluations demonstrate that EAI significantly boosts model performance on complex reasoning tasks, addressing the limitations of human-intensive supervision.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 39 of its 57 references that are in the map. First posted on arXiv 13 Oct 2023.

Builds on

39 of 57 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2021Program Synthesis with Large Language Modelscited by 3,509
  9. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  12. 2023Code Llama: Open Foundation Models for Codecited by 2,105
27 more
  1. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  2. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  3. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  4. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  5. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  6. 2023Teaching Large Language Models to Self-Debugcited by 963
  7. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  8. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  9. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  10. 2022Large Language Models Can Self-Improvecited by 656
  11. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  12. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  13. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  14. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  15. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  16. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  17. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  18. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  19. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  20. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  21. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  22. 2023Controllability-Aware Unsupervised Skill Discoverycited by 49
  23. 2022Learning Math Reasoning from Self-Sampled Correct and Partially-Correct Solutionscited by 38
  24. 2022Submodularity In Machine Learning and Artificial Intelligencecited by 37
  25. 2020Task-Agnostic Exploration via Policy Gradient of a Non-Parametric State Entropy Estimatecited by 21
  26. 2021Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learningcited by 21
  27. 2022Mastering the Unsupervised Reinforcement Learning Benchmark from Pixelscited by 20

Built on by

Cited by year

Citing papers per year: 2024: 2.

2 papers.

  1. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  2. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31