mlsanityData to 14 Oct 2026

MAPLE: A Framework for Active Preference Learning Guided by Large Language Models

arXiv 2412.07207 ↗First posted 10 Dec 2024cs.LG cs.AI cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

The advent of large language models (LLMs) has sparked significant interest in using natural language for preference learning. However, existing methods often suffer from high computational burdens, taxing human supervision, and lack of interpretability. To address these issues, we introduce MAPLE, a framework for large language model-guided Bayesian active preference learning. MAPLE leverages LLMs to model the distribution over preference functions, conditioning it on both natural language feedback and conventional preference learning feedback, such as pairwise trajectory rankings. MAPLE also employs active learning to systematically reduce uncertainty in this distribution and incorporates a language-conditioned active query selection mechanism to identify informative and easy-to-answer queries, thus reducing human burden. We evaluate MAPLE's sample efficiency and preference inference quality across two benchmarks, including a real-world vehicle route planning benchmark using OpenStreetMap data. Our results demonstrate that MAPLE accelerates the learning process and effectively improves humans' ability to answer queries.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 30 of its 58 references that are in the map. First posted on arXiv 10 Dec 2024.

Builds on

30 of 58 references.

References · 30

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  3. 2017Deep Bayesian Active Learning with Image Datacited by 819
  4. 2022A Survey on In-context Learningcited by 721
  5. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  6. 2023Language to Rewards for Robotic Skill Synthesiscited by 312
  7. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
  8. 2020Reward Machines: Exploiting Reward Function Structure in Reinforcement Learningcited by 191
  9. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 156
  10. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 134
  11. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  12. 2018Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Informationcited by 115
18 more
  1. 2024STaR-GATE: Teaching Language Models to Ask Clarifying Questionscited by 104
  2. 2017Robust Imitation of Diverse Behaviorscited by 86
  3. 2018SDRL: Interpretable and Data-efficient Deep Reinforcement Learning Leveraging Symbolic Planningcited by 78
  4. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  5. 2019Asking Easy Questions: A User-Friendly Approach to Active Reward Learningcited by 68
  6. 2023Learning to Model the World with Languagecited by 68
  7. 2022Learning Neuro-Symbolic Skills for Bilevel Planningcited by 58
  8. 2023A Survey on Few-Shot Class-Incremental Learningcited by 48
  9. 2020Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentationcited by 42
  10. 2023Active Preference Inference using Language Models and Probabilistic Reasoningcited by 29
  11. 2022Leveraging Approximate Symbolic Models for Reinforcement Learning via Skill Diversitycited by 22
  12. 2020Value Alignment Verificationcited by 20
  13. 2018Learning from Richer Human Guidance: Augmenting Comparison-Based Learning with Feature Queriescited by 16
  14. 2019Interactive Classification by Asking Informative Questionscited by 16
  15. 2021Learning Perceptual Concepts by Bootstrapping from Human Queriescited by 12
  16. 2022Relative Behavioral Attributes: Filling the Gap between Symbolic Goal Specification and Reward Learning from Human Preferencescited by 7
  17. 2018Cost Functions for Robot Motion Stylecited by 5
  18. 2022Towards customizable reinforcement learning agents: Enabling preference specification through online vocabulary expansioncited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2026Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewardscited by 11
  2. 2026Know You Before You Speak: User-State Modeling for LLM Personalization in Multi-Turn Conversationcited by 3
  3. 2025Efficiently Generating Expressive Quadruped Behaviors via Language-Guided Preference Learningcited by 1
  4. 2026Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelinescited by 1
  5. 2026Adaptive Prompt Elicitation for Text-to-Image Generationcited by 0
  6. 2026Bayesian Active Learning for Intent Disambiguation in Interactive Robot Planningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Deep Bayesian Active Learning for Preference Modeling in Large Language Models
  2. 2026Nearly Optimal Active Preference Learning and Its Application to LLM Alignment
  3. 2023Sample Efficient Preference Alignment in LLMs via Active Exploration
  4. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  5. 2026ActiveUltraFeedback: Efficient Preference Data Generation using Active Learning
  6. 2024Active Preference Learning for Large Language Models
  7. 2024Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Trees
  8. 2025ActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignment