mlsanityData to 7 Oct 2026

CoAct: Co-Active LLM Preference Learning with Human-AI Synergy

arXiv 2604.17501 ↗First posted 19 Apr 2026cs.CLcited by 3

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Learning from preference-based feedback has become an effective approach for aligning LLMs across diverse tasks. However, high-quality human-annotated preference data remains expensive and scarce. Existing methods address this challenge through either self-rewarding, which scales by using purely AI-generated labels but risks unreliability, or active learning, which ensures quality through oracle annotation but cannot fully leverage unlabeled data. In this paper, we present CoAct, a novel framework that synergistically combines self-rewarding and active learning through strategic human-AI collaboration. CoAct leverages self-consistency to identify both reliable self-labeled data and samples that require oracle verification. Additionally, oracle feedback guides the model to generate new instructions within its solvable capability. Evaluated on three reasoning benchmarks across two model families, CoAct achieves average improvements of +13.25% on GSM8K, +8.19% on MATH, and +13.16% on WebInstruct, consistently outperforming all baselines.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 44 of its 68 references that are in the map. First posted on arXiv 19 Apr 2026.

Builds on

44 of 68 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025ActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignment25 May
  2. 2024Self-Consistency Preference Optimization6 Nov
  3. 2024Direct Preference Optimization with an Offset16 Feb
  4. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 39

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  7. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  8. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  9. 2022Large Language Models Can Self-Improvecited by 656
  10. 2022Out-of-Distribution Detection with Deep Nearest Neighborscited by 596
  11. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  12. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
27 more
  1. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  2. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
  3. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  4. 2025General-Reasoner: Advancing LLM Reasoning Across All Domainscited by 132
  5. 2022Natural Language to Code Translation with Executioncited by 122
  6. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  7. 2025Can Large Reasoning Models Self-Train?cited by 66
  8. 2024Active Preference Optimization for Sample Efficient RLHFcited by 61
  9. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  10. 2024Explaining Length Bias in LLM-Based Preference Evaluationscited by 48
  11. 2024Preference Optimization for Reasoning with Pseudo Feedbackcited by 38
  12. 2022Uncertainty Estimation for Language Reward Modelscited by 35
  13. 2023Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Costcited by 33
  14. 2021Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistantscited by 31
  15. 2024AD-LLM: Benchmarking Large Language Models for Anomaly Detectioncited by 25
  16. 2024Large Language Models for Anomaly and Out-of-Distribution Detection: A Surveycited by 24
  17. 2024Let's Ask GNN: Empowering Large Language Model for Graph In-Context Learningcited by 21
  18. 2025Population-Aligned Persona Generation for LLM-based Social Simulationcited by 20
  19. 2025Too Consistent to Detect: A Study of Self-Consistent Errors in LLMscited by 18
  20. 2024Empowering Large Language Models for Textual Data Augmentationcited by 17
  21. 2024Towards Acyclic Preference Evaluation of Language Models via Multiple Evaluatorscited by 13
  22. 2025A Survey of Large Language Models for Text-Guided Molecular Discovery: from Molecule Generation to Optimizationcited by 12
  23. 2025POLO: Preference-Guided Multi-Turn Reinforcement Learning for Lead Optimizationcited by 10
  24. 2025A Survey on Efficient Large Language Model Training: From Data-centric Perspectivescited by 9
  25. 2026HumanLLM: Towards Personalized Understanding and Simulation of Human Naturecited by 5
  26. 2026DS2-Instruct: Domain-Specific Data Synthesis for Large Language Models Instruction Tuningcited by 4
  27. 2026MolMem: Memory-Augmented Agentic Reinforcement Learning for Sample-Efficient Molecular Optimizationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026When the Strongest Teacher Is Not the Best Teacher: Student-Centric Answer Selectioncited by 1
  2. 2026DOG-DPO:Dynamic Optimization in Geometry for Safety Alignmentcited by 1
  3. 2026GTA: Graph Theory Agent and Benchmark for Algorithmic Graph Reasoning with LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024ICPL: Few-shot In-context Preference Learning via LLMs
  2. 2023Sample Efficient Preference Alignment in LLMs via Active Exploration
  3. 2026Nearly Optimal Active Preference Learning and Its Application to LLM Alignment
  4. 2026Optimal Design for Active Preference Learning with Biased LLM Judges
  5. 2025Multi-Level Aware Preference Learning: Enhancing RLHF for Complex Multi-Instruction Tasks
  6. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
  7. 2026Automated Concept Discovery for LLM-as-a-Judge Preference Analysis
  8. 2024Active Preference Learning for Large Language Models