mlsanityData to 7 Oct 2026

Truthful Online Preference Aggregation for LLM Fine-Tuning in Mobile Crowdsourcing

arXiv 2605.24052 ↗First posted 22 May 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

To better serve users' demands in mobile applications (e.g., navigation), mobile crowdsourcing platforms can iteratively align large language model (LLM)-generated content (e.g., AI-generated traffic condition predictions) with human feedback collected from crowdsourcing workers (e.g., mobile users). However, workers may strategically misreport their online preference feedback to maximize their influence or payment. Existing pipelines in mobile crowdsourcing (e.g., EM-based weight estimation) fail to identify the most accurate worker in this online setting, resulting in a linear regret O(T) over T time slots. In this paper, we study truthful online preference aggregation for LLM fine-tuning in mobile crowdsourcing. We formulate a new dynamic Bayesian game to model the multi-agent online learning process between the platform and strategic mobile workers. We propose a novel online weighted aggregation mechanism that dynamically adjusts each worker's weight in the preference aggregation according to their feedback accuracy. We prove that our mechanism ensures truthful feedback from strategic workers and achieves a sublinear regret O(√T) over T time slots. We further extend our mechanism to a challenging scenario with limited worker feedback per time slot, still guaranteeing a sublinear regret O(√T). Experiments on LLM fine-tuning with real-world datasets further demonstrate significant performance gains of our mechanisms over benchmark schemes.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 32 of its 64 references that are in the map. First posted on arXiv 22 May 2026.

Builds on

32 of 64 references.

References · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  9. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  10. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 95
  11. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 71
  12. 2025Reinforcement Learning from Human Feedbackcited by 53
20 more
  1. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  2. 2024Truthful Aggregation of LLMs with an Application to Online Advertisingcited by 29
  3. 2024Auctions with LLM Summariescited by 24
  4. 2024Mechanism Design for LLM Fine-tuning with Multiple Reward Modelscited by 18
  5. 2024Cached Model-as-a-Resource: Provisioning Large Language Model Agents for Edge Intelligence in Space-air-ground Integrated Networkscited by 15
  6. 2020No-Regret and Incentive-Compatible Online Learningcited by 14
  7. 2014Double or Nothing: Multiplicative Incentive Mechanisms for Crowdsourcingcited by 11
  8. 2017Online Prediction with Selfish Expertscited by 10
  9. 2024Online Learning from Strategic Human Feedback in LLM Fine-Tuningcited by 10
  10. 2021Incentive-Compatible Forecasting Competitionscited by 9
  11. 2023Meta-Learning Adversarial Bandit Algorithmscited by 8
  12. 2025Large Language Models and Their Applications in Roadway Safety and Mobility Enhancement: A Comprehensive Reviewcited by 8
  13. 2023Mechanism Design for Collaborative Normal Mean Estimationcited by 7
  14. 2022Collaborative Algorithms for Online Personalized Mean Estimationcited by 5
  15. 2023Efficient Online Learning with Memory via Frank-Wolfe Optimization: Algorithms with Bounded Dynamic Regret and Applications to Controlcited by 5
  16. 2023Truthful Incentive Mechanism for Federated Learning with Crowdsourced Data Labelingcited by 4
  17. 2022Slowly Changing Adversarial Bandit Algorithms are Efficient for Discounted MDPscited by 2
  18. 2024Incentivized Truthful Communication for Federated Banditscited by 2
  19. 2025Online Mixture of Experts: No-Regret Learning for Optimal Collective Decision-Makingcited by 2
  20. 2025On the Universal Near Optimality of Hedge in Combinatorial Settingscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2018From Social to Individuals: a Parsimonious Path of Multi-level Models for Crowdsourced Preference Aggregation
  2. 2024Rank Aggregation in Crowdsourcing for Listwise Annotations
  3. 2025Beyond Majority Voting: LLM Aggregation by Leveraging Higher-Order Information
  4. 2025Crowd-SFT: Crowdsourcing for LLM Alignment
  5. 2024Crowd-PrefRL: Preference-Based Reward Learning from Crowds
  6. 2024Do LLM Agents Have Regret? A Case Study in Online Learning and Games
  7. 2026Consensus vs. Dissent: Dynamic LLM Modeling of Subjective Preferences in Group Recommenders
  8. 2025HiveMind: Contribution-Guided Online Prompt Optimization of LLM Multi-Agent Systems