# Truthful Online Preference Aggregation for LLM Fine-Tuning in Mobile Crowdsourcing

arXiv: https://arxiv.org/abs/2605.24052 · HTML: https://mlsanity.com/paper/2605.24052/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 64 references that are in the map.
- First posted on arXiv 22 May 2026.

## Builds on (most cited first, 30 of 32)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint](https://mlsanity.com/paper/2312.11456/) (2023)
- [RLHF Workflow: From Reward Modeling to Online RLHF](https://mlsanity.com/paper/2405.07863/) (2024)
- [Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback](https://mlsanity.com/paper/2404.10271/) (2024)
- [RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation](https://mlsanity.com/paper/2405.00254/) (2024)
- [Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2504.12501/) (2025)
- [Online Iterative Reinforcement Learning from Human Feedback with General Preference Model](https://mlsanity.com/paper/2402.07314/) (2024)
- [Truthful Aggregation of LLMs with an Application to Online Advertising](https://mlsanity.com/paper/2405.05905/) (2024)
- [Auctions with LLM Summaries](https://mlsanity.com/paper/2404.08126/) (2024)
- [Mechanism Design for LLM Fine-tuning with Multiple Reward Models](https://mlsanity.com/paper/2405.16276/) (2024)
- [Cached Model-as-a-Resource: Provisioning Large Language Model Agents for Edge Intelligence in Space-air-ground Integrated Networks](https://mlsanity.com/paper/2403.05826/) (2024)
- [No-Regret and Incentive-Compatible Online Learning](https://mlsanity.com/paper/2002.08837/) (2020)
- [Double or Nothing: Multiplicative Incentive Mechanisms for Crowdsourcing](https://mlsanity.com/paper/1408.1387/) (2014)
- [Online Prediction with Selfish Experts](https://mlsanity.com/paper/1702.03615/) (2017)
- [Online Learning from Strategic Human Feedback in LLM Fine-Tuning](https://mlsanity.com/paper/2412.16834/) (2024)
- [Incentive-Compatible Forecasting Competitions](https://mlsanity.com/paper/2101.01816/) (2021)
- [Meta-Learning Adversarial Bandit Algorithms](https://mlsanity.com/paper/2307.02295/) (2023)
- [Large Language Models and Their Applications in Roadway Safety and Mobility Enhancement: A Comprehensive Review](https://mlsanity.com/paper/2506.06301/) (2025)
- [Mechanism Design for Collaborative Normal Mean Estimation](https://mlsanity.com/paper/2306.06351/) (2023)
- [Collaborative Algorithms for Online Personalized Mean Estimation](https://mlsanity.com/paper/2208.11530/) (2022)
- [Efficient Online Learning with Memory via Frank-Wolfe Optimization: Algorithms with Bounded Dynamic Regret and Applications to Control](https://mlsanity.com/paper/2301.00497/) (2023)
- [Truthful Incentive Mechanism for Federated Learning with Crowdsourced Data Labeling](https://mlsanity.com/paper/2302.00106/) (2023)
- [Slowly Changing Adversarial Bandit Algorithms are Efficient for Discounted MDPs](https://mlsanity.com/paper/2205.09056/) (2022)
- [Incentivized Truthful Communication for Federated Bandits](https://mlsanity.com/paper/2402.04485/) (2024)

## Abstract (arXiv)

To better serve users' demands in mobile applications (e.g., navigation), mobile crowdsourcing platforms can iteratively align large language model (LLM)-generated content (e.g., AI-generated traffic condition predictions) with human feedback collected from crowdsourcing workers (e.g., mobile users). However, workers may strategically misreport their online preference feedback to maximize their influence or payment. Existing pipelines in mobile crowdsourcing (e.g., EM-based weight estimation) fail to identify the most accurate worker in this online setting, resulting in a linear regret O(T) over T time slots. In this paper, we study truthful online preference aggregation for LLM fine-tuning in mobile crowdsourcing. We formulate a new dynamic Bayesian game to model the multi-agent online learning process between the platform and strategic mobile workers. We propose a novel online weighted aggregation mechanism that dynamically adjusts each worker's weight in the preference aggregation according to their feedback accuracy. We prove that our mechanism ensures truthful feedback from strategic workers and achieves a sublinear regret O(√T) over T time slots. We further extend our mechanism to a challenging scenario with limited worker feedback per time slot, still guaranteeing a sublinear regret O(√T). Experiments on LLM fine-tuning with real-world datasets further demonstrate significant performance gains of our mechanisms over benchmark schemes.