# Can we use LLMs to bootstrap reinforcement learning? -- A case study in digital health behavior change

arXiv: https://arxiv.org/abs/2511.17630 · HTML: https://mlsanity.com/paper/2511.17630/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 20 of its 75 references that are in the map.
- First posted on arXiv 19 Nov 2025.

## Placed in

- Synthetic data and self-improvement for LLMs / [Self-rewarding alignment and self-training](https://mlsanity.com/synth/self-rewarding-alignment-and-self-training/)

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [State of What Art? A Call for Multi-Prompt LLM Evaluation](https://mlsanity.com/paper/2401.00595/) (2023)
- [Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model](https://mlsanity.com/paper/2211.02001/) (2022)
- [LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks](https://mlsanity.com/paper/2406.18403/) (2024)
- [Power Hungry Processing: Watts Driving the Cost of AI Deployment?](https://mlsanity.com/paper/2311.16863/) (2023)
- [Having Beer after Prayer? Measuring Cultural Bias in Large Language Models](https://mlsanity.com/paper/2305.14456/) (2023)
- [AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion Prediction](https://mlsanity.com/paper/2305.09620/) (2023)
- [IntelligentPooling: Practical Thompson Sampling for mHealth](https://mlsanity.com/paper/2008.01571/) (2020)
- [Non-Stationary Bandits with Habituation and Recovery Dynamics](https://mlsanity.com/paper/1707.08423/) (2017)
- [Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-implementation Guidelines](https://mlsanity.com/paper/2206.03944/) (2022)
- [PATIENT-Psi: Using Large Language Models to Simulate Patients for Training Mental Health Professionals](https://mlsanity.com/paper/2405.19660/) (2024)
- [Reward Design For An Online Reinforcement Learning Algorithm Supporting Oral Self-Care](https://mlsanity.com/paper/2208.07406/) (2022)
- [Evaluating Gender Bias of LLMs in Making Morality Judgements](https://mlsanity.com/paper/2410.09992/) (2024)
- [A Systematic Analysis of Large Language Models as Soft Reasoners: The Case of Syllogistic Inferences](https://mlsanity.com/paper/2406.11341/) (2024)
- [A Deployed Online Reinforcement Learning Algorithm In An Oral Health Clinical Trial](https://mlsanity.com/paper/2409.02069/) (2024)
- [Burn After Reading: Do Multimodal Large Language Models Truly Capture Order of Events in Image Sequences?](https://mlsanity.com/paper/2506.10415/) (2025)
- [Persuading to Prepare for Quitting Smoking with a Virtual Coach: Using States and User Characteristics to Predict Behavior](https://mlsanity.com/paper/2304.02264/) (2023)

## Abstract (arXiv)

Personalizing digital applications for health behavior change is a promising route to making them more engaging and effective. This especially holds for approaches that adapt to users and their specific states (e.g., motivation, knowledge, wants) over time. However, developing such approaches requires making many design choices, whose effectiveness is difficult to predict from literature and costly to evaluate in practice. In this work, we explore whether large language models (LLMs) can be used out-of-the-box to generate samples of user interactions that provide useful information for training reinforcement learning models for digital behavior change settings. Using real user data from four large behavior change studies as comparison, we show that LLM-generated samples can be useful in the absence of real data. Comparisons to the samples provided by human raters further show that LLM-generated samples reach the performance of human raters. Additional analyses of different prompting strategies including shorter and longer prompt variants, chain-of-thought prompting, and few-shot prompting show that the relative effectiveness of different strategies depends on both the study and the LLM with also relatively large differences between prompt paraphrases alone. We provide recommendations for how LLM-generated samples can be useful in practice.