# Prosocial Behavior Detection in Player Game Chat: From Aligning Human-AI Definitions to Efficient Annotation at Scale

arXiv: https://arxiv.org/abs/2508.05938 · HTML: https://mlsanity.com/paper/2508.05938/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 44 references that are in the map.
- First posted on arXiv 8 Aug 2025.

## Builds on (most cited first, 19 of 19)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021)
- [Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning](https://mlsanity.com/paper/2301.11916/) (2023)
- [Mitigating Label Biases for In-context Learning](https://mlsanity.com/paper/2305.19148/) (2023)
- [Estimating Expected Calibration Errors](https://mlsanity.com/paper/2109.03480/) (2021)
- [Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biases](https://mlsanity.com/paper/2112.07868/) (2021)
- [Large Language Models in Qualitative Research: Uses, Tensions, and Intentions](https://mlsanity.com/paper/2410.07362/) (2024)
- [The Promises and Pitfalls of LLM Annotations in Dataset Labeling: a Case Study on Media Bias Detection](https://mlsanity.com/paper/2411.11081/) (2024)
- [CONDA: a CONtextual Dual-Annotated dataset for in-game toxicity understanding and detection](https://mlsanity.com/paper/2106.06213/) (2021)
- [LLM Chain Ensembles for Scalable and Accurate Data Annotation](https://mlsanity.com/paper/2410.13006/) (2024)
- [A Structured Review of Literature on Uncertainty in Machine Learning & Deep Learning](https://mlsanity.com/paper/2406.00332/) (2024)
- [Can You Label Less by Using Out-of-Domain Data? Active & Transfer Learning with Few-shot Instructions](https://mlsanity.com/paper/2211.11798/) (2022)
- [LLM Confidence Evaluation Measures in Zero-Shot CSS Classification](https://mlsanity.com/paper/2410.13047/) (2024)
- [ToxBuster: In-game Chat Toxicity Buster with BERT](https://mlsanity.com/paper/2305.12542/) (2023)
- [Unified Game Moderation: Soft-Prompting and LLM-Assisted Label Transfer for Resource-Efficient Toxicity Detection](https://mlsanity.com/paper/2506.06347/) (2025)
- [STFU NOOB! Predicting Crowdsourced Decisions on Toxic Behavior in Online Games](https://mlsanity.com/paper/1404.5905/) (2014)
- [Self-Anchored Attention Model for Sample-Efficient Classification of Prosocial Text Chat](https://mlsanity.com/paper/2506.09259/) (2025)

## Built on by (most cited first, 1 of 1)

- [On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance](https://mlsanity.com/paper/2606.00467/) (2026)

## Abstract (arXiv)

Detecting prosociality in text--communication intended to affirm, support, or improve others' behavior--is a novel and increasingly important challenge for trust and safety systems. Unlike toxic content detection, prosociality lacks well-established definitions and labeled data, requiring new approaches to both annotation and deployment. We present a practical, three-stage pipeline that enables scalable, high-precision prosocial content classification while minimizing human labeling effort and inference costs. First, we identify the best LLM-based labeling strategy using a small seed set of human-labeled examples. We then introduce a human-AI refinement loop, where annotators review high-disagreement cases between GPT-4 and humans to iteratively clarify and expand the task definition-a critical step for emerging annotation tasks like prosociality. This process results in improved label quality and definition alignment. Finally, we synthesize 10k high-quality labels using GPT-4 and train a two-stage inference system: a lightweight classifier handles high-confidence predictions, while only ~35% of ambiguous instances are escalated to GPT-4o. This architecture reduces inference costs by ~70% while achieving high precision (~0.90). Our pipeline demonstrates how targeted human-AI interaction, careful task formulation, and deployment-aware architecture design can unlock scalable solutions for novel responsible AI tasks.