# What We Know About Using Non-Engagement Signals in Content Ranking

arXiv: https://arxiv.org/abs/2402.06831 · HTML: https://mlsanity.com/paper/2402.06831/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Feb 2024.

## Built on by (most cited first, 6 of 6)

- [AI Alignment with Changing and Influenceable Reward Functions](https://mlsanity.com/paper/2405.17713/) (2024)
- [AI and the Future of Digital Public Squares](https://mlsanity.com/paper/2412.09988/) (2024)
- [Value Alignment of Social Media Ranking Algorithms](https://mlsanity.com/paper/2509.14434/) (2025)
- [CTRL-Rec: Controlling Recommender Systems With Natural Language](https://mlsanity.com/paper/2510.12742/) (2025)
- [Large Language Models, and LLM-Based Agents, Should Be Used to Enhance the Digital Public Sphere](https://mlsanity.com/paper/2410.12123/) (2024)
- [Compass: Continuously Aligning Social Media Feeds via In-Situ Reflections](https://mlsanity.com/paper/2608.04274/) (2026)

## Abstract (arXiv)

Many online platforms predominantly rank items by predicted user engagement. We believe that there is much unrealized potential in including non-engagement signals, which can improve outcomes both for platforms and for society as a whole. Based on a daylong workshop with experts from industry and academia, we formulate a series of propositions and document each as best we can from public evidence, including quantitative results where possible. There is strong evidence that ranking by predicted engagement is effective in increasing user retention. However retention can be further increased by incorporating other signals, including item "quality" proxies and asking users what they want to see with "item-level" surveys. There is also evidence that "diverse engagement" is an effective quality signal. Ranking changes can alter the prevalence of self-reported experiences of various kinds (e.g. harassment) but seldom have large enough effects on attitude measures like user satisfaction, well-being, polarization etc. to be measured in typical experiments. User controls over ranking often have low usage rates, but when used they do correlate well with quality and item-level surveys. There was no strong evidence on the impact of transparency/explainability on retention. There is reason to believe that generative AI could be used to create better quality signals and enable new kinds of user controls.