# RADio -- Rank-Aware Divergence Metrics to Measure Normative Diversity in News Recommendations

arXiv: https://arxiv.org/abs/2209.13520 · HTML: https://mlsanity.com/paper/2209.13520/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 88 references that are in the map.
- First posted on arXiv 17 Sep 2022.

## Builds on (most cited first, 15 of 15)

- [BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation](https://mlsanity.com/paper/2101.11718/) (2021)
- [NPA: Neural News Recommendation with Personalized Attention](https://mlsanity.com/paper/1907.05559/) (2019)
- [Neural News Recommendation with Attentive Multi-View Learning](https://mlsanity.com/paper/1907.05576/) (2019)
- [Diversity and Inclusion Metrics in Subset Selection](https://mlsanity.com/paper/2002.03256/) (2020)
- [Recommenders with a mission: assessing diversity in newsrecommendations](https://mlsanity.com/paper/2012.10185/) (2020)
- [Personalized News Recommendation with Context Trees](https://mlsanity.com/paper/1303.0665/) (2013)
- [Equality of Voice: Towards Fair Representation in Crowdsourced Top-K Recommendations](https://mlsanity.com/paper/1811.08690/) (2018)
- [Assessing Viewpoint Diversity in Search Results Using Ranking Fairness Metrics](https://mlsanity.com/paper/2010.14531/) (2020)
- [Beyond Optimizing for Clicks: Incorporating Editorial Values in News Recommendation](https://mlsanity.com/paper/2004.09980/) (2020)
- [Improving Accuracy and Diversity in Matching of Recommendation with Diversified Preference Network](https://mlsanity.com/paper/2102.03787/) (2021)
- [Operationalizing Framing to Support Multiperspective Recommendations of Opinion Pieces](https://mlsanity.com/paper/2101.06141/) (2021)
- [Recent Advances in Diversified Recommendation](https://mlsanity.com/paper/1905.06589/) (2019)
- [A Hybrid Bandit Framework for Diversified Recommendation](https://mlsanity.com/paper/2012.13245/) (2020)
- [Diversity in News Recommendations](https://mlsanity.com/paper/2005.09495/) (2020)
- [Deep Dynamic Neural Network to trade-off between Accuracy and Diversity in a News Recommender System](https://mlsanity.com/paper/2103.08458/) (2021)

## Built on by (most cited first, 7 of 7)

- [EB-NeRD: A Large-Scale Dataset for News Recommendation](https://mlsanity.com/paper/2410.03432/) (2024)
- [RecSys Challenge 2024: Balancing Accuracy and Editorial Values in News Recommendations](https://mlsanity.com/paper/2409.20483/) (2024)
- [Improving and Evaluating the Detection of Fragmentation in News Recommendations with the Clustering of News Story Chains](https://mlsanity.com/paper/2309.06192/) (2023)
- [AMIR: Automated MisInformation Rebuttal -- A COVID-19 Vaccination Datasets based Recommendation System](https://mlsanity.com/paper/2310.19834/) (2023)
- [Navigating the Thin Line: Examining User Behavior in Search to Detect Engagement and Backfire Effects](https://mlsanity.com/paper/2401.11201/) (2024)
- [Constructing Political Coordinates: Aggregating Over the Opposition for Diverse News Recommendation](https://mlsanity.com/paper/2511.17574/) (2025)
- [Normative Alignment of Recommender Systems via Internal Label Shift](https://mlsanity.com/paper/2607.10915/) (2026)

## Abstract (arXiv)

In traditional recommender system literature, diversity is often seen as the opposite of similarity, and typically defined as the distance between identified topics, categories or word models. However, this is not expressive of the social science's interpretation of diversity, which accounts for a news organization's norms and values and which we here refer to as normative diversity. We introduce RADio, a versatile metrics framework to evaluate recommendations according to these normative goals. RADio introduces a rank-aware Jensen Shannon (JS) divergence. This combination accounts for (i) a user's decreasing propensity to observe items further down a list and (ii) full distributional shifts as opposed to point estimates. We evaluate RADio's ability to reflect five normative concepts in news recommendations on the Microsoft News Dataset and six (neural) recommendation algorithms, with the help of our metadata enrichment pipeline. We find that RADio provides insightful estimates that can potentially be used to inform news recommender system design.