# Tail-Influence Sampling for CVaR Policy Evaluation

arXiv: https://arxiv.org/abs/2609.38096 · HTML: https://mlsanity.com/paper/2609.38096/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 41 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Builds on (most cited first, 24 of 24)

- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs](https://mlsanity.com/paper/2503.01743/) (2025)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [Active Testing: Sample-Efficient Model Evaluation](https://mlsanity.com/paper/2103.05331/) (2021)
- [Policy Gradient for Coherent Risk Measures](https://mlsanity.com/paper/1502.03919/) (2015)
- [An Analysis of Categorical Distributional Reinforcement Learning](https://mlsanity.com/paper/1802.08163/) (2018)
- [Universal Off-Policy Evaluation](https://mlsanity.com/paper/2104.12820/) (2021)
- [Active Evaluation Acquisition for Efficient LLM Benchmarking](https://mlsanity.com/paper/2410.05952/) (2024)
- [Distributional Offline Policy Evaluation with Predictive Error Guarantees](https://mlsanity.com/paper/2302.09456/) (2023)
- [ReVar: Strengthening Policy Evaluation via Reduced Variance Sampling](https://mlsanity.com/paper/2203.04510/) (2022)
- [Clip-OGD: An Experimental Design for Adaptive Neyman Allocation in Sequential Experiments](https://mlsanity.com/paper/2305.17187/) (2023)
- [Active Testing: An Efficient and Robust Framework for Estimating Accuracy](https://mlsanity.com/paper/1807.00493/) (2018)
- [Uncertainty Quantification and Exploration for Reinforcement Learning](https://mlsanity.com/paper/1910.05471/) (2019)
- [Estimation and Inference in Distributional Reinforcement Learning](https://mlsanity.com/paper/2309.17262/) (2023)
- [Statistical Efficiency of Distributional Temporal Difference Learning and Freedman's Inequality in Hilbert Spaces](https://mlsanity.com/paper/2403.05811/) (2024)
- [Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model](https://mlsanity.com/paper/2402.07598/) (2024)
- [Near-Optimal Sample Complexity for Iterated CVaR Reinforcement Learning with a Generative Model](https://mlsanity.com/paper/2503.08934/) (2025)
- [Logging Policy Design for Off-Policy Evaluation](https://mlsanity.com/paper/2605.15108/) (2026)
- [Distributional Off-policy Evaluation with Bellman Residual Minimization](https://mlsanity.com/paper/2402.01900/) (2024)
- [SaVeR: Optimal Data Collection Strategy for Safe Policy Evaluation in Tabular MDP](https://mlsanity.com/paper/2406.02165/) (2024)
- [Statistical Efficiency and Inference of Quantile Distributional Reinforcement Learning](https://mlsanity.com/paper/2607.08444/) (2026)
- [Online Inference in Distributional Temporal-Difference Learning](https://mlsanity.com/paper/2608.14408/) (2026)

## Abstract (arXiv)

Policies with similar mean returns can differ sharply in rare failures, yet estimating lower-tail conditional value-at-risk (CVaR) accurately can require many costly rollouts. When different conditional components of a stochastic workflow can be queried separately, we ask how to allocate a fixed evaluation budget to estimate a fixed policy's CVaR most accurately. We derive a tail influence for each queryable conditional law that aggregates how its uncertainty affects CVaR across every Bellman reuse. Its variance yields the fixed-design efficiency bound and the oracle Neyman allocation. Tail-Influence Sampling (TIS) estimates these influence scales from a pilot model and reallocates fresh queries toward kernels that matter most for the tail; a visitation-anchored variant protects against pilot underallocation. Under fixed dimension and a positive quantile margin, TIS attains oracle asymptotic variance and first-order MSE including pilot cost, while the anchored variant is within a factor two of the oracle. We also characterize an exact-grid regime in which tail- and mean-optimal allocations coincide. On CliffWalking, TIS reduces MSE by 41% versus learned occupancy and 76% versus complete rollouts at the same charged transition budget. In frozen language-model review workflows, anchored TIS beats an equally regularized mean-influence blend in 23 of 24 MMLU-Pro settings and reaches 2.4-3.4× lower MSE than rollouts on six-call FinQA reviews.