Distributional Off-Policy Evaluation with Deep Quantile Process Regression

• 成果类型: Article
• 署名作者: Kuang, Qi; Wang, Chao; Jiao, Yuling; Zhou, Fan
• 署名单位: Jiangxi University of Finance & Economics; Jiangxi University of Finance & Economics; Shanghai University of Finance & Economics; Wuhan University; Wuhan University
• 刊物名称: JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION
• ISSN/ISSBN: 0162-1459; 1537-274X
• DOI: 10.1080/01621459.2026.2671449
• 发表日期: 2026-04-03
• 页码: 1025-1036
• 关键词: Deep quantile process regression; Deep ReLU networks; Distributional off-policy evaluation; Distributional reinforcement learning; Sample Complexity; neural-networks; inference

摘要

This article investigates the off-policy evaluation (OPE) problem from a distributional perspective. Rather than focusing solely on the expectation of the total return, as in most existing OPE methods, we aim to estimate the entire return distribution. To this end, we introduce a quantile-based approach for OPE using deep quantile process regression, presenting a novel algorithm called Deep Quantile Process regression-based Off-Policy Evaluation (DQPOPE). We provide new theoretical insights into the deep quantile process regression technique, extending existing approaches from estimating discrete quantiles to estimating a continuous quantile function. A key contribution of our work is the rigorous sample complexity analysis for distributional OPE with deep neural networks, bridging theoretical analysis with practical algorithmic implementations. We show that DQPOPE achieves statistical advantages by estimating the full return distribution using the same sample size required to estimate a single policy value using conventional methods. Empirical studies further show that DQPOPE provides significantly more precise and robust policy value estimates than standard methods, thereby enhancing the practical applicability and effectiveness of distributional reinforcement learning approaches. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.