Off-Policy Evaluation in Doubly Inhomogeneous Environments

• 成果类型: Article
• 署名作者: Bian, Zeyu; Shi, Chengchun; Qi, Zhengling; Wang, Lan
• 署名单位: University of Miami; University of London; London School Economics & Political Science; George Washington University
• 刊物名称: JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION
• ISSN/ISSBN: 0162-1459
• DOI: 10.1080/01621459.2024.2395593
• 发表日期: 2025
• 页码: 1102-1114
• 关键词: multiple-regression; time; guarantees; sepsis; series

摘要

This work aims to study off-policy evaluation (OPE) under scenarios where two key reinforcement learning (RL) assumptions-temporal stationarity and individual homogeneity are both violated. To handle the double inhomogeneities, we propose a class of latent factor models for the reward and transition functions, under which we develop a general OPE framework that consists of both model-based and model-free approaches. To our knowledge, this is the first article that develops statistically sound OPE methods in offline RL with double inhomogeneities. It contributes to a deeper understanding of OPE in environments, where standard RL assumptions are not met, and provides several practical approaches in these settings. We establish the theoretical properties of the proposed value estimators and empirically show that our approach outperforms state-of-the-art methods.