Learning When the Concept Shifts: Confounding, Invariance, and Dimension Reduction

• 成果类型: Article
• 署名作者: Dharmakeerthi, Kulunu; Hur, YoonHaeng; Liang, Tengyuan
• 署名单位: University of Chicago; University of Chicago
• 刊物名称: JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION
• ISSN/ISSBN: 0162-1459; 1537-274X
• DOI: 10.1080/01621459.2026.2656457
• 发表日期: 2026-04-03
• 页码: 1000-1012
• 关键词: Concept shift; distribution shift; Invariance; Representation learning; Structural Causal Model; Unobserved confounding; inference

摘要

Practitioners often face the challenge of deploying prediction models in new environments with shifted distributions of covariates and responses. With observational data, such shifts are often driven by unobserved confounding, and can in fact alter the concept of which model is best. This article studies distribution shifts in the domain adaptation problem with unobserved confounding. We postulate a linear structural causal model to account for endogeneity and unobserved confounding, and we leverage exogenous invariant covariate representations to cure concept shifts and improve target prediction. We propose a data-driven representation learning method that optimizes for a lower-dimensional linear subspace and a prediction model confined to that subspace. This method operates on a non-convex objective-that interpolates between predictability and stability-constrained to the Stiefel manifold, using an analog of projected gradient descent. We analyze the optimization landscape and prove that, provided sufficient regularization, nearly all local optima align with an invariant linear subspace resilient to distribution shifts. This method achieves a nearly ideal gap between target and source risk. We validate the method and theory with real-world datasets to illustrate the tradeoffs between predictability and stability. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.