A Family of Hyperparameter Estimators Linking EB and SURE for Kernel-Based Regularization Methods

• 成果类型: Article
• 署名作者: Zhang, Meng; Chen, Tianshi; Mu, Biqiang
• 署名单位: Chinese Academy of Sciences; Academy of Mathematics & System Sciences, CAS; The Chinese University of Hong Kong, Shenzhen; The Chinese University of Hong Kong, Shenzhen; Shenzhen Research Institute of Big Data; The Chinese University of Hong Kong, Shenzhen
• 刊物名称: IEEE TRANSACTIONS ON AUTOMATIC CONTROL
• ISSN/ISSBN: 0018-9286
• DOI: 10.1109/TAC.2024.3416162
• 发表日期: 2024
• 页码: 8674-8689
• 关键词: Kernel; estimation; CONVERGENCE; Complexity theory; Tuning; Monte Carlo methods; indexes; empirical Bayes (EB); kernel-based regularization methods (KRMs); Stein's unbiased risk estimator (SURE); weighted mean squared error

摘要

Hyperparameter estimation is a critical aspect of kernel-based regularization methods (KRMs), alongside kernel design. Empirical Bayes (EB) and Stein's unbiased risk estimator (SURE) are two widely used hyperparameter estimators for tuning the unknown hyperparameters associated with the kernel matrix of KRMs. However, EB and SURE exhibit different characteristics in both theory and practice. Theoretically, SURE is asymptotically optimal in terms of minimizing the mean squared error (MSE), whereas EB generally is not. However, practical evidence suggests that EB is often more accurate and robust than SURE, especially when the regression matrix is ill-conditioned. Therefore, this article aims to deepen our understanding of these two estimators in a unified manner. First, we construct a family of hyperparameter estimators that encompasses both EB and SURE by introducing an index. We then explain the loss function of this family as a tradeoff between data fit and model complexity for regularized least squares estimators, using least squares estimators as a reference. Second, we establish the convergence and rate of convergence of this family and further demonstrate that it is asymptotically optimal in some weighted MSE for a specific kernel matrix. Finally, Monte-Carlo simulations indicate the existence of other hyperparameter estimators in this family that outperform both EB and SURE methods for certain datasets.