Bayesian Optimization for Branching and Nested Hyperparameters in Deep Learning

• 成果类型: Article; Early Access
• 署名作者: Zhang, Jiazhao; Lin, Chung-Ching; Hung, Ying
• 署名单位: Rutgers University System; Rutgers University New Brunswick; Microsoft
• 刊物名称: JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION
• ISSN/ISSBN: 0162-1459; 1537-274X
• DOI: 10.1080/01621459.2026.2625450
• 发表日期: 2026-04-21
• 关键词: bayesian optimization; Deep learning; Experimental design; Gaussian process; efficient global optimization; computer experiments; variables; models

摘要

Hyperparameter optimization plays a crucial role in the success of neural networks as hyperparameters directly control the behavior and performance of the training algorithms. To obtain efficient tuning, Bayesian optimization based on Gaussian process is widely used. Despite numerous applications in deep learning, the existing methods rely on a convenient but restrictive assumption that the tuning parameters are independent of each other. However, tuning parameters with conditional dependence are common in practice. In this article, we focus on two types of them: branching and nested parameters. Nested parameters refer to those tuning parameters that exist only within a particular setting of another tuning parameter, and a parameter that contains other nested parameters is referred to as a branching parameter. To capture the conditional dependence between branching and nested parameters, a unified Bayesian optimization framework is proposed. The sufficient conditions are rigorously derived to guarantee the validity of the kernel function, and the asymptotic convergence of the proposed optimization framework is proven under the continuum-armed-bandit setting. The new model, which accounts for the dependent structure among input variables through a new kernel function, provides a higher prediction accuracy and a better optimization efficiency in simulations and real data applications in neural networks. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.