Privacy for free in the overparameterized regime

• 成果类型: Article
• 署名作者: Bombari, Simone; Mondelli, Marco
• 署名单位: Institute of Science & Technology - Austria
• 刊物名称: PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA
• ISSN/ISSBN: 0027-11045
• DOI: 10.1073/pnas.2423072122
• 发表日期: 2025-04-11

摘要

Differentially private gradient descent (DP-GD) is a popular algorithm to train deep learning models with provable guarantees on the privacy of the training data. In the last decade, the problem of understanding its performance cost with respect to standard GD has received remarkable attention from the research community, which has led to upper bounds on the excess population risk RP in different learning settings. However, such bounds typically degrade with overparameterization, i.e., as the number of parameters p gets larger than the number of training samples n-a regime which is ubiquitous in current deep-learning practice. As a result, the lack of theoretical insights leaves practitioners without clear guidance, leading some to reduce the effective number of trainable parameters to improve performance, while others use larger models to achieve better results through scale. In this work, we show that in the popular random features model with quadratic loss, for any sufficiently large p, privacy can be obtained for free, i.e., |RP| = o(1), not only when the privacy parameter E has constant order but also in the strongly private setting E = o(1). This challenges the common wisdom that overparameterization inherently hinders performance in private learning.