The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review

• 成果类型: Article; Early Access
• 署名作者: Su, Buxin; Zhang, Jiayao; Collina, Natalie; Yan, Yuling; Li, Didong; Cho, Kyunghyun; Fan, Jianqing; Roth, Aaron; Su, Weijie
• 署名单位: University of Pennsylvania; University of Pennsylvania; University of Wisconsin System; University of Wisconsin Madison; University of North Carolina; University of North Carolina Chapel Hill; New York University; Princeton University
• 刊物名称: JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION
• ISSN/ISSBN: 0162-1459
• DOI: 10.1080/01621459.2025.2510006
• 发表日期: 2025

摘要

We conducted an experiment during the review process of the 2023 International Conference on Machine Learning (ICML), asking authors with multiple submissions to rank their papers based on perceived quality. In total, we received 1342 rankings, each from a different author, covering 2592 submissions. In this article, we present an empirical analysis of how author-provided rankings could be leveraged to improve peer review processes at machine learning conferences. We focus on the Isotonic Mechanism, which calibrates raw review scores using the author-provided rankings. Our analysis shows that these ranking-calibrated scores outperform the raw review scores in estimating the ground truth expected review scores in terms of both squared and absolute error metrics. Furthermore, we propose several cautious, low-risk applications of the Isotonic Mechanism and author-provided rankings in peer review, including supporting senior area chairs in overseeing area chairs' recommendations, assisting in the selection of paper awards, and guiding the recruitment of emergency reviewers. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.