MINING EVENTS WITH DECLASSIFIED DIPLOMATIC DOCUMENTS

• 成果类型: Article
• 署名作者: Gao, Yuanjun; Goetz, Jack; Connelly, Matthew; Mazumder, Rahul
• 署名单位: Columbia University; University of Michigan System; University of Michigan; Massachusetts Institute of Technology (MIT); Massachusetts Institute of Technology (MIT); Columbia University
• 刊物名称: ANNALS OF APPLIED STATISTICS
• ISSN/ISSBN: 1932-6157
• DOI: 10.1214/20-AOAS1344
• 发表日期: 2020
• 页码: 1699-1723
• 关键词: algorithm; selection

摘要

Since 1973, the U.S. State Department has been using electronic record systems to preserve classified communications. Recently, approximately 1.9 million of these records from 1973-77 have been made available by the U.S. National Archives. While some of these communication streams have periods witnessing an acceleration in the rate of transmission, others do not show any notable patterns in communication intensity. Given the sheer volume of these communications, far greater than what had been available until now, scholars need automated statistical techniques to identify the communications that warrant closer study. We develop a statistical framework that can identify from a large corpus of documents a handful that historians would consider more interesting. Our approach brings together techniques from nonparametric signal estimation, statistical hypothesis testing and modern optimization methods-leading to a set of tools that help us identify and analyze various geometrical aspects of the communication streams. Dominant periods of heightened activities, as identified through these methods, correspond well with historical events recognized by standard reference works on the 1970s.