Information rate of meaningful communication

• 成果类型: Article
• 署名作者: Sivan, Doron; Tsodyks, Misha
• 署名单位: Weizmann Institute of Science; Institute for Advanced Study - USA
• 刊物名称: PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA
• ISSN/ISSBN: 0027-11947
• DOI: 10.1073/pnas.2502353122
• 发表日期: 2025-06-24
• 关键词: entropy

摘要

In Shannon's seminal paper, the entropy of printed English, treated as a stationary stochastic process, was estimated to be roughly 1 bit per character. However, considered as a means of communication, language differs considerably from its printed form: i) the units of information are not characters or even words but clauses, i.e., shortest meaningful parts of speech; and ii) what is transmitted is principally the meaning of what is being said or written, while the precise phrasing that was used to communicate the meaning is typically ignored. In this study, we show that one can leverage recently developed large language models to quantify information communicated in meaningful narratives in terms of bits of meaning per clause.