General type-token distribution

• 成果类型: Article
• 署名作者: Hidaka, S.
• 署名单位: Japan Advanced Institute of Science & Technology (JAIST)
• 刊物名称: BIOMETRIKA
• ISSN/ISSBN: 0006-3444
• DOI: 10.1093/biomet/asu035
• 发表日期: 2014
• 页码: 9991002
• 关键词: NUMBER; population

摘要

We consider the problem of estimating the number of types in a corpus using the number of types observed in a sample of tokens from that corpus. We derive exact and asymptotic distributions for the number of observed types, conditioned on the number of tokens and the latent type distribution. We use the asymptotic distributions to derive an estimator of the latent number of types and validate this estimator numerically.