Researchers' data analysis choices: an excess of false positives?

• 成果类型: Article
• 署名作者: Ohlson, James A.
• 署名单位: Hong Kong Polytechnic University
• 刊物名称: REVIEW OF ACCOUNTING STUDIES
• ISSN/ISSBN: 1380-6653
• DOI: 10.1007/s11142-021-09620-w
• 发表日期: 2022
• 页码: 649-667
• 关键词: accounting research

摘要

This paper examines commonly applied methods of data analysis. Predicated on these methods, the main issue pertains to the plausibility of the studies' end products, that is, their conclusions. I argue that the methods chosen often lead to unwarranted conclusions: the data analyses chosen tend to produce looked-for null rejections even though the null may be much more plausible on prior grounds. Two aspects of data analyses applied cause obvious problems. First, researchers tend to dismiss preliminary findings when the findings contradict the expected outcome of the research question (the screen-picking issue). Second, researchers rarely acknowledge that small p-values should be expected when the number of observations runs into the tens of thousands (the large N issue). This obviously enhances the chance for a null rejection even if the null hypothesis holds for all practical purposes. The discussion elaborates on these two aspects to explain why researchers generally avoid trying to mitigate false positives via supplementary data analyses. In particular, for no apparent good reasons, most research studiously avoids the use of hold-out samples. An additional topic in this paper concerns the dysfunctional consequences of the standard (A-journal) publication process, which tends to buttress the use of research methods prone to false or unwarranted null-rejections.