CORRECTING MEASUREMENT ERROR IN REGRESSION MODELS WITH VARIABLES CONSTRUCTED FROM AGGREGATED OUTPUT OF DATA MINING MODELS

• 成果类型: Article
• 署名作者: Qiao, Mengke; Huang, Ke-Wei
• 署名单位: University of Alabama System; University of Alabama Tuscaloosa; National University of Singapore
• 刊物名称: MIS QUARTERLY
• ISSN/ISSBN: 0276-7783
• DOI: 10.25300/MISQ/2024/18026
• 发表日期: 2025-03
• 页码: 29-60
• 关键词: data mining; aggregate-level regression model; measurement error; Central Limit Theorem; law of large numbers; Social media; misclassification; responses; BUSINESS; support; Blogs

摘要

The burgeoning interest in data mining has catalyzed a proliferation of innovative techniques in extracting useful information from unstructured data sources, such as text and images in social sciences. One typical research design involves a two-stage process. In the first stage, researchers apply the classification algorithm to predict an individual-level categorical variable. In the second stage, researchers aggregate the predicted values to construct a group-level variable for further regression analysis. For example, text classification has been applied to classify whether a review is positive or negative. The predicted review sentiment is aggregated at the product level as a focal independent variable in a regression model to examine the impact of the average review sentiment on product sales. Since the first-stage classification will inevitably have errors, the aggregated variable may suffer from a measurement error in the regression analysis. Our study attempts to systematically investigate the theoretical properties of the estimation bias and introduce solutions rooted in theory to mitigate the issue of measurement error. We propose one exact solution and two approximated solutions based on the central limit theorem (CLT) and the law of large numbers (LLN), respectively. Our theoretical analysis and experimentation confirm that the consistency of regression estimators can be recovered across all examined scenarios and the approximated solutions offer a significantly reduced computational complexity compared to the exact solution. We also provide heuristic guidelines to choose one of three solutions.