Turning Words Into Numbers: Assessing Work Attitudes Using Natural Language Processing

• 成果类型: Article
• 署名作者: Speer, Andrew B.; Perrotta, James; Tenbrink, Andrew P.; Wegmeyer, Lauren J.; Delacruz, Angie Y.; Bowker, Jenna
• 署名单位: Wayne State University
• 刊物名称: JOURNAL OF APPLIED PSYCHOLOGY
• ISSN/ISSBN: 0021-9010
• DOI: 10.1037/apl0001061
• 发表日期: 2023
• 页码: 1027-1045
• 关键词: natural language processing; Machine Learning; narrative comments; WORK ATTITUDES; work perceptions

摘要

Researchers and practitioners are often interested in assessing employee attitudes and work perceptions. Although such perceptions are typically measured using Likert surveys or some other closed-end numerical rating format, many organizations also have access to large amounts of qualitative employee data. For example, open-ended comments from employee surveys allow workers to provide rich and contextualized perspectives about work. Unfortunately, there are practical challenges when trying to understand employee perceptions from qualitative data. Given this, the present study investigated whether natural language processing (NLP) algorithms could be developed to automatically score employee comments according to important work attitudes and perceptions. Using a large sample of employees, algorithms were developed to translate text into scores that reflect what comments were about (theme scores) and how positively targeted constructs were described (valence scores) for 28 work constructs. The resulting algorithms and scores are labeled the Text-Based Attitude and Perception Scoring (TAPS) dictionaries, which are made publicly available and were built using a mix of count-based scoring and transformer neural networks. The psychometric properties of the TAPS scores were then investigated. Results showed that theme scores differentiated responses based on their likelihood to discuss specific constructs. Additionally, valence scores exhibited strong evidence of reliability and validity, particularly, when analyzed on text responses that were more relevant to the construct of interest. This suggests that researchers and practitioners should explicitly design text prompts to elicit construct-related information if they wish to accurately assess work attitudes and perceptions via NLP.