From Lexicons to Large Language Models: A Holistic Evaluation of Psychometric Text Analysis in Social Science Research

• 成果类型: Article; Early Access
• 署名作者: Mousavi, Reza; Kitchens, Brent; Oliver, Abbie Griffith; Abbasi, Ahmed
• 署名单位: University of Virginia; University of Notre Dame
• 刊物名称: INFORMATION SYSTEMS RESEARCH
• ISSN/ISSBN: 1047-7047; 1526-5536
• DOI: 10.1287/isre.2024.1143
• 发表日期: 2026-04-22
• 关键词: psychometrics; natural language processing (NLP); transformers; large language models (LLMs); fairness; cognitive-affective processes; dual-process theory (DPT); prompting techniques; quality-of-life; EMOTIONAL INTELLIGENCE; media

摘要

Extracting psychological constructs from text is increasingly essential for social science researchers who study attitudes, perceptions, and traits across digital communication. This paper offers the first holistic comparison of four major approaches used for this purpose-lexicons, custom-built machine learning models, fine-tuned masked language models, and large language models (LLMs). We evaluate these paradigms across multiple performance dimensions and integrate insights from dual-process theory (DPT) to understand how cognitive and affective processes shape annotation quality. Our results show that LLMs match or exceed the performance of established supervised methods while producing more consistent and fair predictions, all without requiring specialized natural language processing (NLP) expertise or extensive labeled data. Using DPT, we further demonstrate that human annotation accuracy depends on the alignment between an annotator's cognitive or emotional abilities and the psychological construct being coded. Misalignment reduces annotation quality and weakens downstream models. Drawing on this insight, we introduce a cognitive-affective prompting strategy for LLMs that emulates these human strengths, yielding performance gains beyond state-of-the-art prompting methods. Together, our findings offer practical guidance for method selection, illuminate how psychological constructs can be measured more reliably from text, and advance the design of psychometric NLP tools in social science research. To support immediate application, we also provide a researcher-friendly cookbook (in the Online Appendix) for using LLMs to annotate text data in practice.