A Taxonomy of Evaluation Methods for Information Systems Artifacts

• 成果类型: Article
• 署名作者: Prat, Nicolas; Comyn-Wattiau, Isabelle; Akoka, Jacky
• 署名单位: ESSEC Business School; heSam Universite; Conservatoire National Arts & Metiers (CNAM); ESSEC Business School; heSam Universite; Conservatoire National Arts & Metiers (CNAM); IMT - Institut Mines-Telecom; Institut Mines-Telecom Business School
• 刊物名称: JOURNAL OF MANAGEMENT INFORMATION SYSTEMS
• ISSN/ISSBN: 0742-1222
• DOI: 10.1080/07421222.2015.1099390
• 发表日期: 2015
• 页码: 229-267
• 关键词: design science research; TECHNOLOGY; MODEL

摘要

Artifacts, such as software systems, pervade organizations and society. In the field of information systems (IS) they form the core of research. The evaluation of IS artifacts thus represents a major issue. Although IS research paradigms are increasingly intertwined, building and evaluating artifacts has traditionally been the purview of design science research (DSR). DSR in IS has not reached maturity yet. This is particularly true of artifact evaluation. This paper investigates the what and the how of IS artifact evaluation: what are the objects and criteria of evaluation, the methods for evaluating the criteria, and the relationships between the what and the how of evaluation? To answer these questions, we develop a taxonomy of evaluation methods for IS artifacts. With this taxonomy, we analyze IS artifact evaluation practice, as reflected by ten years of DSR publications in the basket of journals of the Association for Information Systems (AIS). This research brings to light important relationships between the dimensions of IS artifact evaluation, and identifies seven typical evaluation patterns: demonstration; simulation-and metric-based benchmarking of artifacts; practice-based evaluation of effectiveness; simulation- and metric-based absolute evaluation of artifacts; practice-based evaluation of usefulness or ease of use; laboratory, student-based evaluation of usefulness; and algorithmic complexity analysis. This study also reveals a focus of artifact evaluation practice on a few criteria. Beyond immediate usefulness, IS researchers are urged to investigate ways of evaluating the long-term organizational impact and the societal impact of artifacts.