The Situation Awareness Framework for Explainable AI (SAFE-AI) and Human Factors Considerations for XAI Systems

Abstract

Recent advances in artificial intelligence (AI) have drawn attention to the need for AI systems to beunderstandable to human users. The explainable AI (XAI) literature aims to enhance human under-standing and human-AI team performance by providing users with necessary information about AI sys-tem behavior. Simultaneously, the human factors literature has long addressed importantconsiderations that contribute to human performance, including how to determine human informa-tional needs, human workload, and human trust in autonomous systems. Drawing from the human fac-tors literature, we propose the Situation Awareness Framework for Explainable AI (SAFE-AI), a three-level framework for the development and evaluation of explanations about AI system behavior. Ourproposed levels of XAI are based on the informational needs of human users, which can be deter-mined using the levels of situation awareness (SA) framework from the human factors literature. Basedon our levels of XAI framework, we also suggest a method for assessing the effectiveness of XAI sys-tems. We further detail human workload considerations for determining the content and frequency ofexplanations as well as metrics that can be used to assess human workload. Finally, we discuss theimportance of appropriately calibrating user trust in AI systems through explanations along with othertrust-related considerations for XAI, and we detail metrics that can be used to evaluate user trust inthese systems.