To explain or not to explain? - A Case Study of Artificial Intelligence Explainability in Clinical Decision Support Systems
Abstract
The role of explainability in clinical decision support systems (CDSS) based on artificial intelligence (AI) raises important medical and ethical questions. Some see explainability for CDSS as a necessity, others caution against it or against certain implementations of it. This leads to considerable uncertainty and leaves the usability and utility of explainability in AI-based CDSS systems under controversy.
This paper presents a review of the key arguments in favor and against explainability for CDSS by applying them to a practical use case. We performed a qualitative case study approach combined with a normative analysis using socio-technical scenarios. Our paper builds on the interdisciplinary assessment (via the Z-Inspection® process) of a black-box AI CDSS used in the emergency call setting to identify patients with life-threatening cardiac arrest. Specifically, the assessment informed the development of the two socio-technical scenarios presented in this paper to explore the impact of foregoing or adding explainability for the CDSS in question. Our analytical framework is composed of three layers: technical considerations, human factors, and the designated system role in decision-making.
With regards to the first scenario without implemented explainability, we highlight the importance of a robust validation to maximize performance on a broad spectrum of patients and to compensate for the lack of explainability. Such validation studies may, however, prove challenging since the users involved in the validation phase need to rely on a (not yet) validated system to inform their decision-making. If they fail to implement the system as instructed, the validation phase will not deliver the expected results. Finally, in this case study, the system if used autonomously performs better than the users, suggesting that the role of the system may, in fact, exceed mere CDSS and assume a more prominent role in decision-making.
With regards to the second scenario with implemented explainability, we find certain advantages for the implementation of explainability but identify a major challenge regarding the technical validity of explainability for black-box AI systems. Under the assumption of validated explainability, we foresee increased trust of the users in the systems, but identify the need to include them in the decision process. In contrast to the first scenario, provided explanations allow the system to serve as CDSS since the users have the means to understand each given prediction. Ergo the presence of explainability has a strong influence on what clinical setting a given system is applicable to.
We conclude that whether explainability adds value to CDSS depends on technical feasibility, the context in which the system is implemented (e.g., the time criticality of the decision making), the designated role in the decision-making process (algorithm-based, -driven, or -determined), and the key user group(s). As such, the relevance of explainability cannot be finally answered on a theoretical level. Each system developed for the clinical setting will require an individualized assessment of explainability needs.
Research Areas
