Human Oversight, Trust, and Accountability in AI-Assisted Decision Making: A Systematic Literature Review
Abstract
Artificial intelligence (AI) is increasingly embedded in organizational and professional decision processes, where it is used to generate recommendations, predictions, classifications, and prioritizations that shape human judgment. Although prior research has examined related issues such as trust in AI, algorithm aversion, automation bias, human-in-the-loop oversight, and accountability in algorithmic systems, these streams remain fragmented. As a result, scholarship still lacks an integrated understanding of how AI characteristics, task conditions, user factors, and organizational arrangements jointly shape reliance on AI recommendations, the quality of human oversight, and downstream accountability outcomes. This systematic literature review addresses that gap by synthesizing the literature on human oversight, trust, and accountability in AI-assisted decision making.
Following PRISMA 2020 guidelines, this review systematically identifies and analyzes studies examining AI-assisted or algorithm-assisted decision processes in which a human decision maker remains involved in evaluating, accepting, modifying, or overriding AI outputs. The review focuses on studies addressing one or more of the following themes: trust in AI, reliance and override behavior, human oversight and monitoring, and accountability or responsibility in AI-supported decisions. Studies were identified through structured searches of major multidisciplinary, business, psychology, and technology databases, screened against predefined inclusion and exclusion criteria, and synthesized using a narrative review approach supported by descriptive analysis.
The review develops an integrated framework that connects antecedents of AI reliance and oversight—including AI-related, human-related, task-related, and organizational factors—to mechanisms such as trust calibration, cognitive offloading, and responsibility diffusion. It then links these mechanisms to behavioral outcomes such as acceptance, verification, override, and oversight quality, as well as to accountability outcomes including responsibility clarity, traceability, and decision consequences. By integrating fragmented streams of research, this review contributes a process perspective on AI-assisted decision making and identifies a future research agenda for human oversight, governance, and accountability in AI-enabled organizational contexts.
Following PRISMA 2020 guidelines, this review systematically identifies and analyzes studies examining AI-assisted or algorithm-assisted decision processes in which a human decision maker remains involved in evaluating, accepting, modifying, or overriding AI outputs. The review focuses on studies addressing one or more of the following themes: trust in AI, reliance and override behavior, human oversight and monitoring, and accountability or responsibility in AI-supported decisions. Studies were identified through structured searches of major multidisciplinary, business, psychology, and technology databases, screened against predefined inclusion and exclusion criteria, and synthesized using a narrative review approach supported by descriptive analysis.
The review develops an integrated framework that connects antecedents of AI reliance and oversight—including AI-related, human-related, task-related, and organizational factors—to mechanisms such as trust calibration, cognitive offloading, and responsibility diffusion. It then links these mechanisms to behavioral outcomes such as acceptance, verification, override, and oversight quality, as well as to accountability outcomes including responsibility clarity, traceability, and decision consequences. By integrating fragmented streams of research, this review contributes a process perspective on AI-assisted decision making and identifies a future research agenda for human oversight, governance, and accountability in AI-enabled organizational contexts.
