Counterfactual Explanations for Delegation: Teaching Humans When NOT to Rely on AI

Jakubik et al. (2022) and Chen et al. (2023) reveal that traditional explanations, especially those focused on predicted outcomes, can actually increase overreliance—particularly when the AI is wrong. Building on this, the proposed research would develop and test "counterfactual explanations" that explicitly show users when, in similar past situations, the AI's recommendation would have been incorrect or harmful. Unlike current approaches that mostly justify AI's present recommendation, this method actively educates users about the AI's limits, fostering a more discerning, situation-aware delegation strategy. Such counterfactuals could be tailored to the user's expertise (as in Erlei et al., 2024) and even be dynamically surfaced when the AI's uncertainty is high (as in Shukla et al., 2024). The hypothesis is that this approach will help users avoid blind delegation, improving decision quality and trust calibration, especially in safety-critical or high-uncertainty domains.

References:

1. Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations. Valerie Chen, Q. Liao, Jennifer Wortman Vaughan, Gagan Bansal (2023). Proc. ACM Hum. Comput. Interact..
2. An Empirical Evaluation of Predicted Outcomes as Explanations in Human-AI Decision-Making. Johannes Jakubik, Jakob Schoffer, Vincent Hoge, Michael Vossing, Niklas Kuhl (2022). PKDD/ECML Workshops.
3. Understanding Choice Independence and Error Types in Human-AI Collaboration. Alexander Erlei, Abhinav Sharma, U. Gadiraju (2024). International Conference on Human Factors in Computing Systems.
4. Dynamic Explainability in AI for Neurological Disorders: An Adaptive Model for Transparent Decision-Making in Alzheimer's Disease Diagnosis. Anushka Shukla, Shivanshu Upadhyay, Priya Rachel Bachan, U. N. Bera, RV Kshirsagar, N. Nathani (2024). International Conference on Communication Systems and Network Technologies.