Counterintuitive Risks: Investigating and Mitigating Automation Complacency in AI-Driven Oversight Systems

Adepoju & Chinonyerem (2025) and others report increased efficiency and anomaly detection with AI-powered oversight, but also highlight the risk of “automation complacency.” This phenomenon—where humans defer excessively to AI, even in the face of questionable outputs—remains underexplored in the governance literature. This research would empirically investigate the conditions under which automation complacency emerges (e.g., in policy compliance, fraud detection, etc.), and design interventions (such as adversarial “red team” audits, human-in-the-loop protocols, or periodic “trust calibration” exercises). By challenging the assumption that more AI oversight always equals better oversight, this work could lead to more robust, hybrid audit systems, ultimately reducing the risk of undetected failures due to misplaced trust in AI.

References:

1. ADVANCING GOOD GOVERNANCE THROUGH AI-POWERED OVERSIGHT IN THE UNITED STATES: RISKS AND OPPORTUNITIES FOR PUBLIC INSTITUTIONS. ADEWUMI SUNDAY ADEPOJU, CONFIDENCE ADIMCHI CHINONYEREM (2025). Journal of African Advancement and Sustainability Studies.