From Symbolic to Substantive: Auditing AI Policy Outcomes with Cross-Sectoral Impact Metrics

While Raji (2022) and others highlight the challenges of moving from algorithmic audits to actual accountability, most existing work focuses on audit processes or organizational responses. This idea shifts the focus to impact metrics—specifically, how audits tangibly change outcomes in domains like healthcare, finance, or arbitration. Drawing inspiration from regulatory impact assessment in other industries (as Raji suggests) and integrating the sector-specific standards discussed by Dickey et al. (2024) and Hayati et al. (2025), this project would develop, pilot, and validate a set of cross-sectoral metrics for audit effectiveness. By requiring audits to demonstrate improvements in fairness, transparency, or error reduction (rather than mere procedural compliance), this framework could transform oversight from symbolic ritual to a driver of real change. Such a model could also help address the policy-practice gap noted by DeFranco & Biersmith (2024). The impact? A more mature, empirically grounded approach to holding AI deployments genuinely accountable.

References:

1. From Algorithmic Audits to Actual Accountability: Overcoming Practical Roadblocks on the Path to Meaningful Audit Interventions for AI Governance. Inioluwa Deborah Raji (2022). AAAI/ACM Conference on AI, Ethics, and Society.
2. Information Standards and Guidelines on AI: Ethical Concerns, Trustworthiness, Quality Assessment, and Human Oversight. Timothy J. Dickey, Brian Dobreski, M. Hlava, Brady Lund, Mark Needleman, M. L. Zeng (2024). Proceedings of the Association for Information Science and Technology.
3. Ethical standards in arbitration practice in Indonesia: Challenges and strengthening of oversight. Vivi Hayati, Muhammad Iqbal, Natasya Masthura, Saiful Anwar, Cut Hasmiyati (2025). Jurnal Geuthèë.
4. Assessing the State of AI Policy. Joanna F. DeFranco, Luke Biersmith (2024). arXiv.org.