Participatory Auditing Framework: Co-Creating Evaluation Criteria with Stakeholder Communities for Human-Centered LLM Audits

The current LLM evaluation landscape suffers from an "evaluation crisis" characterized by a disconnect between automated metrics and real-world human impact. This research introduces a participatory auditing framework where affected communities co-design evaluation protocols from the ground up. The approach combines participatory design workshops with technical audits, recruiting stakeholder groups such as Bengali dialect speakers, archivists, and healthcare workers to define audit priorities, co-develop metrics reflecting their lived experiences, and validate findings through iterative feedback loops. Unlike existing methods relying on expert or LLM-generated criteria, this framework centers human-defined success criteria, challenges universalist evaluation norms, and documents ethical tensions arising from stakeholder-technical conflicts. By operationalizing stakeholder authority and pairing qualitative co-design with quantitative audits, the framework aims to democratize audit governance, uncover blind spots in conventional benchmarks, and build trust in LLM deployments across diverse domains.

References:

1. Dialectal Bias in Bengali: An Evaluation of Multilingual Large Language Models Across Cultural Variations. Azmine Toushik Wasi, Raima Islam, Mst Rafia Islam, Farig Sadeque, Taki Hasan Rafi, Dong-Kyu Chae (2025). The Web Conference.
2. ArchiveGPT: A human-centered evaluation of using a vision language model for image cataloguing. Line Abele, Gerrit Anders, Tolgahan Aydın, Jürgen Buder, Helen Fischer, Dominik Kimmel, Markus Huff (2025). arXiv.org.