Beyond the Block: Designing Conversational Content Moderation for Generative AI

Building directly on Gao et al.'s (2025) finding that users experience significant frustration with content moderation failures and the lack of post-moderation support, this research idea challenges the fundamental "block-and-notify" paradigm that dominates current generative AI systems. Instead of a one-way enforcement action where users are met with a cryptic error message like "I Cannot Write This Because It Violates Our Content Policy," we propose designing a two-way conversational interface.

The core innovation is to shift from punitive moderation to collaborative safety. When a user's prompt triggers a policy violation, instead of simply blocking the request, the system would engage in a brief, targeted dialogue. For example, if a user tries to write about historical conflicts and the system flags it for "violence," the conversational agent could ask: "Are you writing fictional violence, or are you creating educational content about historical events?" The user's response would then either allow the generation to proceed or provide crucial context that refines the moderation decision in real-time.

This approach directly addresses the gaps identified by Gao et al. by providing the "practical implementation" details and user agency currently missing from moderation policies. It also extends the work of Anigboro et al. (2024) on identity-related speech suppression. If content about disability is incorrectly flagged for "self-harm," a conversational system could ask clarifying questions to distinguish between harmful content and discussions about lived experiences, thereby reducing stereotypical misclassification. Similarly, it aligns with Shahid's (2024) call for culturally-aware moderation by allowing the system to query users about context that might be lost in a one-size-fits-all policy, preventing the "moral policing of culturally appropriate content."

The novelty lies in treating a moderation failure not as a system error to be hidden, but as a teachable moment for both the user and the AI. This research would involve designing and prototyping these conversational interfaces, then conducting user studies to measure their impact on user frustration, trust, and ability to successfully generate desired content. By fundamentally reimagining the user's role in the moderation process from a passive rule-breaker to an active collaborator, this work has the potential to create more transparent, equitable, and user-friendly generative AI systems.

References:

1. "I Cannot Write This Because It Violates Our Content Policy": Understanding Content Moderation Policies and User Experiences in Generative AI Products. Lan Gao, Oscar Chen, Rachel Lee, Nick Feamster, Chenhao Tan, M. Chetty (2025). arXiv.org.
2. Identity-related Speech Suppression in Generative AI Content Moderation. Oghenefejiro Isaacs Anigboro, Charlie M. Crawford, Danaë Metaxa, Sorelle A. Friedler (2024). arXiv.org.
3. Human-AI Collaboration to Facilitate Culturally-Aware Content Moderation. Farhana Shahid (2024). CSCW Companion.