Policy-Floored RLHF for Moderation: Guarantees Against Regressions Below Rule-Based Baselines

Inspired by Huang et al.’s (2024) Physics-Enhanced RLHF, which guarantees driving policies never underperform a physics-based baseline, we propose Policy-Floored RLHF for moderation assistants. The floor is a platform/community rule set—augmented with culturally specific norms as emphasized by Shahid (2024)—that defines minimally acceptable moderation behavior. The AI learns from human feedback (e.g., moderator decisions, appeals outcomes) but is constrained never to regress below the baseline policy. Minimal intervention mechanisms reduce human burden while retaining veto powers on edge cases. We test this with multimodal brand safety data from Levi et al. (2025), assessing robustness and non-regression in out-of-distribution shifts. This diverges from conventional RLHF in moderation by offering a formal safety guarantee and by explicitly localizing policy floors to communities (addressing Shahid’s finding that one-size-fits-all norms lead to moral policing and missed local harms). The payoff is a trustworthy, updatable assistant that evolves with human feedback without violating the guardrails communities depend on.

References:

1. Human-AI Collaboration to Facilitate Culturally-Aware Content Moderation. Farhana Shahid (2024). CSCW Companion.
2. Trustworthy Human-AI Collaboration: Reinforcement Learning with Human Feedback and Physics Knowledge for Safe Autonomous Driving. Zilin Huang, Zihao Sheng, Sikai Chen (2024). arXiv.org.
3. AI vs. Human Moderators: A Comparative Evaluation of Multimodal LLMs in Content Moderation for Brand Safety. Adi Levi, Or Levi, Sardhendu Mishra, Jonathan Morra (2025). arXiv.org.