Delegation Rules That Learn: Generative Stress-Testing and Repair for OOD Robustness

Conditional delegation (Lai et al., 2022) relies on humans delineating “trust regions” for models, but these rules can silently fail under OOD shifts. We introduce a continuous integration pipeline for moderation: a generative module synthesizes realistic, near-boundary, and OOD content conditioned on current rules; a verifier measures where the rules break (e.g., false leniency against newly emerging slurs or multimodal cues); and a repair assistant suggests minimal edits to rules, with human oversight via visual analytics (Vaidya et al., 2021). We incorporate design insights from Li and Chau (2023) to present high-value information cues under time constraints. Leveraging failure modes from Levi et al. (2025), we evaluate whether stress-testing prevents brand-safety regressions with new trends. This departs from static delegation paradigms by making rules adaptive and testable—catching unexpected model behavior before it hits production. The impact is a safer, more resilient human–AI collaboration that anticipates distribution shift rather than reacting to it.

References:

1. Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation. Vivian Lai, Samuel Carton, Rajat Bhatnagar, Vera Liao, Yunfeng Zhang, Chenhao Tan, Q. Liao (2022). International Conference on Human Factors in Computing Systems.
2. Conceptualizing Visual Analytic Interventions for Content Moderation. Sahaj Vaidya, Jie Cai, Soumyadeep Basu, Azadeh Naderi, D. Y. Wohn, Aritra Dasgupta (2021). Visual ...
3. Human-AI Collaboration in Content Moderation: The Effects of Information Cues and Time Constraints. Haoyan Li, Michael Chau (2023). European Conference on Information Systems.
4. AI vs. Human Moderators: A Comparative Evaluation of Multimodal LLMs in Content Moderation for Brand Safety. Adi Levi, Or Levi, Sardhendu Mishra, Jonathan Morra (2025). arXiv.org.