Hierarchical Safe Exploration with Human-in-the-Loop Constraint Shaping

While Turchetta et al. (2020) leverage automatic instructors and curriculum induction for safety, these frameworks generally treat constraints as fixed and opaque. But in many domains, humans have nuanced knowledge about which constraints matter most or how trade-offs should be made (e.g., “It’s okay to briefly exceed this threshold if another, more critical safety property is preserved”). By incorporating human-in-the-loop feedback at a hierarchical level, the agent can learn to shape and prioritize constraints dynamically. This could be implemented via preference learning, inverse RL, or interactive querying during training. The novelty here is not just in leveraging human knowledge, but in making constraint management itself a learned, adaptive process—potentially leading to RL agents that are both safer and more aligned with human values in complex tasks. This challenges the assumption that constraints are static and externally defined, and could be transformative in human-centered or high-stakes applications.

References:

1. Safe Reinforcement Learning via Curriculum Induction. M. Turchetta, A. Kolobov, S. Shah, A. Krause, Alekh Agarwal (2020). Neural Information Processing Systems.