Learning to Leverage Near-Miss Events: A Counterfactual Approach to Safe RL

Most safe RL methods (e.g., FISOR by Zheng et al., 2024; RLBUS by Rabiee & Safari, 2023) focus on strict avoidance of constraint violations or hard masking of unsafe actions. However, this can lead to overly conservative policies that miss out on learning about the boundaries of safe regions. What if, instead, we systematically collected and analyzed near-miss events—where an agent approached but didn’t quite violate a constraint? By treating these as counterfactuals, we could train auxiliary models that estimate “distance to unsafe” and use these signals to guide exploration more intelligently. This would allow agents to probe the edges of safety, learning richer representations of safe/unsafe boundaries. It builds on the spirit of backup and barrier-based approaches but explicitly exploits the information in near-misses, which is mostly ignored in current literature. The upshot? More efficient exploration, better generalization to unseen unsafe regions, and potentially safer policies in complex, high-dimensional spaces.

References:

1. Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model. Yinan Zheng, Jianxiong Li, Dongjie Yu, Yujie Yang, Shengbo Eben Li, Xianyuan Zhan, Jingjing Liu (2024). International Conference on Learning Representations.
2. Safe Exploration in Reinforcement Learning: Training Backup Control Barrier Functions with Zero Training Time Safety Violations. Pedram Rabiee, Amirsaeid Safari (2023). arXiv.org.