Learning the Covering Distribution with Return-Guided Diffusion

Operationalize Mao’s “additional covering distribution” by training a policy-decoupled, return-guided diffusion model that samples candidate states and trajectories close to the dataset distribution but biased toward higher returns. Use negative sampling and value filters as in HIPODE (arXiv 2023) and classifier-free guidance as in GTA (NeurIPS 2024) to control quality. The learned covering distribution then plugs into algorithms with MIS/CPPO-style guarantees. This is novel because prior work augments data or learns pessimistic models under partial coverage but none explicitly learn a covering distribution with theoretical intent. This idea converts a theoretical requirement into a learnable object with explicit quality control and policy-decoupling. It extends Mao’s theory by learning the covering distribution via diffusion, improves on policy-dependent augmentation by following HIPODE’s decoupling, and inherits stitching/trajectory-plausibility strengths without tying to a specific downstream policy. A learned, verifiable covering distribution reduces reliance on unrealistic exploration assumptions and broadly benefits model-based and model-free offline RL, clarifying when and how augmentation tightens instance-dependent finite-sample bounds. Impact includes enabling stronger guarantees and better performance in low-coverage regimes common in healthcare, traffic, and operations.

References:

1. Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage. Masatoshi Uehara, Wen Sun (2021). International Conference on Learning Representations.
2. Offline Reinforcement Learning with Additional Covering Distributions. Chenjie Mao (2023). Trans. Mach. Learn. Res..
3. HIPODE: Enhancing Offline Reinforcement Learning with High-Quality Synthetic Data from a Policy-Decoupled Approach. Shixi Lian, Yi Ma, Jinyi Liu, Yan Zheng, Zhaopeng Meng (2023). arXiv.org.
4. GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learning. Jaewoo Lee, Sujin Yun, Taeyoung Yun, Jinkyoo Park (2024). Neural Information Processing Systems.