State-Only Reachability Constraints with Generative Trajectory Stitching

Following Hepburn et al. (TMLR 2024), constrain only the state marginal: allow actions outside the dataset so long as predicted next-state distributions remain within a learned invariant set. Use a dynamics model to check short-horizon reachability and a diffusion generator (DiffStitch, ICML 2024; GTA, NeurIPS 2024) to propose stitching transitions that connect low-return to high-return regions through in-distribution states. Train the policy with a penalty proportional to the probability of leaving the invariant state set, not to action support deviations. This is novel because StaCQ frames constraints on states rather than actions but does not incorporate constructive trajectory bridging, while DiffStitch/GTA generate plausible stitches but do not formalize state-only constraints or reachability. This idea unifies them: state-feasible stitching with explicit reachability checks. It extends StaCQ from a conceptual framework to a generative, model-based algorithm; leverages diffusion models to propose high-quality bridges; contrasts with PRDC, which anchors to dataset actions, by instead optimizing for state feasibility and reachability. This combats the “can’t get there from here” problem in sparse-data regimes and avoids over-conservatism by permitting beneficial OOD actions when they quickly return to in-distribution states. The approach should especially help on tasks with few optimal trajectories. Impact includes stronger trajectory recomposition in offline RL with theoretical control over state distribution shift, improving both sample-efficiency and final performance.

References:

1. Policy Regularization with Dataset Constraint for Offline Reinforcement Learning. Yuhang Ran, Yi-Chen Li, Fuxiang Zhang, Zongzhang Zhang, Yang Yu (2023). International Conference on Machine Learning.
2. DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitching. Guanghe Li, Yixiang Shan, Zhengbang Zhu, Ting Long, Weinan Zhang (2024). International Conference on Machine Learning.
3. State-Constrained Offline Reinforcement Learning. Charles A. Hepburn, Yue Jin, Giovanni Montana (2024). Trans. Mach. Learn. Res..
4. GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learning. Jaewoo Lee, Sujin Yun, Taeyoung Yun, Jinkyoo Park (2024). Neural Information Processing Systems.