Batch Quality–Stratified Pessimism: Allocating Conservatism Where It Matters

Partition the dataset into quality strata (e.g., by return quantiles or IQL-style advantages) and learn per-stratum pessimism parameters: CQL’s conservatism coefficient, TD3+BC’s BC weight, and model-based penalties (MOPO/COMBO) are made state- or stratum-conditional. High-quality strata get lighter pessimism to avoid clipping optimal behavior; low-quality or sparse strata get stronger regularization. Incorporate density-ratio or partial-coverage diagnostics to inform the allocation, and optionally distill to a compact dataset for stable training. This approach is novel because current methods set a global pessimism knob, which is brittle when batches mix expert and mediocre behavior—a common source of offline RL failures. It formalizes a “pessimism budget” and distributes it according to estimated uncertainty and quality. It systematizes intuitions behind TD3+BC versus CQL trade-offs and addresses biased exploration by protecting trustworthy regions while encouraging cautious extrapolation elsewhere. Compatible with state-focused constraints and traffic-control settings with structured heterogeneity. Easy to implement in existing pipelines, expected to boost returns on mixed-quality datasets without sacrificing safety or stability. Theoretically, under partial coverage one can bound regret by weighting strata with their concentrability and variance. Impact is a practical recipe to tame the quality–conservatism conflict, improving reliability on real-world logs where data quality is uneven.

References:

1. Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage. Masatoshi Uehara, Wen Sun (2021). International Conference on Learning Representations.
2. Dataset Distillation for Offline Reinforcement Learning. Jonathan Light, Yuanzhe Liu, Ziniu Hu (2024). arXiv.org.
3. State-Constrained Offline Reinforcement Learning. Charles A. Hepburn, Yue Jin, Giovanni Montana (2024). Trans. Mach. Learn. Res..