Conflicting Theories, Unified Framework: Reconciling Divergent Empirical Results in RLHF and Policy Optimization

Zhu et al. (2023) highlight intriguing cases where maximum likelihood estimators behave inconsistently in RLHF, depending on underlying assumptions. Similarly, Hu et al. (2023) survey theoretical nuances in policy optimization. Yet, there’s no systematic foundation for predicting when these theoretical expectations will conflict with empirical findings. This research would build a meta-theoretical framework that catalogs known conflicts (e.g., when MLE-based RLHF fails vs. succeeds), analyzes their root causes, and proposes new conflict-resolving algorithmic variants. The project would use a combination of formal analysis, synthetic benchmark generation (to stress-test hypotheses), and meta-learning to identify regularities in when different RL theories diverge. The result would be both a practical tool for RL practitioners (“Which theory applies here?”) and new foundational insights—potentially leading to a new generation of RL algorithms that adapt their theoretical underpinnings in response to observed empirical anomalies.

References:

1. Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons. Banghua Zhu, Jiantao Jiao, Michael I. Jordan (2023). International Conference on Machine Learning.
2. Toward a Theoretical Foundation of Policy Optimization for Learning Control Policies. Bin Hu, Kaiqing Zhang, Na Li, M. Mesbahi, Maryam Fazel, Tamer Başar (2023). Annu. Rev. Control. Robotics Auton. Syst..