Code World Models Meets DreamGym: Programmatic Environment Synthesis for Transparent RL Training

Research Question: Can generating environment models as executable code (rather than text-based reasoning traces) improve the transparency, correctness, and reproducibility of synthetic experiences for RL, and how does this impact agent performance?

Hypothesis: Code-based world models, as in GIF-MCTS, will reduce ambiguity and facilitate bug detection in synthetic environments, leading to more stable RL training and easier debugging/transfer.

Experiment Plan: Adapt DreamGym’s LLMs to synthesize executable Python environment models, following GIF-MCTS strategies for code generation and improvement. Compare RL agent learning curves, error rates, and sim-to-real transfer for code-based vs. text-based synthetic rollouts. Conduct ablation studies on world model interpretability and error traceability.

References: ['Dainese, N., Merler, M., Alakuijala, M., & Marttinen, P. (2024). Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Search. Neural Information Processing Systems.']