Meta-Synthetic Experience Optimization: Learning to Generate Better Synthetic Experiences

TL;DR: Instead of handcrafting DreamGym’s experience model, what if we use RL to optimize the synthesis process itself? We’ll train a meta-agent that tweaks synthetic task parameters to maximize downstream agent performance.

Research Question: Can a meta-learning loop improve the quality and diversity of DreamGym’s synthetic experiences beyond human-designed heuristics?

Hypothesis: Meta-optimized synthesis will generate more challenging and educationally valuable experiences, accelerating agent learning.

Experiment Plan: - Setup: Implement a two-level system: (a) DreamGym base agent, (b) meta-RL agent that adjusts synthesis parameters (e.g., task difficulty, noise levels) based on base agent’s learning progress.

• Data: Use multi-agent Pommerman; compare learning curves for base agents under (a) fixed synthesis (DreamGym), (b) meta-optimized synthesis.
• Analysis: Measure task diversity (entropy of state-action pairs) and Elo rating gains.
• Expected Outcome: Meta-synthesis agents will achieve 25% higher Elo ratings in 50% fewer episodes by dynamically targeting weaknesses.

References: ['Chen, Z., et al. (2025). Scaling Agent Learning via Experience Synthesis. arXiv.org.', 'Huynh, N.-M., et al. (2024). Multi-Agent Training for Pommerman: Curriculum Learning and Population-based Self-Play Approach. arXiv.org.', 'Dainese, N., et al. (2024). Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Search. Neural Information Processing Systems.']