Curriculum-Driven Replay Buffer Initialization: From Zero to Hero in Synthetic RL

TL;DR: DreamGym initializes replay buffers with real data—but what if we start with curated synthetic data? We’ll test if a progressively seeded buffer (easy → hard synthetic tasks) outperforms real-data initialization for zero-shot transfer.

Research Question: How does the composition of DreamGym’s initial replay buffer (real vs. synthetic vs. mixed) impact early training efficiency and final performance?

Hypothesis: A curriculum-initialized synthetic buffer reduces dependency on scarce real data while maintaining DreamGym’s stability advantages.

Experiment Plan: - Setup: Compare three buffer initializations: (a) real data (DreamGym baseline), (b) synthetic-only (curriculum-sorted), (c) hybrid (50:50).

• Data: Track regret in early training (first 10k episodes) and final performance on WebArena and Overcooked.
• Analysis: Ablate buffer size and curriculum steepness (e.g., task difficulty gaps).
• Expected Outcome: Curriculum-synthetic buffers will match real-data performance with 70% less real data, proving DreamGym’s flexibility.

References: ['Chen, Z., et al. (2025). Scaling Agent Learning via Experience Synthesis. arXiv.org.', 'Bhati, R., et al. (2023). Curriculum Learning for Cooperation in Multi-Agent Reinforcement Learning. arXiv.org.', 'Horváth, D., et al. (2023). HiER: Highlight Experience Replay for Boosting Off-Policy Reinforcement Learning Agents. IEEE Access.']