Cross-Modal Experience Synthesis: Translating Text-to-Video for Vision-Based RL

TL;DR: What if DreamGym could generate synthetic video experiences from text descriptions? We’ll use diffusion models to create visual rollouts for image-based RL tasks like navigation, testing if agents trained on these videos outperform pixel-only baselines.

Research Question: Can multi-modal synthesis (text → video) enhance DreamGym’s applicability to vision-heavy domains where text alone is insufficient?

Hypothesis: Video synthesis provides richer state representations than text, enabling faster learning for vision-based agents while retaining DreamGym’s scalability.

Experiment Plan: - Setup: Integrate a video diffusion model (e.g., Sora) into DreamGym to generate synthetic visual rollouts from text task descriptions.

• Data: Use Habitat and CARLA for navigation/driving; compare agents trained on (a) text-only synthetic experiences, (b) video synthetic experiences, (c) real pixels.
• Analysis: Measure sample efficiency (episodes to 80% success) and transfer to real cameras.
• Expected Outcome: Video synthesis agents will match real-pixel performance in 30% fewer interactions, bridging DreamGym’s gap in non-text domains.

References: ['Chen, Z., et al. (2025). Scaling Agent Learning via Experience Synthesis. arXiv.org.', 'Liu, X., et al. (2024). Learning future representation with synthetic observations for sample-efficient reinforcement learning. Science China Information Sciences.', 'Pang, J.-C., et al. (2025). ImagineBench: Evaluating Reinforcement Learning with Large Language Model Rollouts. arXiv.org.']