Adversarial Experience Synthesis: Stress-Testing DreamGym with LLM-Generated "Impossible" Tasks

TL;DR: Can we break DreamGym by feeding it synthetically generated tasks designed to be unsolvable? We’ll use LLMs to create counterfactual scenarios where physics or logic is violated, measuring how agents collapse or adapt.

Research Question: How do reasoning-based experience models (like DreamGym) handle synthetic experiences that violate fundamental environment invariants?

Hypothesis: Introducing adversarial synthetic experiences will initially degrade performance but ultimately improve agent robustness if the model learns to identify and reject logical inconsistencies.

Experiment Plan: - Setup: Train DreamGym agents with two buffers: (a) standard synthetic experiences, (b) adversarial buffer (e.g., "gravity-reversed" WebArena tasks generated by GPT-4).

• Data: Use metrics like success rate on adversarial vs. standard tasks, and "consistency score" for state transitions (via human evaluation).
• Analysis: Compare to a DreamGym control; test if agents develop internal "sanity checks" for synthetic inputs.
• Expected Outcome: Agents exposed adversarially will show initial performance drops but achieve 10% higher robustness in real-world edge cases (e.g., sensor failures).

References: ['Chen, Z., et al. (2025). Scaling Agent Learning via Experience Synthesis. arXiv.org.', 'Baek, I.-C., et al. (2025). PCGRLLM: Large Language Model-Driven Reward Design for Procedural Content Generation Reinforcement Learning. arXiv.org.', 'Hu, J., et al. (2024). Continual Diffuser (CoD): Mastering Continual Offline Reinforcement Learning with Experience Rehearsal. arXiv.org.']