Robotics-Driven Floorplan-to-3D Scene Synthesis with Embodied Affordance Constraints

Building on HouseCrafter’s method of lifting floorplans to 3D scenes via autoregressive, multi-view RGB-D generation, this research proposes explicitly incorporating robotics-centric affordance constraints into the scene synthesis process. Unlike existing works that focus on visual and geometric consistency, this approach introduces a feedback loop between the diffusion-based scene generator and simulated embodied agents (e.g., robot navigation or manipulation planners). During generation, the system simulates robot trajectories or manipulation tasks within the partial scene to identify affordance violations such as dead-ends, unreachable objects, or physically implausible arrangements. These violations are encoded as conditioning signals or loss functions to guide the diffusion model to produce layouts and object placements that support robot behavior. This fusion of generative diffusion models with embodied AI constraints aims to bridge the gap between appearance and functional usability, enabling the creation of 3D indoor scenes that are both visually compelling and robot-ready. The approach also enables new benchmarks and evaluation protocols based on robotic task success rates, advancing simulation environments for robotics, embodied AI, and AR/VR.

References:

1. Mixed Diffusion for 3D Indoor Scene Synthesis. Siyi Hu, Diego Martín Arroyo, Stephanie Debats, Fabian Manhardt, Luca Carlone, Federico Tombari (2024). arXiv.org.
2. MVD-Fusion: Single-view 3D via Depth-consistent Multi-view Generation. Hanzhe Hu, Zhizhuo Zhou, Varun Jampani, Shubham Tulsiani (2024). Computer Vision and Pattern Recognition.