Geometry-Aware Unified Latents: Integrating Hierarchical 3D Structure into Latent Diffusion Priors

TL;DR: Let’s teach UL to “think in 3D”—by designing its latent space to capture geometric hierarchies, like shape, pose, and fine details, for 3D objects or scenes. This could start with a hierarchical autoencoder (inspired by LaGeM) regularized by a diffusion prior, enabling detailed, controllable 3D synthesis from compact latents.

Research Question: If we architect the UL latent space to explicitly encode hierarchical geometric information, can we improve 3D generative modeling and enable fine-grained geometric editing or interpolation?

Hypothesis: A hierarchical latent structure (e.g., levels encoding coarse shape, pose, and fine details) regularized by a diffusion prior will offer better disentanglement and controllability for 3D generation tasks, leading to higher-fidelity and more editable 3D content.

Experiment Plan: Adapt the UL framework to 3D data (meshes, point clouds) using a hierarchical autoencoder as in LaGeM. Regularize each hierarchical latent level with a diffusion prior, possibly with cascaded diffusion as in LaGeM. Train on large-scale 3D datasets (e.g., ShapeNet, Objaverse). Evaluate on 3D reconstruction, interpolation, and attribute editing tasks (e.g., change pose, scale, fine details). Compare geometry quality, reconstruction accuracy, and editing flexibility to existing 3D diffusion and autoencoder models.

References:

• Zhang, B., & Wonka, P. (2024). LaGeM: A Large Geometry Model for 3D Representation Learning and Diffusion. International Conference on Learning Representations.
• Hahm, J., Lee, J., Kim, S., & Lee, J. (2024). Isometric Representation Learning for Disentangled Latent Space of Diffusion Models. International Conference on Machine Learning.