Depth Meets Sparsity: Dynamic Sparse Training for Ultra-Deep RL Networks

TL;DR: Instead of making every layer in a 1000-layer network dense, let’s try making only the most important connections active—saving memory, speeding up training, and maybe even improving performance.

Research Question: Can dynamic sparse connectivity in ultra-deep RL networks retain or even boost performance while drastically reducing computational and memory overhead?

Hypothesis: By maintaining only the most salient connections at each training step (via importance-based pruning and regrowth), ultra-deep networks can avoid overfitting and vanishing gradients, while remaining efficient and effective for complex goal-reaching tasks.

Experiment Plan: - Implement a dynamic sparse training (DST) algorithm (e.g., as in Xu et al., 2024) within the 1000-layer self-supervised RL framework.

• Compare dense vs. sparse ultra-deep architectures on standard RL benchmarks (same as in the original paper), measuring sample efficiency, memory usage, and final success rates.
• Analyze how the pattern of sparse connectivity evolves over time and whether certain layers or regions become more/less critical as the agent learns.
• Explore the impact of different sparsity levels and regrowth strategies on both learning speed and generalization.

References:

• Wang, K., et al. (2024). 1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities.
• Xu, M., Chen, X., & Wang, J. (2024). A Novel Topology Adaptation Strategy for Dynamic Sparse Training in Deep Reinforcement Learning. IEEE Transactions on Neural Networks and Learning Systems.