Deep Networks, Shallow Rewards? Probing Exploration and Representation in Ultra-Deep Self-Supervised RL

TL;DR: Super-deep RL networks might learn to reach goals better, but are they actually exploring more, or just memorizing?

Research Question: How does scaling network depth affect the exploration strategies and learned representations in self-supervised, goal-conditioned RL—do deeper networks drive fundamentally different exploration behaviors?

Hypothesis: Ultra-deep networks, via their greater representational capacity, will develop more global and temporally-extended exploration strategies (e.g., learning longer-term dependencies), yielding more robust generalization to out-of-distribution goals.

Experiment Plan: - Train agents with varying depths in sparse-reward navigation and manipulation tasks.

• Use metrics such as state visitation entropy, goal coverage, and novelty seeking to quantify exploration.
• Employ probing tasks and dimensionality reduction on learned representations to assess generality and abstraction level.
• Compare against exploration strategies from cognitive-inspired RL (e.g., from Tai & Liu, 2016), and vary exploration hyperparameters (epsilon decay, etc.) to see if depth interacts with explicit exploration.

References:

• Wang, K., et al. (2024). 1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities.
• Tai, L., & Liu, M. (2016). Towards Cognitive Exploration through Deep Reinforcement Learning for Mobile Robots. arXiv.org.
• Thadikamalla, S., Joshi, P., Raman, B., Manipriya, S., & Sanodiya, R. K. (2024). Optimizing Traffic Signal Control with Deep Reinforcement Learning: Exploring Decay Rate Tuning for Enhanced Exploration-Exploitation Trade-off. SPIN.