Ultra-Deep RL Networks with Spiking Neurons: Biologically Plausible Scaling

TL;DR: What if we make our 1000-layer RL networks use spiking neurons, like the brain?

Research Question: Can integrating spiking neural network (SNN) architectures into ultra-deep self-supervised RL models provide greater robustness and efficiency, and does it alter the qualitative behaviors learned?

Hypothesis: SNN-based ultra-deep RL networks will demonstrate improved robustness to noise and energy efficiency, and may develop sparser, more biologically plausible representations that affect exploration and goal-reaching.

Experiment Plan: - Adapt BrainQN (Feng et al., 2024) to the ultra-deep self-supervised RL setting, constructing 512- to 1024-layer SNN-based agents.

• Train and evaluate these agents on the same goal-conditioned tasks as in the original paper, measuring success, energy use (on neuromorphic hardware if possible), and robustness to input noise/perturbations.
• Compare qualitative and quantitative differences in learned policies, exploration, and generalization versus dense ANN-based ultra-deep networks.

References:

• Wang, K., et al. (2024). 1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities.
• Feng, S., Cao, J., Ou, Z., Chen, G., Zhong, Y., Wang, Z., Yan, J., Chen, J., Wang, B., Zou, C., Feng, Z., & Wang, Y. (2024). BrainQN: Enhancing the Robustness of Deep Reinforcement Learning with Spiking Neural Networks. Advanced Intelligent Systems.