Beyond the Identity: Learning Alternative Manifold Constraints for Hyper-Connections

TL;DR: What if, instead of always restoring the classic identity mapping, we let the network learn which manifold constraints work best for different layers or tasks? Imagine a network that adapts its skip connections to what the data needs, not just what tradition dictates. An initial experiment could compare learned manifold constraints against fixed identity mapping in mHC across diverse vision and audio tasks.

Research Question: Can learnable manifold constraints in hyper-connections outperform traditional identity mapping, leading to enhanced stability and task-specific adaptability in deep networks?

Hypothesis: Dynamically learned manifold constraints will allow the model to flexibly balance stability and expressivity, improving both training robustness and generalization compared to fixed identity mappings.

Experiment Plan: - Setup: Extend the mHC framework by parameterizing the manifold constraint (e.g., via a small neural network or differentiable selection mechanism) so that the optimal constraint can be learned during training.

• Data: Use standard image classification datasets (CIFAR-10, ImageNet) and at least one audio (e.g., raw waveform) dataset to test generality.
• Measurements: Compare convergence speed, final accuracy, and training stability metrics (gradient norms, loss oscillations) between standard mHC and the learnable constraint version.
• Expected Outcome: If the hypothesis holds, the learnable-mHC should show improved stability and accuracy, especially in domains where the ideal skip connection diverges from identity.

References:

• Xie, Z., Wei, Y., Cao, H., Zhao, C., Deng, C., Li, J., Dai, D., Gao, H., Chang, J., Zhao, L., Zhou, S., Xu, Z., Zhang, Z., Zeng, W., Hu, S., Wang, Y., Yuan, J., Wang, L., & Liang, W. (2025). mHC: Manifold-Constrained Hyper-Connections.
• Naranjo-Alcazar, J., Perez-Castanos, S., Martín-Morató, I., Zuccarello, P., & Cobos, M. (2019). On the performance of residual block design alternatives in convolutional neural networks for end-to-end audio classification. arXiv.org.