Manifold-Constrained Non-Identity Mappings: Exploring Nonlinear and Stochastic Skip Connections

TL;DR: What if skip connections didn’t have to be straight lines? This idea explores nonlinear or even stochastic manifold constraints instead of deterministic identity projections, possibly unlocking new forms of regularization or expressivity. A pilot study could compare nonlinear and stochastic manifold projections to standard mHC in terms of training dynamics and generalization.

Research Question: Can nonlinear or stochastic manifold-constrained skip connections improve robustness and expressivity in deep architectures compared to deterministic identity-mapping-based mHC?

Hypothesis: Introducing controlled nonlinearity or stochasticity in the manifold constraint will act as an implicit regularizer, potentially reducing overfitting and improving generalization, especially in low-data or noisy settings.

Experiment Plan: - Setup: Implement variants of mHC where the manifold constraint is a nonlinear (e.g., learned autoencoder) or stochastic mapping (e.g., Gaussian perturbation in the projected space).

• Data: Use noisy or small datasets (e.g., few-shot learning benchmarks, UCI datasets).
• Measurements: Generalization gap, robustness to label noise, and training/validation loss trajectories.
• Expected Outcome: Nonlinear/stochastic mHC variants should demonstrate enhanced generalization on difficult tasks, with stability comparable to (or better than) standard mHC.

References:

• Xie, Z., et al. (2025). mHC: Manifold-Constrained Hyper-Connections.
• Shi, J., Wu, T., Qin, A. K., Shao, T., Lei, Y., & Jeon, G. (2023). Deep-Growing Neural Network With Manifold Constraints for Hyperspectral Image Classification. IEEE Transactions on Neural Networks and Learning Systems.