Universal Representation Alignment: Learning Cross-Architecture Invariants in Deep Nets

Van Rossem & Saxe (2024) and Yang et al. (2021) both touch on universality in representation learning, showing that different architectures can converge to surprisingly similar representations. This idea proposes to formalize and exploit this: train a "meta-representation learner" that, given features from diverse architectures (CNNs, GNNs, Transformers, SSMs), learns a shared space where semantically equivalent representations align—even if their architectures or modalities differ. This could involve contrastive learning, mutual information maximization, or novel alignment losses. The goal is to enable plug-and-play transfer of learned features across networks and domains, facilitating model ensembling, transfer learning, or even zero-shot generalization to entirely new architectures. The project would both deepen theory (what invariants exist across architectures?) and yield practical tools for building more interoperable, robust AI systems.

References:

1. A theory of representation learning gives a deep generalisation of kernel methods. Adam X. Yang, Maxime Robeyns, Edward Milsom, Ben Anson, Nandi Schoots, L. Aitchison (2021). International Conference on Machine Learning.
2. When Representations Align: Universality in Representation Learning Dynamics. Loek van Rossem, Andrew M. Saxe (2024). International Conference on Machine Learning.