Redundancy-Driven Self-Supervision: Information-Theoretic Representation Learning via Multi-Source Fusion

Inspired by the Graph Redundancy Network (Fu et al., 2023), which uses cognitive neuroscience principles to fuse information from multiple graph sources, this idea extends the concept to general neural architectures and data modalities. The proposal is to create new self-supervised loss functions that explicitly maximize the mutual information between redundant components (shared across modalities or augmentations) and minimize it for complementary (unique) components, in the spirit of multi-view learning. Unlike current methods that often focus on contrastive alignment or simple aggregation, this approach formalizes redundancy and complementarity using information-theoretic measures and neural estimators. The result: richer, more disentangled representations that better capture structural invariants and variations across sources. This could significantly impact multi-modal learning, sensor fusion, and robustness to missing data, opening new research into the theoretical limits of redundancy-driven learning.

References:

1. Cognitive-inspired Graph Redundancy Networks for Multi-source Information Fusion. Yao Fu, Junhong Wan, Junlan Yu, Weihao Jiang, Shi Pu (2023). International Conference on Information and Knowledge Management.