Beyond Linearity: Probing, Quantifying, and Challenging Linearity Assumptions in Representation Spaces

Much foundational work—like Collins et al. (2023) on multi-task learning—assumes or leverages the linearity of neural representations (e.g., final layers as linear classifiers, linear subspace recovery). Yet, real-world data and tasks often require capturing non-linear relations. This idea proposes to formally characterize where and how linearity breaks down in modern architectures, possibly leveraging tools from non-linear manifold learning or information geometry. The research would involve constructing datasets and synthetic tasks that challenge linear assumptions, then introducing new neural architectures or regularizers that encourage non-linear but structured (e.g., piecewise linear, manifold-constrained) representations. This could lead to a new class of neural models that adaptively choose between linear and non-linear representation strategies, and new theory for when linearity can be safely assumed. Such advances would clarify foundational limits and inform the design of future architectures, especially in cases where current models falter.

References:

1. Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks. Liam Collins, Hamed Hassani, M. Soltanolkotabi, Aryan Mokhtari, S. Shakkottai (2023). International Conference on Machine Learning.
2. Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks. Liam Collins, Hamed Hassani, M. Soltanolkotabi, Aryan Mokhtari, S. Shakkottai (2023). International Conference on Machine Learning.