HyperTransformer-MAML: Task-Conditioned Parameter Transformations for Heterogeneous Domains

Moon et al. (2024) showed that static initializations can hurt on heterogeneous animal sound tasks and proposed task-adaptive parameter transformations (TAPT). Let’s generalize this idea into a unified “HyperTransformer-MAML” (HT-MAML): a small hypernetwork consumes a task embedding (from a few support examples) and outputs per-layer low-rank adapters or affine transformations applied to the base model prior to the inner-loop. The meta-objective trains both the hypernetwork and base initialization so they cooperate: the hypernetwork makes the parameters task-specific, then a few gradient steps polish them. Two additional twists:

• Learn task embeddings via modality-appropriate self-supervision (e.g., contrastive audio features for ASC; multilingual sentence embeddings as in Rahimi & Veisi, 2024 for intent/hate speech), and allow cross-domain sharing (HateMAML’s self-supervision strategy; Awal et al., 2023).
• Add channel-exchanging augmentation (Zhang et al., 2022) during meta-training to expose the hypernetwork to inter-class and inter-task feature recombinations, improving robustness.
  Compared to TAPT, HT-MAML learns a general hypernetwork that can condition transformations on rich task embeddings and deploy different transform families per layer or module, making it less brittle across modalities (audio, text, medical imaging; cf. Alsaleh et al., 2024) and architectures. This could close the performance gap in highly diverse task families where plain MAML deviates from expectations.

References:

1. Model-Agnostic Meta-Learning for Multilingual Hate Speech Detection. Rabiul Awal, Roy Ka-Wei Lee, Eshaan Tanwar, Tanmay Garg, Tanmoy Chakraborty (2023). IEEE Transactions on Computational Social Systems.
2. A Task-Adaptive Parameter Transformation Scheme for Model-Agnostic-Meta-Learning-Based Few-Shot Animal Sound Classification. Jaeuk Moon, Eunbeen Kim, Junha Hwang, Eenjun Hwang (2024). Applied Sciences.
3. Integrating Model-Agnostic Meta-Learning with Advanced Language Embeddings for Few-Shot Intent Classification. Ali Rahimi, Hadi Veisi (2024). 2024 32nd International Conference on Electrical Engineering (ICEE).
4. Few-Shot Learning for Medical Image Segmentation Using 3D U-Net and Model-Agnostic Meta-Learning (MAML). Aqilah M. Alsaleh, Eid Albalawi, A. Algosaibi, Salman S. Albakheet, S. B. Khan (2024). Diagnostics.
5. Improving Generalization of Model-Agnostic Meta-Learning by Channel Exchanging. Ce Zhang, Ruixuan Chen, Yifeng Zeng, Shaolong Ren, Qingshan Cui (2022). 2022 International Conference on Electronics and Devices, Computational Science (ICEDCS).