Adaptive Perturbation Sampling via Local Curvature for Efficient Task Expert Discovery

TL;DR: Instead of randomly shaking up model weights, what if we nudge them in smarter directions based on how “bumpy” the loss landscape is? We'll design an adaptive sampling strategy that targets promising expert regions more efficiently.

Research Question: Can we improve the efficiency of finding diverse task experts near pretrained weights by adapting the direction and magnitude of parameter perturbations based on local curvature or gradient information?

Hypothesis: Adaptive perturbation methods informed by local curvature (e.g., using the Fisher Information Matrix or Hessian approximations) will more effectively uncover high-quality and diverse task experts than isotropic random sampling, especially in high-dimensional parameter spaces.

Experiment Plan: Develop a perturbation sampling algorithm where the covariance of perturbations is guided by curvature estimates (inspired by Sawada et al., 2025). Compare the discovered expert density, diversity, and ensemble performance to the original random sampling method from Neural Thickets. Key metrics: number of unique experts found per unit computation, ensemble accuracy, and diversity. Expected outcome: Adaptive methods yield higher-quality ensembles with fewer samples.

References:

• Gan, Y., & Isola, P. (2026). Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights.
• Sawada, H., Aoyama, K., & Notomi, M. (2025). Layered-Parameter Perturbation for Zeroth-Order Optimization of Optical Neural Networks. AAAI Conference on Artificial Intelligence.