Optics-in-the-Loop: Co-Designing Spectrum-Splitting Filters and Fusion Networks

Liu et al. (Photonics 2024) proposed correcting wavefront aberrations from spectrum-splitting filters by optimizing coating phases to minimize scalar phase aberrations while maximizing transmission. That’s powerful on the hardware side but disconnected from downstream tasks. Here, we bring the optics into the learning loop: combine a differentiable wave-optics simulator of the filter stack with a multi-modal fusion model (e.g., FiHam’s progressive fusion for MRI sequences; Lu et al. 2025, or a generic cross-attention fusion) and backpropagate task loss (segmentation, subtype prediction, etc.) through the optical parameters.

We further integrate registration and calibration effects: include a feature-based registration layer (Shojaei et al. 2025; SPP-inspired multi-level features) with uncertainty penalties, and incorporate external calibration priors (Qiu et al. 2023 review) as constraints during optimization. The objective blends: (i) optical constraints (transmission, manufacturability), (ii) wavefront error, (iii) registration stability and (iv) downstream task loss. This co-design shifts “minimize aberration” from a purely optical criterion to “minimize the impact of aberration on fused representation quality,” which is ultimately what matters.

This is novel because it bridges physical filter design and learned fusion in one differentiable pipeline. It could yield spectrum-splitting designs that are task-optimized, not just optically optimized—potentially improving SNR, reducing artifacts in fused images, and making downstream models more robust, especially in high-throughput microscopy or multispectral endoscopy.

References:

1. Novel feature-based method for multi-modal biomedical image registration compared to intensity-based technique. M. Shojaei, Lichen Yang, Kazem Shojaei, Jeerapat Doungchawee, R. Vachet (2025). Scientific Reports.
2. The Correction Method for Wavefront Aberration Caused by Spectrum-Splitting Filters in Multi-Modal Optical Imaging System. Xiaolin Liu, Ying Huang, Xu Yan, Li Wang, Qiang Li, Tingcheng Zhang, Bin Hu, Wenping Lei, Shengbo Mu, Xiaohong Zhang (2024). Photonics.
3. Fine-Grained Hierarchical Progressive Modal-Aware Network for Brain Tumor Segmentation.. Chenggang Lu, Jianwei Zhang, Dan Zhang, Lei Mou, Jinli Yuan, Kewen Xia, Zhitao Guo, Jiong Zhang (2025). IEEE journal of biomedical and health informatics.
4. External multi-modal imaging sensor calibration for sensor fusion: A review. Z. Qiu, J. Martínez-Sánchez, Pedro Arias-Sánchez, R. Rashdi (2023). Information Fusion.