Neuromorphic BioFusion: Event-Frame Fusion for Dynamic Biomedical Imaging

Event cameras excel at high temporal resolution and dynamic range; EISNet (Xie et al. 2024) fused events with frames for semantic segmentation, addressing two core problems: encoding events (their AEIM module) and adaptive fusion (their MRFM). Biomedical domains increasingly need dynamic imaging—think calcium transients in neuroimaging, beating-heart OCT, or flickering endoscopic illumination.

This project adapts EISNet’s ideas to biomedical fusion: (1) design a biomedical Activity-Aware Event Integration Module to accumulate events into task-specific representations (e.g., motion-aware accumulation tuned to physiological frequencies), and (2) a Modality Recalibration and Fusion Module that learns when to trust event vs frame data, depending on motion, illumination, and signal quality. For endoscopic tool/tissue segmentation under motion and lighting changes, event streams can stabilize temporal features; for fluorescence lifetime imaging, events can capture photon bursts during rapid decay windows; for intravascular OCT, events can aid motion compensation.

Unlike typical multi-modal biomedical fusion (MRI+PET, image+clinical), this brings a bio-inspired sensor modality with distinct sampling physics. It differs from general video fusion and from DMC-Fusion’s static modality mix (Sathya 2024). The promise is motion-robust, temporally precise fusion that maintains fidelity in challenging, fast-changing scenes—crucial for intraoperative guidance and live-cell imaging.

References:

1. Multi-Modal Image Fusion for Early Disease Diagnosis: AI in Medical Imaging. Sathya A (2024). Multidisciplinary Journal for Applied Research in Engineering and Technology.
2. EISNet: A Multi-Modal Fusion Network for Semantic Segmentation With Events and Images. Bochen Xie, Yongjian Deng, Z. Shao, Youfu Li (2024). IEEE transactions on multimedia.