Representation Learning via Synthetic Cross-Domain Counterfactuals: Generalizing Beyond Training Domains

Cross-modal synthesis (e.g., MR-to-CT translation, Qin et al., 2024; Longuefosse et al., 2023) is often limited to transferring between two known domains. This idea innovates by using generative models (GANs, diffusion models) to synthesize "counterfactual" inputs—plausible but out-of-domain examples—for both seen and unseen modalities/domains. For instance, in medical imaging, the model could generate hypothetical scans that interpolate between real MRIs and CTs, or even other imaging modalities. The representation learner is then trained not only to perform its core task but also to maintain consistency and robustness across these synthetic domain-shifted counterfactuals. This method would test and enforce the generality of learned features, moving beyond the static domain adaptation framework and enabling models to better handle "unknown unknowns" in deployment.

References:

1. Cross-Modality Program Representation Learning for Electronic Design Automation with High-Level Synthesis. Zongyue Qin, Yunsheng Bai, Atefeh Sohrabizadeh, Zijian Ding, Ziniu Hu, Yizhou Sun, Jason Cong (2024). Workshop on Machine Learning for CAD.
2. MR to CT Synthesis Using GANs: A Practical Guide Applied to Thoracic Imaging. Arthur Longuefosse, B. D. Senneville, G. Dournes, I. Benlala, F. Laurent, P. Desbarats, F. Baldacci (2023). VISIGRAPP.