Reflect-and-Generate: Closed-Loop Co-Training of VLMs with Diffusion Counterfactuals and Self-Reflection

Create a co-training pipeline that (i) generates contrastive counterfactuals via image-difference and diffusion (Img-Diff; Jiao et al., 2024; UniDiff; Dong et al., 2023), (ii) fine-tunes VLMs jointly on discriminative (contrastive/ITC) and generative (caption, rationale) tasks with reciprocal semantic consistency (UniDiff), (iii) optimizes reasoning via reflection-based self-training (R3V; Cheng et al., 2024) and a hybrid RL reward (à la WeThink; Yang et al., 2025) that combines rule checks, model-based assessment, and safety preference constraints (SPA-VL; Zhang et al., 2024). Prior efforts study these elements largely in isolation—contrastive data synthesis, generative-discriminative unification, reflective reasoning, RL for multimodal QA, and safety preference alignment. This is a closed loop where diffusion generates targeted counterfactuals for failure modes surfaced by reflection, and RL rewards prioritize both correctness and harmlessness. UniDiff shows generative–discriminative synergy; Img-Diff shows how to synthesize fine-grained differences; R3V shows reflection can self-improve VL reasoning; WeThink contributes a hybrid RL reward design; SPA-VL provides safety preferences. The loop explicitly attacks the sources of multimodal hallucination and brittle reasoning, while continuously supplying tailored counterfactual supervision. It also integrates safety alignment as a first-class optimization objective rather than a post-hoc filter. More reliable multimodal reasoning across domains (math, spatial, commonsense) with stronger robustness to distribution shifts, and safer outputs without sacrificing utility.

References:

1. SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Models. Yongting Zhang, Luyao Chen, Guodong Zheng, Yifeng Gao, Rui Zheng, Jinlan Fu, Zhen-fei Yin, Senjie Jin, Yu Qiao, Xuanjing Huang, Feng Zhao, Tao Gui, Jing Shao (2024). Computer Vision and Pattern Recognition.
2. Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Models. Qirui Jiao, Daoyuan Chen, Yilun Huang, Yaliang Li, Ying Shen (2024). Computer Vision and Pattern Recognition.
3. UniDiff: Advancing Vision-Language Models with Generative and Discriminative Learning. Xiao Dong, Runhu Huang, Xiaoyong Wei, Zequn Jie, Jianxing Yu, Jian Yin, Xiaodan Liang (2023). arXiv.org.
4. WeThink: Toward General-purpose Vision-Language Reasoning via Reinforcement Learning. Jie Yang, Feipeng Ma, Zitian Wang, Dacheng Yin, Kang Rong, Fengyun Rao, Ruimao Zhang (2025). arXiv.org.
5. Vision-Language Models Can Self-Improve Reasoning via Reflection. Kanzhi Cheng, Yantao Li, Fangzhi Xu, Jianbing Zhang, Hao Zhou, Yang Liu (2024). North American Chapter of the Association for Computational Linguistics.