Vid-RAG: Knowledge-Graph–Augmented Video Thinking

TL;DR: Before a model “thinks in video,” let it look up relevant real-world concepts (with text, images, audio, and video) and then imagine a video that obeys that knowledge. As a first experiment, we plug VAT-KG retrieval into the prompt of a video-of-thought generator and test on SOK-Bench and TemporalCook; we hypothesize large gains on situated and procedural reasoning.

Research Question: Can retrieval-augmented generation with a multimodal knowledge graph (visual–audio–text) improve the factuality, situated commonsense, and procedural coherence of video-of-thought reasoning?

Hypothesis: Conditioning video-of-thought on VAT-KG concept-level evidence will reduce hallucinations, improve alignment to real-world constraints, and enhance performance on tasks that require integrating situated visual context with broad knowledge.

Experiment Plan: - Setup:

• Retrieval: VAT-KG retrieves concept nodes and linked media given the query and context frames.
• Fusion: A CREMA-like modular fusion to project retrieved multimodal features into the unified model’s token space; optionally pre-encode retrieved clips as “evidence shots” appended to the imagined video.
• Generator: BAGEL or compatible unified model; prompting includes retrieved snippets and structured concept descriptions.
• Data/Materials:
  • SOK-Bench (situated + open-world knowledge reasoning).
  • TemporalCook (temporal/procedural predictions from a single image with optional external instructional videos).
  • FortisAVQA for audio–visual knowledge grounding.
• Measurements:
  • QA accuracy; rationale faithfulness to retrieved nodes; text relevance and naturalness for generated video plans; robustness under distribution shift (FortisAVQA rare vs. frequent splits).
• Expected Outcomes:
  • Significant improvements on SOK-Bench and TemporalCook with evidence-conditioned video-of-thought vs. no retrieval.
  • Clear reduction in knowledge-conflicting hallucinations, as judged by human and automated fact-consistency checks.

References: ['Park, H., Jang, M., Baek, H., Chang, G., Seo, J., Park, J., Park, H., & Kim, S. (2025). VAT-KG: Knowledge-Intensive Multimodal Knowledge Graph Dataset for Retrieval-Augmented Generation. arXiv.org.', 'Wang, A., Wu, B., Chen, S., Chen, Z., Guan, H., Lee, W.-N., Li, L. E., Tenenbaum, J. B., & Gan, C. (2024). SOK-Bench: A Situated Video Reasoning Benchmark with Aligned Open-World Knowledge. Computer Vision and Pattern Recognition.', 'Zarei, M., Akkasi, A., Hassan, M., & Komeili, M. (2025). TemporalCook: Benchmarking Temporal and Procedural Reasoning in Multimodal Large Language Models. Proceedings of the 1st International Workshop on MLLM for Unified Comprehension and Generation.', 'Deng, C., Zhu, D., Li, K., Gou, C., Li, F., Wang, Z., Zhong, S., Yu, W., Nie, X., Song, Z., Guang, S., & Fan, H. (2025). Emerging Properties in Unified Multimodal Pretraining. arXiv.org.', 'Yu, S., Yoon, J., & Bansal, M. (2024). CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusion. International Conference on Learning Representations.', 'Tong, J., Mou, Y., Li, H., Li, M., Yang, Y., Zhang, M., Chen, Q., Liang, T., Hu, X., Zheng, Y., Chen, X., Zhao, J., Huang, X., & Qiu, X. (2025). Thinking with Video: Video Generation as a Promising Multimodal Reasoning Paradigm.']