Temporal Self-Consistency for “Video-of-Thought”

TL;DR: Let the model “imagine” multiple short videos as its thoughts and only trust answers that are consistent across those imagined futures. Concretely, we sample k video rollouts as internal chains-of-thought, score them with a temporal-consistency verifier, and aggregate answers via majority vote; our first controlled experiment compares single-sample vs k-sample video-of-thought on VideoThinkBench and StreamingCoT, hypothesizing that temporal self-consistency closes a significant portion of the temporal-reasoning gap.

Research Question: Does sampling and selecting among multiple video “thought” rollouts (temporal self-consistency) improve multimodal reasoning compared to a single video-of-thought, particularly on fine-grained temporal and streaming tasks?

Hypothesis: Temporal self-consistency—i.e., generating multiple imagined videos and selecting the answer from rollouts that pass a temporal verifier—yields significant gains on temporal ordering, state transitions, and causal reasoning, outperforming single-rollout video-of-thought. Combining self-consistency with neurosymbolic temporal constraints will further improve reliability.

Experiment Plan: - Setup:

• Base generator: a unified model capable of video generation and reasoning (e.g., BAGEL or an open-source DiT-based video generator) prompted in the “Thinking with Video” style.
• Verifier: a lightweight temporal-consistency module combining (a) a streaming CoT interpreter trained on StreamingCoT, (b) fine-grained temporal probes from TemporalBench, and (c) a neurosymbolic checker that encodes temporal predicates with finite/pushdown automata (e.g., event order, non-overlap, persistence).
• Data/Materials:
  • VideoThinkBench (vision- and text-centric subsets) for baseline comparability with Tong et al.
  • StreamingCoT for evolving-answer, streaming VideoQA with explicit CoT.
  • TemporalBench for fine-grained temporal understanding (frequency, order, motion magnitude).
• Measurements:
  • Accuracy on benchmark tasks; temporal order accuracy; Multiple Binary Accuracy (MBA) from TemporalBench; StreamingCoT CoT quality alignment; verifier acceptance rate vs. task accuracy; cost-performance trade-off vs. number of rollouts (k).
• Expected Outcomes:
  • k>1 substantially improves accuracy on temporal tasks; verifier acceptance correlates with correctness.
  • Neurosymbolic constraints reduce failure modes where videos “look plausible” but violate temporal logic.
  • Ablations show the temporal verifier contributes more on streaming and fine-grained dynamics than on static tasks.

References: ['Tong, J., Mou, Y., Li, H., Li, M., Yang, Y., Zhang, M., Chen, Q., Liang, T., Hu, X., Zheng, Y., Chen, X., Zhao, J., Huang, X., & Qiu, X. (2025). Thinking with Video: Video Generation as a Promising Multimodal Reasoning Paradigm.', 'Hu, Y., Yang, Z., Wang, S., Qian, S., Wen, B., Yang, F., Gao, T., & Xu, C. (2025). StreamingCoT: A Dataset for Temporal Dynamics and Multimodal Chain-of-Thought Reasoning in Streaming VideoQA. Proceedings of the 33rd ACM International Conference on Multimedia.', 'Cai, M., Tan, R., Zhang, J., Zou, B., Zhang, K., Yao, F., Zhu, F., Gu, J., Zhong, Y., Shang, Y., Dou, Y., Park, J., Gao, J., Lee, Y. J., & Yang, J. (2024). TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models. arXiv.org.', 'Deng, C., Zhu, D., Li, K., Gou, C., Li, F., Wang, Z., Zhong, S., Yu, W., Nie, X., Song, Z., Guang, S., & Fan, H. (2025). Emerging Properties in Unified Multimodal Pretraining. arXiv.org.', 'Sasaki, S., Lopez, D. M., & Johnson, T. T. (2025). Neurosymbolic Finite and Pushdown Automata: Improved Multimodal Reasoning versus Vision Language Models (VLMs). NeuS.', 'Mahmood, A., Vayani, A., Naseer, M., Khan, S. H., & Khan, F. (2024). VURF: A General-purpose Reasoning and Self-refinement Framework for Video Understanding. arXiv.org.']