When Not to Think with Video: An Adaptive Gating Framework and Diagnostic Benchmark

TL;DR: Sometimes drawing a movie in your head hurts—like for exact arithmetic or purely symbolic logic. We propose an “adaptive gate” that decides when to think in text, image, or video; the first experiment builds CounterVideoThinkBench with tasks that penalize unnecessary videoization and tests whether gating improves accuracy and safety.

Research Question: What are the failure regimes of video-of-thought, and can a learned controller decide when to avoid video reasoning in favor of text or symbolic reasoning for better accuracy, efficiency, and safety?

Hypothesis: Video-of-thought excels on dynamic, spatial, and embodied commonsense but degrades performance on high-precision symbolic tasks and long-form discourse; an adaptive gating policy—trained with meta-RL or uncertainty signals—will outperform always-video baselines while reducing unsafe generations.

Experiment Plan: - Setup:

• Controller: A small policy network predicts the modality of thought (text/image/video ± neurosymbolic program) given the query and early features; informed by uncertainty, task taxonomy, and safety detectors.
• Reasoners: (a) video-of-thought; (b) text CoT; (c) neurosymbolic automata for formal temporal/logical subproblems; (d) VURF-style visual programs for certain video tasks.
• Data/Materials:
  • CounterVideoThinkBench: curated from TemporalBench (fine temporal), VSI-Bench (spatial maps), CinePile (long-form), math/text problems (from VideoThinkBench text-centric), plus red-team prompts for safety assessment.
  • T2VShield pipeline to evaluate jailbreak robustness for any video-generative steps.
• Measurements:
  • Accuracy, compute cost, and latency; safety violation rate (visual-level); error taxonomy vs. chosen modality; ablations with and without neurosymbolic branch.
• Expected Outcomes:
  • The gated system matches or beats the best single-modality baseline across mixed workloads, reducing unnecessary video generation, improving exactness on symbolic tasks, and lowering jailbreak success in pipelines that avoid video when risky.

References: ['Tong, J., Mou, Y., Li, H., Li, M., Yang, Y., Zhang, M., Chen, Q., Liang, T., Hu, X., Zheng, Y., Chen, X., Zhao, J., Huang, X., & Qiu, X. (2025). Thinking with Video: Video Generation as a Promising Multimodal Reasoning Paradigm.', 'Cai, M., Tan, R., Zhang, J., Zou, B., Zhang, K., Yao, F., Zhu, F., Gu, J., Zhong, Y., Shang, Y., Dou, Y., Park, J., Gao, J., Lee, Y. J., & Yang, J. (2024). TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models. arXiv.org.', 'Yang, J., Yang, S., Gupta, A. W., Han, R., Li, F.-F., & Xie, S. (2024). Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces. Computer Vision and Pattern Recognition.', 'Rawal, R., Saifullah, K., Basri, R., Jacobs, D., Somepalli, G., & Goldstein, T. (2024). CinePile: A Long Video Question Answering Dataset and Benchmark. arXiv.org.', 'Sasaki, S., Lopez, D. M., & Johnson, T. T. (2025). Neurosymbolic Finite and Pushdown Automata: Improved Multimodal Reasoning versus Vision Language Models (VLMs). NeuS.', 'Mahmood, A., Vayani, A., Naseer, M., Khan, S. H., & Khan, F. (2024). VURF: A General-purpose Reasoning and Self-refinement Framework for Video Understanding. arXiv.org.', 'Liang, S., Liu, J., Zhai, J., Fang, T., Tu, R., Liu, A., Cao, X., & Tao, D. (2025). T2VShield: Model-Agnostic Jailbreak Defense for Text-to-Video Models. arXiv.org.']