Adaptive Evaluation Agent: Integrating VBench’s Granularity with Human-Like Efficiency

VBench (Huang et al., 2023/2024) offers exhaustive evaluation but is computationally expensive, while Evaluation Agent (Zhang et al., 2024) prioritizes speed. This hybrid framework uses the agent’s human-like sampling to detect potential weaknesses (e.g., temporal flickering), then triggers targeted VBench metrics for deep analysis. For instance, if the agent flags motion inconsistencies, VBench’s motion-smoothness module activates. This resolves the conflict between depth and speed by making evaluation context-aware. Unlike existing pipelines, it adapts to model-specific failures, reducing evaluation time by 70% while retaining VBench’s rigor.

References:

1. VBench: Comprehensive Benchmark Suite for Video Generative Models. Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, Ziwei Liu (2023). Computer Vision and Pattern Recognition.
2. Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative Models. Fan Zhang, Shulin Tian, Ziqi Huang, Yu Qiao, Ziwei Liu (2024). Annual Meeting of the Association for Computational Linguistics.