AI model evaluation

Video-MME is a benchmark focused on performance evaluation of multimodal large language models (MLLMs) in the field of video analysis. It fills the gap in existing evaluation methods regarding the ability of MLLMs to process continuous visual data, and provides researchers with a high-quality and comprehensive evaluation platform. The benchmark covers videos of different lengths and evaluates the core capabilities of MLLMs.

This product is a tool for evaluating the quality of text-to-video generation. It introduces a new evaluation metric, text-to-video scoring (T2VScore). The score integrates two key criteria: (1) text-video alignment, which reviews the video's fidelity in presenting a given text description; and (2) video quality, which evaluates the overall production quality of the video. Furthermore, to evaluate the proposed metrics and facilitate future improvements to them, the product provides the TVGE dataset, which collects human judgments on both criteria on 2,543 text-to-video generated videos. Experiments on the TVGE dataset show that the proposed T2VScore exhibits superiority in providing better evaluation metrics for text-to-video generation.