AI model evaluation

This tool aims to improve the transparency of MLLMs by conducting qualitative research on the latest proprietary and open-source MLLMs, assessing their generalization ability, credibility, and causal reasoning capabilities from the perspectives of four modalities: text, code, image, and video. We believe these properties are several representative factors that define the reliability of MLLMs, supporting various downstream applications. Specifically, we evaluate the closed-source GPT-4 and Gemini and 6 open-source LLMs and MLLMs. In total, we evaluated 230 manually designed cases and the qualitative results were summarized into 12 scores (i.e. 4 modalities times 3 attributes). In total, we reveal 14 empirical findings that contribute to understanding the capabilities and limitations of proprietary and open source MLLMs to more reliably support multimodal downstream applications.