multimodal

Phi-4-multimodal-instruct is a multimodal basic model developed by Microsoft that supports text, image and audio input and generates text output. The model is built based on the research and data sets of Phi-3.5 and Phi-4.0, and undergoes processes such as supervised fine-tuning, direct preference optimization, and human feedback reinforcement learning to improve instruction compliance and safety. It supports text, image and audio input in multiple languages, has a context length of 128K, and is suitable for a variety of multi-modal tasks, such as speech recognition, speech translation, visual question answering, etc. The model has achieved significant improvements in multi-modal capabilities, especially on speech and visual tasks. It provides developers with powerful multi-modal processing capabilities that can be used to build various multi-modal applications.

Valley-Eagle-7B is a multi-modal large-scale model developed by Bytedance and is designed to handle a variety of tasks involving text, image and video data. The model achieved best results in internal e-commerce and short video benchmarks, and demonstrated superior performance compared to models of the same size in OpenCompass tests. Valley-Eagle-7B combines LargeMLP and ConvAdapter to build the projector, and introduces VisionEncoder to enhance the model's performance in extreme scenes.

Megrez-3B-Omni is an end-to-end full-modal understanding model developed by Wuwen Xinqiong. It is based on the large language model Megrez-3B-Instruct extension and has the ability to understand and analyze three modal data: pictures, text, and audio. This model achieves optimal accuracy in image understanding, language understanding, and speech understanding. It supports Chinese and English voice input and multiple rounds of dialogue, supports voice questions on input pictures, and directly responds to text based on voice commands. It has achieved leading results on multiple benchmark tasks.

InternVL 2.5 is a family of advanced multimodal large language models based on InternVL 2.0, which introduces significant enhancements in training and testing strategies and data quality while maintaining the core model architecture. This model provides an in-depth look at the relationship between model scaling and performance, systematically exploring performance trends for visual encoders, language models, dataset sizes, and test-time configurations. Through extensive evaluation on a wide range of benchmarks including multi-disciplinary reasoning, document understanding, multi-image/video understanding, real-world understanding, multi-modal hallucination detection, visual localization, multi-language capabilities and pure language processing, InternVL 2.5 has demonstrated competitiveness on par with leading commercial models such as GPT-4o and Claude-3.5-Sonnet. In particular, the model is the first open source MLLM to exceed 70% on the MMMU benchmark, achieve a 3.7 percentage point improvement via chain-of-thinking (CoT) inference, and demonstrate strong potential for test-time scaling.