research tools

Wan2.2 Animate is a free online advanced AI character animation tool. It is developed based on cutting-edge research and rigorous academic research results of Alibaba Tongyi Laboratory. It uses open source technology and model weights are available on the Hugging Face and ModelScope platforms. Its main advantage is that it provides precise facial expression control, body movement copying, seamless character replacement and other functions. It can create character animations while maintaining the original movements, environmental background and lighting conditions. It does not require registration and can be run directly in the browser. It is suitable for academic research, effect display and creative experiments.

CameraBench is a model for analyzing camera motion in video, aiming to understand camera motion patterns through video. Its main advantage lies in utilizing generative visual language models for principle classification of camera motion and video text retrieval. By comparing with traditional structure-from-motion (SfM) and real-time localization and construction (SLAM) methods, the model shows significant advantages in capturing scene semantics. The model is open source and suitable for use by researchers and developers, and more improved versions will be released in the future.

LongVU is an innovative long video language understanding model that reduces the number of video tags through a spatiotemporal adaptive compression mechanism while retaining visual details in long videos. The importance of this technology lies in its ability to process a large number of video frames with only a small loss of visual information within the limited context length, significantly improving the ability to understand and analyze long video content. LongVU outperforms existing methods on multiple video understanding benchmarks, especially on the task of understanding hour-long videos. Additionally, LongVU is able to efficiently scale to smaller model sizes while maintaining state-of-the-art video understanding performance.

Movie Gen Bench is a video generation evaluation benchmark published by Facebook Research, aiming to provide a fair and easily comparable standard for future research in the field of video generation. The benchmark test includes two parts, Movie Gen Video Bench and Movie Gen Audio Bench, which evaluate video content generation and audio generation respectively. The release of Movie Gen Bench is of great significance to promote the development and evaluation of video generation technology. It can help researchers and developers better understand and improve the performance of video generation models.

DenseAV is a novel dual-encoder localization architecture that learns high-resolution, semantically meaningful audio-visual alignment features by watching videos. It is able to discover the "meaning" of words and the "place" of sounds without explicit localization supervision, and automatically discovers and differentiates between these two types of associations. DenseAV’s localization capabilities come from a new multi-head feature aggregation operator that directly compares dense image and audio representations for comparative learning. Furthermore, DenseAV significantly surpasses the previous state-of-the-art on semantic segmentation tasks and outperforms ImageBind on cross-modal retrieval using less than half the parameters.

Ego-Exo4D is a multi-modal multi-view video dataset and benchmark challenge centered on capturing egocentric and exocentric videos of skilled human activities. It supports multimodal machine perception research in daily life activities. The dataset was collected by 839 camera-wearing volunteers in 13 cities around the world, capturing 1,422 hours of video of skilled human activity. This dataset provides three natural language datasets such as expert comments, tutorial-style narratives provided by participants, and atomic action descriptions in one sentence, paired with videos. Ego-Exo4D also captures multiple viewpoints and multiple perception modalities, including multiple viewpoints, seven microphone arrays, two IMUs, a barometer, and a magnetometer. Datasets were recorded in strict compliance with privacy and ethics policies and with formal consent from participants. For more information, please visit the official website.