AI audio generation

Video-Foley is an innovative video-to-sound generation system that achieves highly controllable and synchronized video-sound synthesis by using root mean square (RMS) as the temporal event condition, combined with semantic timbre cues (audio or text). The system uses a label-free self-supervised learning framework, including two stages, Video2RMS and RMS2Sound, and combines novel concepts such as RMS discretization and RMS-ControlNet with a pre-trained text-to-audio model. Video-Foley delivers state-of-the-art performance in audio and video alignment and control of sound timing, intensity, timbre and detail.

MaskVAT is a video-to-audio (V2A) generative model that exploits the visual features of videos to generate realistic sounds that match the scene. This model places special emphasis on the synchronization of the starting point of the sound with the visual action to avoid unnatural synchronization problems. MaskVAT combines a full-band high-quality universal audio codec and a sequence-to-sequence mask generation model to achieve competitiveness comparable to non-codec generation audio models while ensuring high audio quality, semantic matching, and time synchronization.

vta-ldm is a deep learning model focused on video-to-audio generation, capable of generating audio content based on video content that is semantically and temporally aligned with the video input. It represents a new breakthrough in the field of video generation, especially after the significant progress in text-to-video generation technology. This model was developed by Manjie Xu and others from Tencent AI Lab. It has the ability to generate audio that is highly consistent with video content and has important application value in fields such as video production and audio post-processing.

Video-to-audio (V2A) technology is a DeepMind innovation that combines video pixels with natural language text cues to generate rich soundscapes synchronized with on-screen action. This technology can be combined with video generation models such as Veo to generate dramatic soundtracks for videos, realistic sound effects, or dialogue that matches the characters and tone of the video. It can also generate soundtracks for traditional material, including archival material, silent films, and more, opening up a wider range of creative opportunities.