AI video generation

Vmotionize is a leading AI animation and 3D animation software that can convert videos, music, text, pictures and other content into stunning 3D animations. The platform makes high-quality 3D content and motion graphics more accessible through advanced AI animation and motion capture tools. Vmotionize innovatively provides a new platform for independent creators and global brands to realize creativity, share stories and build virtual worlds through artificial intelligence and human imagination.

Dynamic Typography is an automated text animation solution that combines the two challenging tasks of semantic communication and dynamic motion. The technology leverages vector graphics representation and an end-to-end optimization framework to convert letters into underlying shapes via neural displacement fields and applies per-frame motion to enhance consistency with the intended text concept. Maintain readability and structural integrity during animation through shape-preserving techniques and perceptual loss regularization. Our method demonstrates generality across a variety of text-to-video models and highlights the superiority of our end-to-end approach, which may include separate tasks. Through quantitative and qualitative evaluations, we demonstrate the effectiveness of our framework in generating coherent text animations that faithfully explain user prompts while maintaining readability.

Stable Video 3D is a new model launched by Stability AI, which has made significant progress in the field of 3D technology, providing greatly improved quality and multi-view support compared to the previously released Stable Zero123. The model is able to generate orbital videos based on a single image input without camera conditions, and is able to create 3D videos along specified camera paths.

Media2Face is a co-linguistic facial animation generation tool guided by audio, text and image multimodality. It first utilizes General Neural Parameterized Facial Assets (GNPFA) to map facial geometry and images into a highly general expression latent space, and then extracts high-quality expressions and accurate head poses from a large number of videos to construct the M2F-D dataset. Finally, the diffusion model in the GNPFA latent space is adopted for co-linguistic facial animation generation. The tool not only delivers high fidelity in facial animation synthesis, but also expands expressiveness and style adaptability.

4D-fy is a text-to-4D generation method that uses mixed fractional distillation sampling technology and combines the supervision signals of multiple pre-trained diffusion models to achieve high-fidelity text-to-4D scene generation. Its approach parametrizes 4D radiation fields through neural representations, uses static and dynamic multi-scale hash table features, and utilizes volume rendering to render images and videos from the representations. By mixed fractional distillation sampling, gradients of a 3D-aware text-to-image model (3D-T2I) are first used to optimize representation, then gradients of a text-to-image model (T2I) are combined to improve appearance, and finally gradients of a text-to-video model (T2V) are combined to increase the motion of the scene. 4D-fy can generate 4D scenes with attractive appearance, 3D structure and motion.

LiveSketch is a tool for adding animation effects to hand-drawn sketches. It can automatically generate vector animations based on text prompts to bring sketches to life. The tool does not require complex training and uses a pre-trained text-to-video model to guide the movement of strokes. It is suitable for designers, animators and other users who need to add animation effects to sketches. Animated drawings can be used on the website.

MCVD is a general model for video generation, prediction and interpolation. It uses a score-based diffusion loss function to generate novel frames. By injecting Gaussian noise into the current frame and conditionally denoising past and/or future frames, it is trained by randomly masking past and/or future frames to achieve four cases of unconditional generation, future prediction, past reconstruction and interpolation. The model uses a 2D convolutional U-Net to condition past and future frames through concatenation or spatiotemporal adaptive normalization to produce high-quality and diverse video samples. It is trained using 1-4 GPUs and can be scaled to more channels. MCVD is a simple non-recursive 2D convolutional architecture capable of generating video samples of arbitrary length with SOTA results.

Story-to-Motion is a brand new task that takes a story (top green area) and generates actions and trajectories that match the text description. The system utilizes modern large-scale language models as text-driven motion schedulers to extract a sequence of (text, position) pairs from long texts. It also develops a text-driven motion retrieval scheme that combines classical motion matching with motion semantics and trajectory constraints. In addition, it is designed with a progressive masking transformer to solve common problems in transition movements, such as unnatural postures and sliding steps. The system performs well in the evaluation of three different subtasks: trajectory following, temporal action combination and action blending, outperforming previous action synthesis methods.