AI video editing

Animate-X is a universal LDM-based animation framework for various character types (collectively referred to as X), including human mimic characters. This framework enhances motion representation by introducing pose indicators, which can more comprehensively capture motion patterns from driving videos. Key benefits of Animate-X include in-depth modeling of motion, the ability to understand the motion patterns driving video and flexibly apply them to target characters. In addition, Animate-X introduces a new Animated Anthropomorphic Benchmark (A2Bench) to evaluate its performance on general and widely applicable animated images.

DressRecon is a method for reconstructing temporally consistent 4D human models from monocular videos, focusing on handling very loose clothing or handheld object interactions. The technique combines general prior knowledge of the human body (learned from large-scale training data) with specific "bag-of-skeleton" deformations for individual videos (fitted through test-time optimization). DressRecon separates body and clothing deformations by learning a neural implicit model as separate motion model layers. To capture the subtle geometry of clothing, it leverages image-based prior knowledge such as human pose, surface normals, and optical flow to adjust during the optimization process. The generated neural fields can be extracted into temporally consistent meshes or further optimized into explicit 3D Gaussians to improve rendering quality and enable interactive visualization. DressRecon delivers higher 3D reconstruction fidelity than previous techniques on datasets containing highly challenging clothing deformations and object interactions.

FaceFusion is an industry-leading facial manipulation platform specializing in face swapping, lip sync, and deep manipulation technologies. It utilizes advanced artificial intelligence technology to provide users with a highly realistic facial operation experience. FaceFusion has a wide range of applications in image processing and video production, especially in the entertainment and media industries.

PortraitGen is a 2D portrait video editing tool based on multi-modal generation priors. It can upgrade 2D portrait videos to 4D Gaussian fields to achieve multi-modal portrait editing. This technology can quickly generate and edit 3D portraits by tracking SMPL-X coefficients and using a neural Gaussian texture mechanism. It also proposes an iterative dataset update strategy and multi-modal face-aware editing module to improve expression quality and maintain personalized facial structure.

Robust Dual Gaussian Splatting (DualGS) is a novel Gaussian-based volumetric video representation method that captures complex human performances by optimizing joint Gaussians and skin Gaussians and achieves robust tracking and high-fidelity rendering. Demonstrated at SIGGRAPH Asia 2024, the technology enables real-time rendering on low-end mobile devices and VR headsets, providing a user-friendly and interactive experience. DualGS achieves a compression ratio of up to 120 times through a hybrid compression strategy, making the storage and transmission of volumetric videos more efficient.

Svd Keyframe Interpolation is a keyframe interpolation model based on singular value decomposition (SVD) technology, which is used to automatically generate intermediate frames in animation production, thereby improving the work efficiency of animators. This technology automatically calculates the images of intermediate frames by analyzing the characteristics of key frames, making the animation more smooth and natural. Its advantage is that it reduces the animator's workload of manually drawing in-between frames while maintaining high-quality animation effects.

This product is an image-to-video diffusion model that can generate continuous video sequences with coherent motion from a pair of key frames through lightweight fine-tuning technology. This method is particularly suitable for scenarios where a smooth transition animation needs to be generated between two static images, such as animation production, video editing, etc. It leverages the power of large-scale image-to-video diffusion models by fine-tuning them to predict videos between two keyframes, achieving forward and backward consistency.

ComfyUI-AdvancedLivePortrait is an advanced tool for real-time preview and editing of facial expressions. It allows users to track and edit faces in videos, insert expressions into videos, and even extract expressions from sample photos. This project simplifies the installation process by automating the installation using ComfyUI-Manager. It combines image processing and machine learning technologies to provide users with a powerful tool for creating dynamic and interactive media content.

Segment Anything 2 for Surgical Video Segmentation is a surgical video segmentation model based on Segment Anything Model 2. It uses advanced computer vision technology to automatically segment surgical videos to identify and locate surgical tools, improving the efficiency and accuracy of surgical video analysis. This model is suitable for various surgical scenarios such as endoscopic surgery and cochlear implant surgery, and has the characteristics of high accuracy and high robustness.

Live_Portrait_Monitor is an open source project designed to animate portraits via a monitor or webcam. This project is based on the LivePortrait research paper and uses deep learning technology to efficiently implement portrait animation through splicing and redirection control. The author is actively updating and improving this project and is for research use only.

LivePortrait is an efficient tool for portrait animation that transforms static images into vivid animations through splicing and redirection control technology. This technology is of great significance in the fields of image processing and animation production, and can greatly improve the efficiency and quality of animation production. Product background information shows that it was developed by shadowcz007 and used in conjunction with comfyui-mixlab-nodes to better achieve portrait animation effects.

MASA is an advanced model for object matching in video frames, which is capable of handling multi-object tracking (MOT) in complex scenes. MASA does not rely on domain-specific annotated video data sets, but learns instance-level correspondences through rich object segmentation through the Segment Anything Model (SAM). MASA has designed a universal adapter that can be used with basic segmentation or detection models to achieve zero-shot tracking capabilities that can perform well even in complex domains.

CameraCtrl is dedicated to providing precise camera pose control for text generation video models, and achieves camera control in the video generation process by training camera encoders to achieve parameterized camera trajectories. By comprehensively studying the effects of various data sets, the product proves that videos with diverse camera distributions and similar appearances can enhance controllability and generalization capabilities. Experiments have proven that CameraCtrl is very effective in achieving precise, domain-adaptive camera control, and is an important advance in enabling dynamic, customized video storytelling from text and camera gesture input.

GoEnhance AI is a video-to-video, image enhancement and upscaling platform. It can convert your videos into many different styles of animation, including pixel and flat animation. Through AI technology, it is able to enhance and upgrade images to the ultimate level of detail. Whether it's personal creation or commercial application, GoEnhance AI provides you with powerful image and video editing tools.

Move API can convert videos containing human body movements into 3D animation assets, supports converting video files to usdz, usdc and fbx file formats, and provides preview videos. Ideal for integrating into production workflow software, enhancing application motion capture capabilities, or creating new experiences.

VisFusion is a technology that uses video data for online 3D scene reconstruction. It can extract and reconstruct a 3D environment from videos in real time. This technology combines computer vision and deep learning to provide users with a powerful tool for creating accurate 3D models.

PRISMA is a computational photography pipeline that can perform multiple inferences from any image or video. Just like light refracted into different wavelengths through a prism, this pipeline expands the image into data that can be used for 3D reconstruction or real-time post-processing operations. It combines different algorithms and open source pre-trained models, such as monocular depth (MiDAS v3.1, ZoeDepth, Marigold, PatchFusion), optical flow (RAFT), segmentation mask (mmdet), camera pose (colmap), etc. The resulting bands are stored in a folder with the same name as the input files, with each band stored individually as a .png or .mp4 file. For videos, in the last step, it tries to perform sparse reconstruction, which can be used for NeRF (such as NVidia's Instant-ngp) or Gaussian diffusion training. The inferred depth information is exported by default as a heat map that can be decoded in real time using LYGIA's heatmap GLSL/HLSL sampling, while the optical flow is encoded as HUE (angle) and saturation, which can also be decoded in real time using LYGIA's optical flow GLSL/HLSL sampler.

DragNUWA is a video generation tool that can convert actions into camera movements or target object movements by directly manipulating backgrounds or images to generate corresponding videos. DragNUWA 1.5 is based on stabilized video diffusion technology that animates images according to specific paths. DragNUWA 1.0 utilizes text, images, and trajectories as three important control factors to facilitate highly controllable video generation semantically, spatially, and temporally. Users can clone the warehouse through git, download the pre-trained model, and drag and drop images on the desktop to generate animations.

Wild2Avatar is a neural rendering method for rendering human appearance in occluded wild monocular videos. It can render humans in realistic scenes, even if obstacles may block the camera view and cause partial occlusion. This method is implemented by decomposing the scene into three parts (occlusions, humans and background) and using a specific objective function to enforce the separation of humans from occlusions and background to ensure the integrity of the human model.

AnimateZero is a zero-sample image animation generator that generates videos by separating appearance and motion, solving problems such as black box, inefficiency, and uncontrollability. It can convert the pre-trained T2V model into an I2V model through zero-sample modification, thereby achieving zero-sample image animation generation. AnimateZero can also be used in scenarios such as video editing, frame interpolation, loop video generation and real image animation, with high subjective quality and matching.

FLATTEN is an optical flow-guided attention plugin for text-to-video editing. It solves the consistency problem in text-to-video editing by introducing optical flow in U-Net of diffusion model. FLATTEN improves the visual consistency of edited videos by forcing patches on the same optical flow path on different frames to pay attention to each other in the attention module. Furthermore, FLATTEN is training-free and can be seamlessly integrated into any diffusion-based text-to-video editing method and improves its visual consistency. Experimental results show that our proposed method achieves state-of-the-art performance on existing text-to-video editing benchmarks. In particular, our method performs well in maintaining the visual consistency of edited videos.

Imitator is a novel approach to personalized voice-driven 3D facial animation. Given an audio sequence and a personalized style embedding as input, we generate individual-specific motion sequences with accurate lip closure for bilabial consonants ('m', 'b', 'p'). The subject's style embedding can be calculated from a short reference video (e.g. 5 seconds).

ProPainter is an advanced model for video restoration. It combines enhanced propagation and Transformer mechanisms to quickly and efficiently perform tasks such as video repair, object removal, and watermark removal. ProPainter improves performance and efficiency through dual-domain propagation and sparse Transformers, which can significantly improve PSNR values ​​by 1.46 dB while maintaining good results. This model is suitable for a wide range of video repair scenarios, with flexible and reasonable pricing.

Frame Interpolation is a high-quality inter-frame interpolation neural network model. The model adopts a unified single-network approach that does not require additional pre-trained networks such as optical flow or deep networks, but still achieves state-of-the-art results. The model uses a multi-scale feature extractor that shares the same convolution weights at different scales. The model is trained only on frame triples.

Wonder Studio is an AI tool that automatically animates, lights and composites CG characters into live action scenes. Users only need a camera, and there is no need to use complex 3D software or expensive production hardware. By uploading a CG character model to one or more shots, the system automatically detects the clip and tracks the actor throughout the sequence. The system automatically detects the actor's performance based on a single camera lens, and then transfers the performance to the user-selected CG character for automatic animation, lighting and compositing. Wonder Studio can be widely used in film production, special effects production and other fields.