AI 3D tools

GameGen-O is the first diffusion transformation model tailored for generating open-world video games. The model enables high-quality, open-domain generation by simulating multiple features of game engines, such as innovative characters, dynamic environments, complex actions, and diverse events. Additionally, it provides interactive controllability, allowing gameplay simulation. The development of GameGen-O involved a comprehensive data collection and processing effort from scratch, including building the first open world video game dataset (OGameData), efficiently sorting, scoring, filtering and decoupling titles through a proprietary data pipeline. This powerful and extensive OGameData forms the basis of the model training process.

VideoDoodles is an interactive system that simplifies the creation of video doodles by letting users place flat canvases in a 3D scene and then trace them. This technique allows hand-drawn animations to have correct perspective distortion and occlusion in video, and the ability to move as the camera and other objects in the scene move. The system enables users to finely control the canvas through a 2D image space UI, set position and orientation through keyframes, and automatically interpolate keyframes to track the motion of moving objects in the video.

Stable Video 4D is the latest AI model launched by Stability AI, which is able to convert a single object video into multiple novel view videos from eight different angles/views. This technology represents a leap in capabilities from image-based video generation to full 3D dynamic video synthesis. It has potential applications in areas such as game development, video editing, and virtual reality, and is being continuously optimized.

SpatialTracker, one of CVPR's 2024 highlights, works on recovering dense pixel motion in video in 3D space. The method estimates 3D trajectories by lifting 2D pixels into 3D space, using a three-plane representation to represent the 3D content of each frame, and iteratively updating the transformer. Tracking in 3D allows us to exploit rigid constraints while learning a rigid embedding that clusters pixels into different rigid parts. Compared with other tracking methods, SpatialTracker achieves excellent results in terms of both quality and measurement, especially in challenging cases with out-of-plane rotations.

SceneScript is a new 3D scene reconstruction technology developed by the Reality Labs research team. The technology uses AI to understand and reconstruct complex 3D scenes, enabling the creation of detailed 3D models from a single image. SceneScript significantly improves the accuracy and efficiency of 3D reconstruction by combining multiple advanced deep learning techniques, such as semi-supervised learning, self-supervised learning and multi-modal learning.

Meshy-2 is the latest addition to our 3D generative AI product family, coming three months after the release of Meshy-1. This version is a huge leap forward in the field of Text to 3D, providing better structured meshes and rich geometric details for 3D objects. In Meshy-2, Text to 3D offers four style options: Realistic, Cartoon, Low Polygon and Voxel to satisfy a variety of artistic preferences and inspire new creative directions. We've increased the speed of generation without compromising quality, with preview time around 25 seconds and fine results within 5 minutes. Additionally, Meshy-2 introduces a user-friendly mesh editor with polygon count control and a quad mesh conversion system to provide more control and flexibility in 3D projects. The Text to Texture feature has been optimized to render textures more clearly and twice as fast. Enhanced features of Image to 3D produce higher quality results in 2 minutes. We are shifting our focus from Discord to web applications, encouraging users to share AI-generated 3D art in the web application community.

Traditional 3D content creation tools give users direct control over scene geometry, appearance, motion and camera paths to bring their imaginations to life. However, creating computer-generated videos is a tedious manual process that can be automated through emerging text-to-video diffusion models. Although promising, video diffusion models are difficult to control, limiting users from applying their own creativity rather than amplifying it. To address this challenge, we propose a novel approach that combines the controllability of dynamic 3D meshes with the expressiveness and editability of emerging diffusion models. To this end, our approach takes an animated low-fidelity rendered mesh as input and injects ground-truth correspondence information obtained from the dynamic mesh into various stages of a pre-trained text-to-image generative model to output high-quality and temporally consistent frames. We demonstrate our approach on various examples where motion can be obtained by animating rigged assets or changing the camera path.