video generation

GameGen-X is a diffusion transformation model designed for generating and interactively controlling open-world game videos. This model achieves high-quality, open-domain video generation by simulating various features of game engines, such as innovative characters, dynamic environments, complex actions, and diverse events. In addition, it provides interactive control capabilities to predict and change future content based on current video clips, enabling gameplay simulation. To realize this vision, we first collected and built from scratch an Open World Video Game Dataset (OGameData), the first and largest open world game video generation and control dataset, containing more than 1 million diverse gameplay video clips from more than 150 games, all equipped with informative subtitles from GPT-4o. GameGen-X has gone through a two-stage training process, including basic model pre-training and instruction tuning. First, the model is pre-trained with text-to-video generation and video sequels, giving it the ability to generate long-sequence, high-quality open-domain game videos. Further, in order to achieve interactive control capabilities, we designed InstructNet to integrate game-related multi-modal control signal experts. This enables the model to adapt latent representations based on user input, unifying character interaction and scene content control in video generation for the first time. During instruction tuning, only InstructNet is updated, while the pre-trained base model is frozen, allowing the integration of interactive control capabilities without losing the diversity and quality of the generated video content. GameGen-X represents a major leap forward in using generative models for open-world video game design. It demonstrates the potential of generative models as an adjunct to traditional rendering techniques, effectively combining creative generation with interactive capabilities.

ComfyUI-MochiWrapper is a wrapper node for the Mochi video generator, which allows users to interact with Mochi models through the ComfyUI interface. The main advantage of this project is that it can use Mochi models to generate video content and simplify the operation process through ComfyUI. It is developed based on Python and is completely open source, allowing developers to use and modify it freely. The project is currently under active development and already has some basic functions, but has not yet officially released a version.