Any GPT

AnyGPT is a unified multi-modal large-scale language model that uses discrete representations to perform unified processing of various modalities, including speech, text, images and music. AnyGPT can stabilize training without changing the current large language model architecture or training paradigm. It relies entirely on data-level preprocessing, which promotes the seamless integration of new modalities into language models, similar to the addition of new languages. We construct a text-centric multimodal dataset for multimodal alignment pre-training. Leveraging generative models, we synthesize the first large-scale any-to-any multimodal instruction dataset. It consists of 108,000 multi-turn dialogue examples, with multiple modalities intertwined, thus enabling the model to handle any combination of multi-modal input and output. Experimental results show that AnyGPT can promote any-to-any multi-modal dialogue while achieving performance comparable to dedicated models in all modalities, proving that discrete representations can effectively and conveniently unify multiple modalities in language models.