Spirit LM

Spirit LM is a basic multi-modal language model that can freely mix text and speech. The model is based on a 7B pre-trained text language model and extends to speech modes by continuously training on text and speech units. Speech and text sequences are concatenated into a single token stream and trained using a small automatically curated speech-text parallel corpus using a word-level interleaved approach. Spirit LM has two versions: the basic version uses speech phoneme units (HuBERT), while the expression version uses pitch and style units in addition to phoneme units to simulate expressivity. For both versions, the text is encoded using subword BPE tokens. This model not only demonstrates the semantic capabilities of the text model, but also demonstrates the expressive capabilities of the speech model. Furthermore, we show that Spirit LM is able to learn new tasks (e.g., ASR, TTS, speech classification) across modalities with a small number of samples.

The target audience of Spirit LM is researchers and developers in the field of natural language processing (NLP), especially those interested in multi-modal language models. This product is suitable for them because it provides a powerful tool to process and understand data that mixes text and speech, which is crucial for developing more natural and intuitive human-computer interaction systems. In addition, it can help researchers quickly train and deploy new task models with a small number of samples, thus accelerating the research and development process.