AI speech synthesis

Fish Audio is a platform that provides text-to-speech conversion services. Using generative AI technology, users can convert text into natural and smooth speech. The platform supports voice cloning technology, allowing users to create and use personalized voices. It is suitable for a variety of scenarios such as entertainment, education and business, providing users with an innovative way of interaction.

Bailing-TTS is a large-scale text-to-speech (TTS) model series developed by Giant Network’s AI Lab that focuses on generating high-quality Chinese dialect speech. The model uses continuous semi-supervised learning and a specific Transformer architecture to effectively align text and speech tokens through a multi-stage training process to achieve high-quality speech synthesis in Chinese dialects. Bailing-TTS has demonstrated speech synthesis effects close to natural human expressions in experiments, which is of great significance to the field of dialect speech synthesis.

Pandrator is an open source software-based tool capable of converting text, PDF, EPUB and SRT files into speech audio in multiple languages, including speech cloning, LLM-based text preprocessing and saving the generated subtitle audio directly to the video file, mixed with the video's original audio track. It is designed to be easy to use and install, with a one-click installer and graphical user interface.

StreamVC is a real-time low-latency speech conversion solution developed by Google that can match the timbre of the target speech while maintaining the content and rhythm of the source speech. The technology is particularly suitable for real-time communication scenarios such as phone calls and video conferencing, and can be used for use cases such as voice anonymization. StreamVC utilizes the architecture and training strategy of the SoundStream neural audio codec to achieve lightweight and high-quality speech synthesis. It also demonstrates the effectiveness of learning soft speech units and providing whitened fundamental frequency information to improve pitch stability without leaking source timbre information.

CosyVoice is a large-scale multi-lingual speech generation model that not only supports speech generation in multiple languages, but also provides full-stack capabilities from inference to training to deployment. This model is important in the field of speech synthesis because it can generate natural and smooth speech that is close to real people and is suitable for multiple language environments. Background information on CosyVoice shows that it was developed by the FunAudioLLM team under the Apache-2.0 license.

FunAudioLLM is a framework designed to enhance natural speech interaction between humans and Large Language Models (LLMs). It contains two innovative models: SenseVoice is responsible for high-precision multilingual speech recognition, emotion recognition and audio event detection; CosyVoice is responsible for natural speech generation and supports multilingual, timbre and emotion control. SenseVoice supports more than 50 languages ​​and has extremely low latency; CosyVoice is good at multilingual voice generation, zero-sample context generation, cross-language voice cloning and command following capabilities. The relevant models have been open sourced on Modelscope and Huggingface, and the corresponding training, inference and fine-tuning codes have been released on GitHub.

SenseVoice is a basic speech model that includes multiple speech understanding capabilities such as automatic speech recognition (ASR), speech language recognition (LID), speech emotion recognition (SER), and audio event detection (AED). It focuses on high-precision multilingual speech recognition, speech emotion recognition and audio event detection, supports more than 50 languages, and its recognition performance exceeds the Whisper model. The model uses a non-autoregressive end-to-end framework with extremely low inference latency, making it ideal for real-time speech processing.

Fish Speech V1.2 is a text-to-speech (TTS) model trained on 300,000 hours of English, Chinese and Japanese audio data. This model represents the latest progress in speech synthesis technology and can provide high-quality speech output and is suitable for multiple language environments.

Azure Cognitive Services Speech is a speech recognition and synthesis service launched by Microsoft that supports speech-to-text and text-to-speech functions in more than 100 languages ​​and dialects. It improves the accuracy of your transcriptions by creating custom speech models that handle specific terminology, background noise, and accents. In addition, the service also supports real-time speech-to-text, speech translation, text-to-speech and other functions, and is suitable for a variety of business scenarios, such as subtitle generation, post-call transcription analysis, video translation, etc.

OpenVoice is an open source voice cloning technology that can accurately clone reference timbres and generate voices in multiple languages ​​and accents. It can flexibly control speech style parameters such as emotion and accent, as well as rhythm, pauses and intonation, etc. It implements zero-shot cross-language speech cloning, that is, neither the language of the generated speech nor the reference speech needs to appear in the training data.