AI audio editing

Podcastfy is an open source Python package that uses generative artificial intelligence technology to transform web content, PDF files, and text into engaging multilingual audio conversations. Unlike traditional user interface-based tools, Podcastfy focuses on programmatic and customized generation of engaging, conversational audio and text from multiple text sources, enabling customization and scale.

seed-vc is a sound conversion model based on the SEED-TTS architecture, which can achieve zero-sample sound conversion, that is, the sound can be converted without the need for a specific person's voice sample. This technology performs well in terms of audio quality and timbre similarity, and has high research and application value.

Whisper-diarization is an open source project that combines Whisper's automatic speech recognition (ASR) capabilities, vocal activity detection (VAD), and speaker embedding technology. It improves the accuracy of speaker embeddings by extracting the sound parts in the audio, then using Whisper to generate transcripts and correcting timestamps and alignments with WhisperX to reduce segmentation errors due to time offsets. Next, MarbleNet is used for VAD and segmentation to exclude silence, TitaNet is used to extract speaker embeddings to identify the speaker of each paragraph, and finally the results are associated with timestamps generated by WhisperX, the speaker of each word is detected based on the timestamp, and realigned using a punctuation model to compensate for small temporal shifts.

Audio Isolation is an online audio processing service provided by ElevenLabs that focuses on separating vocals or background music from audio. This technology has important application value in fields such as music production and video post-production, and can significantly improve the efficiency and quality of audio editing. The product provides services through API, supports calls in multiple programming languages, and is highly flexible and convenient. In terms of pricing, the API is charged per minute based on the number of audio characters processed, and the specific price is not clearly marked on the page.

AudioSeal is a localized watermarking technology for AI-generated speech audio with state-of-the-art robustness and extremely fast detection speed. By jointly training a watermark-embedded generator and a detector, it can detect watermarked segments in longer audio even in the presence of audio editing. AudioSeal designed a fast single-pass detector that is two orders of magnitude faster than existing models, making it ideal for large-scale and real-time applications.

LookOnceToHear is an innovative smart headphone interaction system that allows users to select the target speaker they want to hear through simple visual recognition. This technology received an honorable mention for Best Paper at CHI 2024. It achieves real-time speech extraction by synthesizing audio mixes, head-related transfer functions (HRTFs) and binaural room impulse responses (BRIRs), providing users with a novel way to interact.