We released TontaubeV1, a character-level TTS model for long-form generation [P]
Our take
![We released TontaubeV1, a character-level TTS model for long-form generation [P]](https://preview.redd.it/dq70r0hwiwmh1.png?width=140&height=83&auto=webp&s=b5c68e7aa20f1aba3177bdc9769e7025694baf89)
The release of TontaubeV1 by craitech is a noteworthy development in the rapidly evolving landscape of text-to-speech (TTS) models, particularly for those seeking expressive, long-form narration capabilities. This 2.9B-parameter open-weight model, primarily focused on English and German, stands out not just for its performance—achieving 50.1% against ElevenLabs Flash v2.5 in a recent LLM-as-a-judge benchmark—but also for the innovative design choices underpinning its architecture. The focus on character-level tokenization, a departure from the more common BPE tokenizer approach seen in many LLM-based TTS models, is particularly compelling. This decision, coupled with their unique chunking and position scheme, addresses challenges inherent in generating coherent, high-quality audio for extended passages, a problem that has plagued many previous attempts. It's interesting to consider this in light of Perplexity’s recent launch of hybrid compute, Your files stay put: Perplexity’s hybrid AI keeps confidential data off the cloud, which highlights the growing demand for secure and efficient AI processing, potentially impacting the resource requirements and deployment strategies for models like TontaubeV1.
The character-level tokenization strategy, as explained by the developers, avoids the out-of-distribution issues frequently encountered when using standard BPE tokenizers, especially within the narrower confines of TTS training data. The approach’s ability to retain language understanding even with this unconventional representation is a significant observation, suggesting a potential pathway for further optimization in TTS models. Furthermore, the meticulous design of their chunking and position scheme—incorporating split markers and reserved character positions—demonstrates a sophisticated understanding of the challenges associated with maintaining context and preventing audio leakage across chunk boundaries. This contrasts with the simpler approaches often seen and echoes the ongoing need for robust data management and security, as seen in Anthropic’s recent cost reduction efforts for Fable cache reads, Anthropic's Claude Fable 5.1 and Mythos 5.1 arrive with a 75% cost reduction for Fable cache reads. The careful handling of position IDs, ensuring they remain monotonic and approximately linear, speaks to the level of engineering detail invested in this project.
TontaubeV1's open-weight release is particularly valuable to the research community, providing a platform for further experimentation and refinement of these novel techniques. While the current VRAM requirements (24-32 GB) do present a barrier to entry for some users, the developers’ stated plans for quantized versions and fine-tuning support promise to broaden accessibility. The model's performance in the audiobook benchmark, even with the caveat of relying on an LLM-as-a-judge, underscores the potential for AI-driven TTS to rival and potentially surpass human narration in certain contexts. The team’s commitment to participating in TTS Arena V2 and the Artificial Analysis Text to Speech Arena further validates their dedication to rigorous evaluation and continuous improvement, acknowledging the importance of human listening tests as the ultimate benchmark. The fact that AI agents, even those passing authentication, can still drift and expose data AI agents that pass authentication can still drift, expose data, or get memory-poisoned reinforces the need for continuous monitoring and refinement, a factor that will undoubtedly influence the development and deployment of models like TontaubeV1.
Looking ahead, the success of TontaubeV1’s character-level tokenization and chunking strategy raises a fundamental question: Will this approach become a new standard for long-form TTS generation, or will other techniques emerge to address the challenges of maintaining coherence and context? The current focus on English and German also suggests a potential avenue for future development – expanding language support while retaining the model’s unique strengths. As the demand for high-quality, AI-generated audio continues to grow across various applications, from audiobook production to virtual assistants, models like TontaubeV1 are paving the way for a future where synthetic voices are not just intelligible, but genuinely expressive and engaging.
| | Hey everyone, My brother and I just released TontaubeV1, a 2.9B-parameter open-weight TTS model focused on expressive speech, long-form generation/narration, and low-latency local inference. It is primarily aimed at English and German and supports zero-shot voice cloning from up to one minute of reference audio. It builds on DualCodec, a multi-codebook discrete audio codec. It was trained on 7 languages and ~200k hours of audio (mostly tested in English and German). I wanted to make a post to highlight two choices that worked well for us and seem less common in current TTS models: 1. Character-level tokenization We start from a Qwen3-1.7B checkpoint for our semantic codebook model. Many modern, and especially LLM-based, TTS models use the tokenizer from the backbone model, add special/audio tokens, and train the model on predicting the next token. We experimented early on with character-level tokenization and found that it generally worked better than using the original BPE tokenizer from Qwen. We still use the tokens emitted by the Qwen tokenizer, but force it to tokenize spoken text as a sequence of individual characters. When experimenting with forcing Qwen to predict text in this mode, we found that it was still able to answer questions correctly, which suggested to us that language understanding was retained even with this unusual representation. We did this because a) context length usually is not as much of an issue for TTS as it is for regular LLMs, since we do not require huge reasoning budgets and use chunking, and b) it makes the character-to-sound mapping much simpler internally. Speech is a lot about syllables and short character sequences. When using the regular BPE tokenizer, we found that the model went out of distribution more often and was more likely to encounter a sequence of tokens that was rare or absent from the TTS training data. Complex sequences of special characters can be particularly confusing because they may tokenize into combinations that the TTS model has barely seen. This is amplified by the fact that TTS training covers far fewer text-token combinations than the full pretraining of an LLM. 2. Chunking and position scheme The important part here is not just that we split long text into chunks. The chunk boundaries are part of the token layout and position scheme used during training. The model processes several rows in one flat sequence: text, semantic audio, and the completed lower acoustic codebooks. If we simply used normal sequential position IDs, tokens representing the same moment in the audio would end up far apart because the rows are serialized one after another. Instead, the physical sequence order determines which tokens the model can see, while we assign separate logical position IDs. Codec tokens for the same audio frame share a position across codebooks, and text and audio are placed on the same approximate timeline. Text advances by one position per character, while audio advances at 12.5 frames per second. These rates are fairly close for normal speech, but not identical, so the two streams usually need slight realignment after every chunk. We use paired text and audio split markers that share the same position. We also reserve an additional 25 character positions at each boundary. This prevents the generated audio positions from leaking into the next chunk’s text positions, while keeping the position IDs monotonic and approximately linear across the passage. For every chunk, the semantic model sees the previous text and audio chunk, the current text, and a short lookahead into the next text. Once a chunk is finished, the oldest text-audio pair is discarded and the window moves forward. This keeps the model context bounded even for very long passages, while still retaining nearby text and audio context. The higher acoustic codebook models work on one chunk at a time and do not carry acoustic state between chunks. DualCodec’s decoder is forward-looking, which makes directly decoding and joining separate chunks problematic. For streaming, we therefore decode overlapping DualCodec windows, re-encode them into the VibeVoice acoustic space, keep the stable middle sections, and decode them with a shared causal VibeVoice decoder state. This reduces audible seams between chunks and lets us emit audio before the full passage has been generated. Here is figure 1 from our technical report: The current release requires a GPU with at least 24 GB of VRAM for the low-VRAM and balanced profiles, or 32 GB for the high-throughput profile. A substantial part of the current VRAM requirement comes from vLLM’s KV-cache reservation and our multi-engine serving setup, which are designed for high concurrency and low latency. We plan to release quantized versions aimed at much smaller memory capacities and on-device use, as well as fine-tuning support. We also ran a 400-passage LLM-as-a-judge audiobook benchmark. On prosody, TontaubeV1 scored 50.1% against ElevenLabs Flash v2.5 and was preferred over Fish Audio S2 Pro, Gradium, and Cartesia Sonic 3. The methodology, caveats, and confidence intervals are described in the report. Human listening tests remain the gold standard, so take these results with a grain of salt. We were not able to conduct a large-scale human study before release, but we plan to submit TontaubeV1 to TTS Arena V2 and the Artificial Analysis Text to Speech Arena. Links: - HF model page: https://huggingface.co/TontaubeAI/TontaubeV1 - HF demo: https://huggingface.co/spaces/TontaubeAI/tontaube-v1-tts-demo - Inference code: https://github.com/craitech/tontaube - Technical report: https://tontaube.ai/papers/tontaube-v1-technical-report.pdf Let me know if you have any questions! [link] [comments] |
Read on the original site
Open the publisher's page for the full experience