CPU TTS benchmark with UTMOS MOS scoring: Kokoro, Supertonic, Inflect-Nano, and Kyutai's new Pocket TTS [P]
Our take
The recent CPU TTS benchmark, meticulously shared by u/gvij, offers a valuable glimpse into the evolving landscape of small text-to-speech models. It’s refreshing to see such a detailed, objective comparison, particularly given the increasing focus on efficient and accessible AI solutions. The inclusion of Kyutai’s Pocket TTS, an architecture distinct from its competitors, is especially noteworthy. This kind of granular analysis, alongside efforts like those detailed in "TRACE: open-source hierarchical memory for LLM agents, 82.5% on MemoryAgentBench’s EventQA using gpt-oss-20B [P]," exemplifies the trend toward optimizing AI for resource-constrained environments. Considering the growing demand for edge computing and on-device AI, benchmarks focusing on CPU performance are becoming increasingly crucial, and this contribution fills a significant gap. Similarly, the exploration of Edge AI ASL Recognition on Raspberry Pi 5 [P] highlights the wider movement towards deploying sophisticated AI capabilities on accessible hardware.
The findings presented are genuinely insightful, moving beyond simple speed comparisons to delve into architectural nuances and the limitations of current evaluation metrics. The observation that Pocket TTS exhibits flat RTF scaling – a linear cost proportional to output length – is a compelling advantage for interactive applications. This contrasts sharply with models like Kokoro, where latency increases with text length, a critical consideration for real-time responsiveness. Furthermore, the critique of UTMOS, a commonly used objective scoring metric, is particularly important. The benchmark rightly points out its potential failure to differentiate between "clean and mechanical" and "clean and natural" speech, especially within smaller models. This reinforces the need for a more holistic evaluation approach, incorporating human listening tests or metrics like NISQA, as suggested by the author. The documented issue with Inflect-Nano’s output cap also serves as a reminder of the importance of rigorous testing and validation, particularly when dealing with potentially undocumented limitations.
Beyond the specific results, this benchmark underscores a broader shift in the AI community towards a more pragmatic and nuanced understanding of model evaluation. The author's transparency regarding the use of an AI engineering agent (Neo) to assist in code development also sparks an interesting conversation about the evolving role of AI in AI development itself. It’s no longer sufficient to simply tout “revolutionary” or “cutting-edge” technology; instead, users and developers demand concrete performance data and a clear understanding of trade-offs. The comprehensive documentation, including raw CSVs and WAV samples, further demonstrates a commitment to reproducibility and community contribution – essential elements for fostering progress in the field. This aligns with the broader trend seen in areas like LingBot-Vision: masked boundary modeling for self-supervised pretraining, where researchers are meticulously documenting their methodologies and results to facilitate further exploration and improvement.
Ultimately, this CPU TTS benchmark provides a valuable resource for anyone evaluating small TTS models, highlighting the importance of architecture, evaluation methodology, and hardware optimization. As AI continues to move beyond cloud-based behemoths and toward more accessible and efficient solutions, benchmarks like this will play an increasingly vital role in guiding development and informing user choices. The question remains: how can we develop more robust and holistic evaluation metrics that accurately reflect the subjective qualities of AI-generated speech, particularly as models continue to shrink and become more specialized?
Sharing a CPU TTS benchmark with objective MOS scores in case it's useful for anyone evaluating small TTS models. Adding this because Kyutai's Pocket TTS is architecturally different from the others in the field and I hadn't seen a head-to-head with it yet.
Models:
- Kokoro 82M (PyTorch and ONNX Runtime, StyleTTS2-inspired)
- Supertonic 3 at 2 and 5 flow-matching steps (Vector Estimator backbone)
- Inflect-Nano-v1 (4.6M param FastSpeech-style, tiny end of the spectrum)
- Pocket TTS (~100M param streaming LM over Kyutai's Mimi neural audio codec)
Setup: Intel Xeon 8272CL, 4 cores, 15.6GB RAM. CUDA disabled at env level. ONNX sessions pinned to CPUExecutionProvider. Six configs, six text lengths (12 to 1712 chars), five timed reps per cell after a discarded warmup. 180 total runs. Every saved WAV scored with UTMOS (utmos22_strong) for objective MOS.
Aggregate results:
| Config | Mean RTF | UTMOS |
|---|---|---|
| Supertonic 3 (2-step) | 0.121 | 1.53 |
| Inflect-Nano-v1 | 0.145 | 3.48 |
| Supertonic 3 (5-step) | 0.240 | 4.32 |
| Kokoro 82M (ONNX) | 0.641 | 4.44 |
| Kokoro 82M (PyTorch) | 0.665 | 4.46 |
| Pocket TTS | 0.714 | 4.10 |
Findings I think are actually interesting:
1. Streaming LM architecture produces flat RTF scaling. Pocket TTS's RTF is 0.69 to 0.76 across the entire text length range. Because it emits audio tokens autoregressively at a steady rate, cost is linear in output length with no fixed overhead to amortize. Compare to Kokoro PyTorch, which climbs from 0.49 on tiny to 0.83 on long inputs, or Supertonic which goes the other way (0.36 on tiny down to 0.20 on medium) because of high per-call fixed overhead. If you're budgeting worst-case latency for an interactive system, flat is worth a lot.
2. UTMOS has a known failure mode on small vocoders. Inflect-Nano-v1 scored 3.48, which reads mid-pack. By ear it's buzzy and robotic. This is a documented issue: UTMOS rewards HiFi-GAN outputs for being clean even when they lack prosodic naturalness. Pocket TTS scored similarly (4.10) but sounds legitimately natural. The point isn't that UTMOS is broken, it's that a single quality number can't distinguish "clean and mechanical" from "clean and natural" on small models. Worth pairing with human listening or a naturalness-specific metric like NISQA.
3. Inflect-Nano has an undocumented ~15s output cap. The model config sets max_frames = 1400, which caps synthesis at ~14.93s regardless of input text length. Its RTF and throughput on long/paragraph/extended inputs are inflated because it's doing less work than the models it's compared against. Real comparison for that model is on tiny/short/medium only.
4. Kokoro ONNX vs PyTorch results reverse from the previous run. I ran an earlier version of this benchmark on AMD EPYC and PyTorch beat ONNX in aggregate. On this Xeon, ONNX is faster (0.641 vs 0.665). Same code, different silicon. AMD vs Intel kernel optimization differences at CPU inference are apparently real enough to flip the ranking. If anyone has replicated this on ARM I'd be curious.
Zero-shot voice cloning as a capability that doesn't fit the benchmark axes:
Pocket TTS can clone a voice from ~5 seconds of reference audio, zero-shot, on CPU. No other model in this field does this. I pinned it to a preset voice for the speed/quality comparison to be fair, so the cloning capability isn't reflected in the numbers. This is a real limitation of RTF-and-MOS-based comparisons: they can't capture capabilities that only one model has. Might want a separate speaker-similarity evaluation for a v2.
Limitations:
- Single hardware platform
- English only
- UTMOS is one MOS predictor; NISQA or a listening panel would strengthen the quality claims
- Voice cloning quality was not evaluated
- No batched inference tested
Disclosure: The benchmark harness was written by an AI engineering agent (Neo) from a prompt I specified. I chose the methodology, validated the outputs, and reviewed the audio. Mentioning it because it's relevant to how you'd want to weight the code.
All code, raw CSVs (180 rows), MOS CSV (36 rows), and WAV samples are in the repo mentioned in the comments below 👇
Feedback on the protocol welcome, especially on the MOS methodology and what a proper voice-cloning eval would look like.
[link] [comments]
Read on the original site
Open the publisher's page for the full experience