VoxCPM2: Tokenizer-Free Multilingual TTS with Voice Design and Cloning

VoxCPM2 is OpenBMB's 2-billion-parameter speech generation model, released after VoxCPM and VoxCPM1.5. It does not first compress a waveform into a fixed vocabulary of discrete audio tokens and predict token IDs. Instead, it performs conditional modeling and diffusion generation in the continuous latent space of AudioVAE V2. That choice shapes the entire engineering stack: reference audio is encoded into continuous frames, a local encoder groups neighboring frames into patches, a text-semantic language model and a residual acoustic language model jointly condition the next latent block, and a local diffusion transformer turns that condition into decodable audio. The model produces native 48 kHz audio and supports multilingual synthesis, natural-language voice design, reference-audio cloning, continuation cloning with an exact transcript, and full or LoRA fine-tuning.

What “tokenizer-free” does and does not mean

VoxCPM2 still uses a text tokenizer, and it also applies finite scalar quantization, or FSQ, to some language-model hidden states. “Tokenizer-free” therefore cannot mean that every discrete operation has disappeared. What the repository avoids is the conventional discrete audio tokenizer: audio is not mapped to a finite set of acoustic token identities and predicted one class at a time. The AudioVAE V2 encoder returns continuous mean latents, and the local diffusion module predicts continuous latent patches. The main acoustic path has no codebook lookup and does not reduce each time position to a single integer class.

FSQ acts as a structured interface between semantic and acoustic representations, not as a replacement codec. The configuration sets scalar_quantization_latent_dim=512 and scalar_quantization_scale=9. The scalar quantization layer projects a language-model hidden state into 512 dimensions, applies a hyperbolic tangent to keep values within a quantizable range, and rounds at nine times the original scale. Training uses a straight-through estimator to preserve gradients; inference returns the rounded continuous vector. This regularizes hidden states at audio positions without building a large speech codebook that must be enumerated and maintained.

Layer	Actual form in the repository	Effect on generation quality
Text representation	Text tokenizer and MiniCPM-4 embeddings	Handles language, numbers, and punctuation; this is not “text-tokenizer-free”
Acoustic representation	Continuous latent frames from AudioVAE V2	Preserves timbre, prosody, and fine variation without a discrete codebook
Semantic interface	512-dimensional FSQ state with scale 9	Regularizes audio-position states and stabilizes transfer to the acoustic path
Generation core	Local diffusion transformer predicting continuous patches	Trades iterative sampling compute for continuous-detail modeling

The design eases the hard tradeoff between codebook capacity and reconstruction detail, but moves cost into sampling. Each language-model patch requires a flow-matching solver, with defaults of ten Euler steps and a classifier-free guidance value of 2.0. A low-latency deployment therefore cannot estimate cost from parameter count alone. It must measure diffusion steps, batch size, cache behavior, and decoder work together.

How LocEnc, TSLM, RALM, and LocDiT connect

The generation backbone has four stages. LocEnc, the local encoder, turns each four-frame AudioVAE block into one language-model time step. It does not generate a waveform directly from text; it turns the current acoustic context into a vector that can be mixed with text embeddings. TSLM, the text-semantic language model, is built on MiniCPM-4. It sees text positions and acoustic positions in one causal sequence and produces the semantic condition for the next step. RALM is an eight-layer residual acoustic language model. Its input combines the current TSLM state with the current acoustic embedding, and it focuses on local acoustic dependencies. LocDiT is the local diffusion transformer. It projects both language-model streams into a diffusion condition and generates the next block of four 64-dimensional latent frames, conditioned on the previous acoustic patch.

The order inside the autoregressive loop matters. The model first encodes any reference audio, prompt audio, and text, and fills the key-value caches for MiniCPM-4 and RALM. On each step, the TSLM hidden state and the previous RALM hidden state are projected separately and concatenated into the diffusion condition. After LocDiT creates a new patch, that patch passes through the local encoder and input projection, updates both language-model caches, and becomes the acoustic prefix for the next diffusion call. A stop head reads the TSLM hidden state to decide when to end, while minimum and maximum patch limits prevent premature stops and unbounded generation.

Stage	Input	Output	Responsibility
LocEnc	An acoustic block shaped [patch=4, latent=64]	Acoustic embedding at language-model width	Local compression and cross-modal alignment
TSLM	Text, reference-acoustic, and generated-acoustic embeddings	Semantic state for the next time step	Global text-to-speech semantic modeling
RALM	Fusion of the TSLM state and current acoustic embedding	Residual acoustic state	Phoneme transitions, prosody, and local continuity
LocDiT	Both language-model states and the previous acoustic patch	Next continuous latent block	Ten-step flow-matching sampling and detail generation
Stop head	Current TSLM hidden state	Continue or stop classification	Termination decision

This division prevents one large language model from carrying both sentence-level semantics and high-frequency acoustic recovery at incompatible time scales. TSLM decides what to say and how the next region should develop, RALM fills in local acoustic context, and LocDiT performs high-resolution iteration over only the next short block. The price is architectural complexity: a production runtime must manage two language-model key-value caches, the acoustic prefix, and the state of the diffusion sampler.

AudioVAE V2's asymmetric 16 kHz to 48 kHz path

AudioVAE V2 receives audio at 16 kHz. Its encoder rates are [2, 5, 8, 8], for an overall downsampling factor of 640. Dividing 16,000 by 640 gives 25 latent frames per second. The language model does not consume each frame independently. Four frames form one patch, so the effective language-model rate is 6.25 patches per second, with each patch spanning 160 milliseconds. That rate is far below sample-by-sample generation while retaining continuous vectors rather than a finite codebook approximation.

The decoder uses a different sequence: [8, 6, 5, 2, 2, 2]. Multiplying those factors gives 1,920. At 25 latent frames per second, 25 times 1,920 is exactly 48,000 samples per second. “16 kHz input, 48 kHz output” therefore does not mean copying samples or attaching a separate upsampler. The AudioVAE V2 decoder is trained to reconstruct a wider-band 48 kHz waveform from the lower-rate reference latent space. This asymmetry reduces upload and encoding cost, but the output sample rate is not itself a guarantee of high-frequency fidelity; that still depends on training data and the learned decoder.

Parameter	AudioVAE V2 configuration	Result or meaning
Encoder input	16,000 Hz	Unified sample rate for reference and training audio
Encoder rates	[2, 5, 8, 8]	Hop of 640; latent rate of 25 frames per second
Latent representation	64 continuous dimensions	Each acoustic frame is not a discrete ID
Language-model patch	4 latent frames	6.25 patches per second; 160 ms per patch
Decoder rates	[8, 6, 5, 2, 2, 2]	Overall upsampling of 1,920, yielding 48,000 Hz

Streaming decode does not repeatedly decode the whole latent prefix and discard an overlap. The repository supplies a stateful decoding context that keeps causal-convolution and transposed-convolution history. Each new patch is processed once, with boundary state carried into the next call. This preserves continuity without redoing the entire prefix. The language model remains autoregressive over patches, while the decoder can emit audio continuously.

Voice design, reference cloning, and continuation cloning are different paths

VoxCPM2 exposes three capabilities that look similar at the user level but have different inputs and control boundaries. If there is no reference audio and a natural-language description is placed in parentheses at the start of the target text, the request is voice design. It creates a new voice and does not bind the output to a real speaker. If only reference audio is supplied, the request uses isolated-reference mode. The reference is enclosed by dedicated start and end markers, and the model extracts timbre before generating the target text. If prompt audio and its exact transcript are supplied, the request enters continuation mode. The prompt audio is treated as an already spoken prefix, making timbre, rhythm, emotion, and recording conditions easier to carry forward. Supplying both reference and prompt audio places an isolated reference first, then prompt text, target text, and the prompt-audio suffix.

The command-line interface enforces the mode boundary. The design command rejects prompt or reference audio. The clone command requires reference audio, or prompt audio paired with prompt text. This restriction is useful: casually mixing descriptions, references, and transcripts into one unrestricted request obscures the fact that each field changes the sequence layout seen during training.

Mode	Inputs	Internal conditioning	Good fit and limits
Voice design	A parenthesized voice description plus target text	No reference audio; description becomes a text prefix	Good for new character voices; output can vary and may need retries
Isolated-reference cloning	Target text plus one reference clip	An independent 103 + reference latents + 104 prefix	For short references when the request should clone timbre without continuing the recording
Continuation cloning	Target text, prompt audio, exact transcript	Prompt and target text first; prompt audio supplies suffix context	Best for preserving rhythm, emotion, and recording texture; transcript must match
Combined cloning	Reference audio, prompt audio, transcript, target text	Isolated reference prefix plus continuation suffix	Passing the same clip in both roles can improve similarity, at higher cost

“Controllable cloning” is not precise parameter control. Text such as “slightly faster, cheerful tone” at the beginning of the target can steer expression while the reference conditions timbre. The authors also state that voice-design and controllable-cloning results can vary between runs and recommend generating one to three candidates. A fixed seed helps, but it does not turn the model into a deterministic performance engine.

How the training packet isolates reference audio from target audio

A reference-mode training sample can be written as [103, reference latents, 104, text, 101, target latents, 102]. Tokens 103 and 104 delimit reference audio; 101 and 102 delimit target audio. The text and acoustic tracks align position by position: reference latents appear only between 103 and 104, target latents appear only between 101 and 102, and the other positions use zero acoustic blocks. Text and audio masks are mutually exclusive, telling the model whether to read text or an acoustic vector at each time step.

The loss mask is more important than the visual layout. Reference patches, text positions, audio boundary markers, and the final marker all receive zero loss. Only target-audio patches contribute to diffusion loss. Reference audio therefore conditions generation without teaching the model to copy the reference waveform; the target clip supplies the prediction target. A separate stop label marks the final target-audio position. The local acoustic input is also shifted so that the model does not see the answer for the patch it is about to predict.

Training field	Reference span	Text span	Target-audio span
Text track	Reference positions between 103 and 104	Real text tokens	Target positions between 101 and 102
Acoustic track	Reference AudioVAE latents	Zero placeholders	Target AudioVAE latents
Loss mask	All zero	All zero	One for each target patch
Stop label	Zero	Zero	One at the final target patch

Diffusion uses conditional flow matching. Training constructs an interpolation path between the true target latent and Gaussian noise, and the model regresses the vector field from noise toward the target. The repository uses an Euler solver and a log-normal time schedule by default, and drops the condition for roughly ten percent of training examples so that classifier-free guidance has matching conditional and unconditional branches at inference. The default inference path uses ten steps and a guided scaling method with a zero-initialized early segment. There is no additional discrete acoustic cross-entropy; error propagates through continuous-latent regression.

Streaming deployment, fine-tuning, and the real resource bill

VoxCPM2 supports both one-shot and streaming generation. The one-shot path collects all patches before decoding the full latent sequence. The streaming path sends each new patch to a stateful AudioVAE decoder and returns waveform chunks as they become available. Before generation, streaming can reuse a few trailing prompt patches as acoustic context to reduce discontinuity at the first generated block. A prompt cache can also reuse the encoded reference and prefilled language-model state across requests, avoiding repeated work for a fixed voice.

The README lists Python 3.10 or newer but below 3.13, PyTorch 2.5 or newer, and CUDA 12 or newer. Its setup guidance cites about 8 GB of VRAM for base PyTorch inference. The Docker fine-tuning guide recommends at least 16 GB, with more memory preferred. These are project guidance, not hard guarantees for every backend, batch size, context length, and concurrency level.

Scenario	Project-reported reference	What deployment testing must add
Standard PyTorch on an RTX 4090	Real-time factor around 0.30	End-to-end latency for target sentence lengths, diffusion steps, batches, and concurrency
Nano-vLLM or vLLM-Omni on an RTX 4090	Real-time factor around 0.13	Batched throughput, cache behavior, and limits of the OpenAI-compatible interface
Apple M4 Pro with Q8 GGUF	Real-time factor around 1.76	This is not real time; validate device-specific latency and quantization quality
Base inference	About 8 GB of VRAM	Peak memory with full context, concurrent requests, and decoder state
Docker fine-tuning	At least 16 GB of VRAM	Full-parameter updates, optimizer state, and longer samples can raise demand

Fine-tuning has full-parameter and LoRA routes. The VoxCPM2 LoRA configuration targets both language models and the local diffusion transformer, while projection layers are disabled by default. The example uses rank 32 and alpha 32. Full supervised fine-tuning uses a learning rate of 1e-5; LoRA uses 1e-4. The README claims that five to ten minutes of audio can adapt the model to a particular speaker, language, or domain. That should be read as an experimental data threshold, not proof of production quality, broad accent coverage, or resistance to overfitting. Validation should include held-out speakers, text domains, and recording conditions.

Architecture selection must also separate real-time factor from time to first audio. Even when the factor is below one for a long sentence, the first streamed block still waits for text prefill, at least one diffusion patch, and decoder startup. A prompt cache can sharply reduce repeated work for a fixed voice, but multiple active voices turn cache capacity and VRAM into a new bottleneck. Production telemetry should record first-chunk latency, generated audio seconds per wall-clock second, tail latency, and retry rate separately.

What the project-reported evaluation does and does not establish

The figures below come from the VoxCPM2 README and were organized by the project team. They are not an independent third-party reproduction. Seed-TTS is a common zero-shot speech-synthesis evaluation family. Word or character error rate measures intelligibility, while speaker similarity measures closeness to the reference timbre. VoxCPM2 reports English WER 1.84 with similarity 75.3, Chinese CER 0.97 with similarity 79.5, and hard-set CER 8.13 with similarity 75.3. These show competitive results on selected public subsets, but they do not automatically transfer to noise, far-field audio, telephone bandwidth, children's voices, or professional dubbing.

Project-reported evaluation	Metric	Result	Reading boundary
Seed-TTS English	WER, lower is better	1.84%	Reflects intelligibility under the test text and conditioning setup
Seed-TTS English	SIM, higher is better	75.3%	A similarity metric does not cover every dimension of human listening
Seed-TTS Chinese	CER, lower is better	0.97%	Segmentation, numbers, and polyphonic characters affect the statistic
Seed-TTS Chinese	SIM, higher is better	79.5%	Emotion, speaking rate, and recording variation still require independent review
Seed-TTS hard set	CER / SIM	8.13% / 75.3%	Hard and regular subsets must not be conflated
Internal 30-language ASR	Average error rate	1.68%	500 samples per language, transcribed with Gemini 3.1 Flash Lite

The 30-language average needs to be unpacked. The project team constructed the internal samples and used one chosen external ASR interface to judge transcript agreement. The result measures intelligibility, not naturalness, emotional accuracy, dialect authenticity, or long-form stability. The README also claims support for nine Chinese dialects, but it does not publish a dialect-by-dialect table with the same granularity as the 30-language test. This analysis therefore does not treat dialect breadth as independently verified. The project's own multilingual tables already show substantial language-to-language differences, so adopters should test the languages their product actually serves.

The InstructTTSEval results for voice design indicate some instruction-following capability, but they are not a product-level stability test. A serious evaluation should measure whether descriptions are understood, outputs remain usable across repeated generations, different described voices are distinguishable, and identical descriptions produce coherent variation. The authors already acknowledge that voice design and controllable cloning vary between runs and recommend one to three attempts. Model benchmarks are useful for comparison; they do not justify a promise that every description will work reliably.

Voice cloning risk belongs in the system design, not in a post-launch note

VoxCPM2's capability set inherently includes high-fidelity speaker imitation. The README explicitly forbids impersonation, fraud, and disinformation, and recommends visibly marking generated content. An Apache-2.0 release does not remove personality rights, privacy law, data-protection duties, consumer-protection rules, or platform policy. Cloning a real person's voice requires a documented rights basis and verifiable consent. For user-uploaded audio, a platform also needs permission broad enough to cover processing, cloning, and derivative generation.

Technical safeguards should be designed alongside data governance. A service can restrict reference length and source, avoid retaining raw biometric audio longer than necessary, attach a verifiable provenance marker to generated audio, route high-risk text through human review, and give voice owners a takedown path. A policy statement prohibiting impersonation is not sufficient because misuse often happens outside the interface that wrote the policy. Public-facing products also need an incident response plan for claimed impersonation before the first complaint arrives.

Training-data governance remains an open question. The README states a training scale above two million hours and lists language coverage, but the repository page does not fully document the licensing of every source, speaker consent, regional distribution, deduplication, sensitive-content filtering, or biometric retention. Apache-2.0 is the code license and the model license declared by the project; it does not prove that every training source can be used without restriction, and it does not automatically satisfy local rules for voice biometrics. Commercial adopters should request a data provenance inventory, license matrix, and deletion mechanism from the provider.

Risk	Minimum control	Acceptance evidence
Impersonation and fraud	Identity checks, synthesis marking, review for high-risk text	Impersonation tests, block logs, human-review records
Cloning without consent	Voice-rights declaration, consent record, takedown process	Authorization evidence, removal SLA, third-party audit
Biometric leakage	Data minimization, encryption, access auditing	Data-flow map, retention limits, key and permission audit
Training-source rights	Source inventory, license review, regional restrictions	Dataset cards, license matrix, supplier commitments
Multilingual quality misjudgment	Blind test sets for each business language	Independent intelligibility, naturalness, similarity, and stability reports

Where VoxCPM2 fits

VoxCPM2 is a strong fit for workflows that need local deployment, fine-tuning, 48 kHz output, and multiple cloning conditions: licensed character dubbing, an owned corporate announcer, accessible reading, interactive devices, and multilingual content prototypes. It should not sit behind a frictionless “upload any person's voice” endpoint. News, financial notices, healthcare communication, and public warnings require stronger human approval, signing, and provenance checks because failures carry much higher costs than they do in ordinary entertainment content.

Adoption checklist and conclusion

Start with a pinned model version, a fixed seed, and ten diffusion steps. Compare no-reference synthesis, voice design, short-reference cloning, and continuation cloning before deciding whether LoRA is needed. Do not validate only on clean recordings from one speaker. Add the accents, speaking rates, numbers, proper nouns, background noise, and long sentences that the target product will encounter. Track word or character error rate, first-chunk latency, full-generation latency, human naturalness, speaker similarity, repetition and omission rate, and whether retries converge after a bad generation.

VoxCPM2's significance is not simply that its output sample rate is 48 kHz. It combines continuous acoustic latents, a semantic language model, a residual acoustic language model, and local diffusion into a speech system that can be trained, streamed, and fine-tuned. The implementation is transparent enough for engineering teams to evaluate deeply. The public quality evidence remains largely project-reported, while language coverage, dialect quality, training-data provenance, and misuse prevention are not fully answered by the repository page. The safer adoption stance is to treat VoxCPM2 as a capable, rights-gated speech generation component, not as a universally trustworthy machine for copying any voice.