PersonaPlex: Steering Full-Duplex Conversation with Role Text and a Voice Sample

A full-duplex conversation model with two independent control surfaces

PersonaPlex-7B is a real-time speech-to-speech model that can listen and produce speech on the same timeline. Its distinguishing idea is not simply “a language model with a voice.” Each session receives two forms of conditioning: a short audio sample that establishes how the agent sounds, and a text role prompt that establishes who the agent is, what information it knows, and how it should conduct the conversation. The code placed in this repository shows how those controls enter the model state, how audio is streamed through a browser, and how the same setup can be exercised offline.

The system builds on Moshi and the Helium language backbone. Instead of running speech recognition, a text dialogue model, and speech synthesis as three separate services, PersonaPlex encodes user audio with Mimi and advances one autoregressive model over a shared sequence of user and agent channels. The model predicts text and audio tokens for the agent while user audio continues to arrive. Mimi then turns the generated audio tokens back into PCM. This architecture is the reason interruption and overlap belong to the core interaction rather than being bolted onto a turn-taking wrapper.

The 80 ms frame is the basic unit of interaction

Mimi operates at 24 kHz with a frame rate of 12.5 Hz. One frame therefore contains 24000 / 12.5 = 1920 samples and lasts 80 ms. The server uses exactly that value when it accumulates decoded browser audio and when it advances the language model. This granularity matters more than a generic claim of “low latency”: it determines how frequently the model can observe the user, emit agent audio, and react to an interruption.

The model-side configuration is large but specific. The temporal Transformer has width 4096, 32 layers, 32 attention heads, and a context of 3000 frames. A separate depth Transformer has width 1024, 16 heads, 6 layers, and an 8-frame context for coordinating audio codebooks within a time step. The LM configuration declares 16 audio query slots and 16 delayed streams, while the live server path decodes tokens[:, 1:9], or eight agent audio streams, through a Mimi model configured with eight codebooks. Those are two layers of the implementation, not contradictory headline specifications.

Layer	Concrete implementation	Effect on the conversation
Mimi codec	24 kHz, 12.5 Hz, causal SEANet encoder and decoder	Converts continuous speech into a frame sequence that can be modelled jointly
Voice conditioning	Raw audio resampled and normalized to -24 LUFS, or precomputed .pt state	Sets the agent's acoustic identity before the first user turn
Temporal Transformer	4096 wide, 32 layers, 32 heads, 3000-frame context	Carries dialogue history and coordinates the user and agent streams
Depth Transformer	1024 wide, 16 heads, 6 layers, 8-frame context	Generates the codebook combination needed for each audio frame
Browser transport	WebSocket carrying Opus audio and incremental text tokens	Keeps microphone input and generated speech active at the same time

The hybrid prompt is a state transition, not just a concatenated string

At connection setup, the server runs four consecutive phases: the selected voice prompt, approximately 0.5 seconds of silence, the text role prompt wrapped in <system> tags, and another approximately 0.5 seconds of silence. In the voice phase, the text channel is held at a zero control token and the agent audio channel carries a sine-wave marker. In the text phase, the agent audio side remains silent while each system-prompt token is forced into the text channel. The conversation only begins after both conditioning phases are complete.

This ordering prevents an important conceptual error. The speaker sample is not treated as the user's opening utterance, and the role prompt is not ordinary conversation history. They initialize different channels of the same model state. The repository ships precomputed voice embeddings and their corresponding cache so that a common voice can be restored without re-encoding the reference waveform at every connection. When a raw audio file is used instead, the code resamples it to the codec rate and normalizes loudness before encoding.

voice sample → 0.5 s silence → <system>role prompt<system> → 0.5 s silence → live duplex stream

The synchronization code exposes an asynchronous version of this initialization. It checks whether the WebSocket is still alive between prompt chunks and even between model steps. A browser that disconnects during a long role prompt can therefore stop the session before all text has been injected. Once initialization finishes, Mimi streaming state is reset so the user's first input begins from a clean conversational boundary.

The browser connection runs three coordinated loops

The WebSocket endpoint is /api/chat. During setup it reads the chosen voice file, the role text, and optional random seeds. It prepares an Opus reader and writer, resets the Mimi and language-model state, applies the system prompt, and then sends a one-byte handshake to tell the browser that conversation may start.

After the handshake, three tasks run together. The receive loop accepts binary frames, with message kind 1 carrying Opus payloads into the decoder. The audio loop converts decoded PCM into fixed 80 ms chunks, encodes each chunk with Mimi, steps the language model, decodes the agent's audio tokens, and appends the resulting speech to an Opus writer. The send loop drains that writer and sends audio back as message kind 1. Text token pieces are sent separately as message kind 2. None of these loops waits for a full conversation turn, which is how simultaneous listening and speaking are represented in the transport layer.

Concurrency within one process is deliberately limited. ServerState wraps a complete chat session in an asyncio.Lock, so one loaded model instance serves one session at a time. Additional WebSocket clients can connect, but their sessions wait for the lock and cannot share generation state. A deployment that needs parallel conversations must run multiple processes or multiple model replicas and place scheduling in front of them. The lock is an honest capacity boundary, not a missing line of application code.

The connection terminates when any of the three loops finishes. The server cancels the other tasks and closes the socket. The generated Web UI uses temporary SSL certificates for local microphone access, and the Python server listens on port 8998 by default. Those defaults are convenient for a workstation demonstration. They are not a public-service design: authentication, rate limiting, TLS management, origin policy, and conversation isolation still have to be added around the model.

Voice samples, role prompts, and what the model was trained to imitate

The Web UI exposes 18 fixed voices: four natural female, four natural male, five varied female, and five varied male samples. NATF0 is the default. These names describe a packaged catalogue, not a promise that any uploaded recording can be cloned. The interface also limits the role text to 1000 characters and supplies presets for a teacher, medical-office intake, bank support, and a Mars astronaut scenario.

The training description explains why those presets make sense. Synthetic conversations cover one fixed assistant role and a range of customer-service roles, including companies, names, prices, policies, and customer requests. The open-ended portion comes from Fisher English conversations with prompts generated by an LLM. The resulting model is meant to follow a role that can be stated in natural language, not to replace a database-backed agent that must retrieve authoritative records on every turn.

Control	Good fit	Needs an external system
Fixed voice sample	Choosing a recognizable agent voice and keeping it stable during a session	Arbitrary voice cloning, consent tracking, or voice identity governance
Role text	Persona, tone, conversational objective, background facts, service boundaries	Live inventory, payment, authentication, refunds, and audited business actions
Incremental text tokens	Displaying a transcript that follows the generated speech	Guaranteed word-perfect captions or post-hoc compliance review
Opus duplex stream	Interactive conversation with barge-in and overlapping audio	Call recording retention, telephony integration, and omnichannel routing

The README also points to Helium's broad language training as a source of out-of-distribution behavior. That is an invitation to test novel prompts, not evidence that every invented scenario will be handled consistently. A reasonable evaluation separates persona adherence, conversational turn behavior, factual grounding, and speech quality. A single “sounds natural” judgement collapses all four and hides failures that matter in deployment.

Offline inference is useful, but it answers a narrower question

python -m moshi.offline loads the same codec, tokenizer, language model, and prompt phases as the server, then streams a user WAV through the model one frame at a time. It writes the agent output as a WAV of the same duration and stores generated text tokens as JSON. This is a practical fixture for comparing voices and prompts without a microphone or WebSocket in the loop.

The offline path deliberately mirrors the server's phase order, including both silence spacers. It also exposes controls for audio and text temperature, top-k sampling, greedy decoding, and random seeds. Because the same components are used, a prompt change that breaks the model in offline mode is likely to break the server too. This makes offline runs valuable for regression testing before exercising the browser stack.

Equal input and output duration is a property of the evaluation script, not a general law of the model. The script trims generated audio if it runs long and pads with zeros if it runs short. That makes paired audio files easy to compare, but it does not measure barge-in, network delay, audio-device buffering, or a user who changes their mind halfway through a reply. Those behaviours require the live connection path and should be tested separately.

Licensing and hardware constraints are part of the system design

The repository code is MIT-licensed. The PersonaPlex model weights are not covered by that same blanket claim: they are released under the NVIDIA Open Model License and require accepting the Hugging Face model terms before download. Calling the project “fully open source” without this distinction is inaccurate. Developers should review the weight license whenever they expose the model to third parties or distribute outputs at scale.

The --cpu-offload option is another common source of misunderstanding. It uses Accelerate to keep layers that do not fit in GPU memory on the CPU and move them back for computation. It does not turn the default CUDA server into CPU-only inference. Pure-CPU evaluation requires installing a CPU build of PyTorch and accepting the resulting performance tradeoff. No latency or real-time capacity should be inferred from an offload run.

There is also an input-boundary issue in the demo server. The requested voice filename is joined to a configured directory and loaded from that result. The bundled UI selects from a fixed list, but the raw endpoint does not itself establish a public-service security boundary. A production wrapper should validate against an exact allowlist, reject path components, isolate the model service from general storage, and add authentication and rate limits. These are deployment requirements, not features supplied by the research demo.

This review corresponds to commit 3428dfd95309a7f3c84fd93259ded0f810d1ff91 and the paper at arXiv:2602.06053. The repository provides a clear research implementation of real-time persona-conditioned speech, a browser client, and an offline harness. It does not claim to provide production multitenancy, arbitrary voice cloning, a complete business-agent toolchain, or a public-service security layer.

Where PersonaPlex is technically interesting

The strongest contribution is the explicit separation between conversational behaviour and acoustic identity. A text prompt can change the agent from teacher to service representative to astronaut, while a voice embedding keeps the sound stable. Both are injected into the same autoregressive model before live audio begins. That design makes the model a useful testbed for questions that text-only dialogue systems cannot answer: how quickly does it yield after an interruption, how much overlap feels natural, whether a role remains coherent when the user changes topic, and whether a voice stays recognizable under emotional shifts.

Its practical limits follow from the same design. PersonaPlex is one loaded model instance per conversation in the included server, it relies on fixed voice artefacts, and its role is carried primarily by prompt text and learned behavior. Systems that need deterministic transactions should keep the speech model at the interaction edge and move identity checks, retrieval, permissions, and actions into services that can be audited. PersonaPlex can make that edge feel conversational; it should not be mistaken for the whole enterprise stack.