NVIDIA Releases Open Full-Duplex Speech Model with ~450ms Turn-Taking and Live Tool Calling
Key Highlights
NVIDIA NemotronLabs recently open-sourced VoiceChat-11B, an end-to-end full-duplex speech-to-speech model. Simply put, it packs the acts of listening, thinking, and speaking into a single neural network, instead of chaining automatic speech recognition, a large language model, and text-to-speech into one pipeline as traditional voice systems do. The most immediate and practical benefit of this unified design is a sharp reduction in latency: measured on the Full-Duplex-Bench 1.0 benchmark, the smooth turn-taking latency is only 448 milliseconds, which sits close to the natural rhythm of a human conversation. This contrasts with the cascaded approach that dominates today's assistants, where separate models must pass information between stages and each handoff adds measurable delay. For developers building real-time voice agents, this latency budget is the difference between a system that feels responsive and one that feels sluggish.
Capabilities and What Happened
The model has two major selling points that set it apart. The first is genuine full-duplex capability: it listens while it speaks, and when a user interrupts in the middle of its utterance it immediately yields the floor, reaching a user-interruption takeover rate of 1.00 within 480 milliseconds, which means it almost never talks over the user. The ability to barge in matters for natural conversation, because most current voice bots force the user to wait until the agent finishes speaking. In practice, the on-hold message means a customer service agent can say something like let me check that for you the instant it decides to call a tool, keeping the user engaged instead of staring at dead air. The second point is that VoiceChat-11B is the first open full-duplex model to support live tool calling during a conversation: when it needs to check the weather or look up an order status, it emits a TOOLCALL script through a separate output channel, and then uses an operator-defined on-hold line to fill the waiting gap so the conversation never goes silent.
Technical Details
VoiceChat-11B uses a hybrid Mamba and Transformer architecture assembled from existing NVIDIA components. A Fast Conformer speech encoder continuously encodes the incoming 16kHz audio stream. The Nemotron Nano v2 model serves as the 9B-scale large language model backbone that consumes audio tokens and predicts text tokens. A text-to-speech decoder then outputs 22.05kHz agent speech. The choice of a Mamba-Transformer hybrid aims to balance the long-context efficiency of state-space models with the reasoning strength of attention layers. NVIDIA notes the model outputs not just agent audio but also a running user transcription, which is useful for logging and downstream processing. Training used roughly 550,000 hours of real and synthetic audio, building on earlier NVIDIA speech work such as SALM-Duplex and Audio Flamingo 3. The weights are released under the permissive OpenMDW-1.1 license, but NVIDIA currently labels the checkpoint as research only.
Comparison with Competitors
Compared with the traditional cascaded ASR, LLM, and TTS stack, VoiceChat-11B removes the need for multi-model orchestration and API handoffs, which lowers end-to-end latency. On the VoiceBench leaderboard it ranks second among open full-duplex models, and it also places second among open models on Full-Duplex-Bench 1.0. On the BFCL-v3 spoken tool-calling benchmark its average score is 56.1 percent. Among open models it is competitive but not state of the art, and NVIDIA itself acknowledges the checkpoint is best treated as a research artifact rather than a finished product. Against closed commercial voice systems, the open release lets researchers audit and fine-tune the model.
Industry Impact and Use Cases
The deployment floor is a single GPU with 80GB of VRAM, such as an A100, H100, or B200, and there is no hosted API available today. Suitable scenarios include barge-in-capable voice agents, in-car assistants, drive-thru ordering and interactive voice response systems, and game NPC dialogue. Because there is no hosted API, only teams with their own GPU capacity can evaluate it today, which limits immediate adoption but lowers vendor lock-in. The permissive license allows self-hosting and modification without per-call fees, which is attractive to enterprises that care about data compliance. The weaknesses are also clear: an audio context ceiling of about two minutes, possible degradation into gibberish after several turns, and runaway self-talk after a turn ends, so the model still remains some distance from production use.
Who Should Use It and Caveats
In plain terms, this model is currently better suited to teams with their own GPU engineering capacity who want to experiment and build on top of it, rather than ordinary users looking for a ready-made product. The people who should pay closest attention are engineers building voice customer service, in-car assistants, and smart-hardware products, as well as researchers who need low-latency real-time dialogue. A few caveats matter: it demands a single 80GB GPU, which is a real barrier for individual developers; the audio context lasts only about two minutes, so long conversations can break down; and NVIDIA explicitly labels it 'research only,' so you must check the license boundary before any commercial deployment. If all you want is a chatty Chinese voice assistant, this is not yet that answer, and you would be better served by a hosted API elsewhere.