Everybody can now build the best voice agent into their own product. The #1 loop is open source, and the benchmark that says so is public.

Why we built it

We wanted the fluid JARVIS feel in the browser for TODOforAI — you talk, it answers within a second, you interrupt it mid-sentence and it just stops. We could not find a stack that did this properly. Closed APIs were close but not ours; the open frameworks talked over the user, or worse, heard their own voice through the speakers and cut themselves off.

It is 2026. This should be a solved problem. So we solved it and published the whole thing: the loop, the numbers, and the rig that produced the numbers.

What voiceloop is

A zero-dependency JavaScript library that runs the full loop in the browser: VAD → STT → LLM → TTS, with the hard parts already handled.

  • Real barge-in — triggers on transcribed novel words, not mic energy, so the agent’s own voice leaking into the mic never cuts it off.
  • Self-echo filtering — a word-match filter compares what the mic hears against what the agent is currently saying. 0 self-interruptions in 30 echo-coupled turns with echo cancellation fully off.
  • First audio under a second — TTS speaks sentence 1 while the LLM is still writing sentence 2, and the LLM call starts speculatively during your end-of-turn pause.
  • Local-first — Silero VAD and Piper TTS run as WASM in the tab. Free, no cloud round-trip.
  • Serialized turns — rapid-fire turns, tool results, holds and replays can never talk over each other. Locked in by 178 tests.

Everything is pluggable: any OpenAI-compatible LLM, four STT providers (Web Speech, ElevenLabs Scribe, Deepgram Flux, Speechmatics), swappable TTS (Piper local, ElevenLabs cloud, or your own).

import { VoiceAgent, unlockAudio } from '@todoforai/voiceloop';

const agent = new VoiceAgent({
  llmUrl: '/api/chat/completions',   // any OpenAI-compatible endpoint, behind your proxy
  model: 'claude-haiku-4-5',
  persona: 'You are a friendly cooking assistant.',
  onEvent: (e) => { if (e.type === 'assistant') render(e.text); },
});

button.onclick = async () => { unlockAudio(); await agent.start(); };

That is the entire integration.

The numbers

Latency claims in voice AI are usually self-reported and unreproducible. We did not want to add another one, so we built voice-agent-bench: a black-box rig. A scripted “person” (byte-identical pre-generated speech) talks into a virtual mic, the agent’s speaker output is recorded, and every score is derived from the audio alone. No integration needed — any agent that makes sound can be measured, including closed ones.

Every system gets the same scripted conversations and, where the system allows it, the same fixed mock LLM (300ms TTFT), so the comparison isolates the voice loop from the model. 5 conversations × 6 turns pooled, n=30, median and p95 — single runs jitter by ±300ms and are not worth printing.

Smalltalk — same mock LLM, 5×6 turns pooled

configurationvoice→voicep95barge-in stopstalls
OpenAI Realtime (speech-to-speech, own LLM) *866ms1644429ms20
voiceloop · deepgram + ElevenLabs flash862ms1067944ms16
voiceloop · deepgram + Piper (free, local TTS)974ms12871463ms19
Pipecat 1.8.1 · deepgram + EL flash1046ms3573542ms14
ElevenLabs ConvAI1454ms16321042ms8
voiceloop · EL Scribe + EL flash1562ms18551566ms12
voiceloop · Speechmatics + EL flash1706ms20691046ms17
voiceloop · webspeech + Piper (zero-key)2113ms26071257ms30

Across scenarios

systemcleanhesitationtalked through userechocut itself
OpenAI Realtime *87012900 (yields 130ms)79017/30
voiceloop · deepgram + EL flash86014000 (420ms)9300
voiceloop · deepgram + Piper97014000——
Pipecat105012902 (200ms)132020/30
ElevenLabs ConvAI145018100 (490ms)14100

* Realtime is speech-to-speech and can’t use the fixed mock LLM, so its row isn’t fully apples-to-apples.

How to read it

Clean audio: voiceloop with Deepgram Flux + ElevenLabs flash is the fastest configuration we measured, at 862ms median — and its p95 (1067ms) is the tightest in the table by a wide margin. Pipecat’s p95 of 3573ms on the same providers means one turn in twenty takes over three seconds. The free, fully local Piper path lands at 974ms with no cloud TTS at all.

Echo is the failure that separates the stacks. Feed each system its own voice back through the mic (−15dB, 30ms delay, no AEC — what a laptop with the speakers on actually does) and the other fast stacks hear themselves as the user and cut their own replies: Pipecat on 20 of 30 turns, OpenAI Realtime on 17. voiceloop cut itself zero times and ran echo-coupled turns at 930ms — parity with clean. Word-level echo filtering costs no latency once it classifies correctly.

Hesitation: a user who pauses mid-sentence should not be talked over. Every stack except Pipecat backs off; voiceloop enters 2 of 30 hesitation turns and yields within 420ms.

The zero-key default is honest about its cost. Browser Web Speech + Piper needs no account anywhere and runs the demo, but it is ~1.2s slower to close a turn than cloud STT (2113ms). Pick a pipeline STT provider for the numbers above.

Full per-scenario tables, methodology and reproduction steps: results/RESULTS.md.

Hundreds of configurations, so you don’t have to

The rows above are the survivors. Behind them are hundreds of runs across STT providers, TTS engines, VAD thresholds, end-of-turn debounces, barge-in minimum lengths, prefetch stability windows and echo-match thresholds. Every knob that mattered is exposed in src/tuning.js with the default set to what won on the bench. The edge cases you would otherwise discover one production bug at a time — the agent interrupting itself, tool calls firing on a sentence the user was still amending, a hung tool stalling the next turn — are already handled and regression-tested.

Try it, use it, beat it

  • Try: todoforai.github.io/voiceloop — 20 seconds, no keys, runs in your tab.
  • Use: npm i @todoforai/voiceloop — MIT, zero dependencies. Point it at any OpenAI-compatible endpoint and wire in your tools.
  • Beat it: the bench is black-box and public. If your stack does better, add it — ADDING_A_SUT.md is the contract. We will print the row.

This is the voice loop inside TODOforAI’s JARVIS; the integration overhead between the library and the product is nil, which is exactly the point. Everybody should have the best voice loop. Star it, share it, contribute — let’s keep the best one open source.