LiveKit Agents: Realtime Voice AI Agents as Programmable Participants

Realtime voice agents as server-side programmable participants

LiveKit Agents is LiveKit's open-source framework for realtime voice AI agents (14,999 stars and 4,028 forks as of 2026-10-30, Apache-2.0, version 1.8.10). Its own positioning is deliberately narrow: build programmable participants that run on servers, and use them to create conversational, multi-modal voice agents that can see, hear, and understand. In engineering terms, the agent is not a chatbot behind an HTTP endpoint; it joins a WebRTC room like a human participant, subscribes to the user's microphone and camera tracks, publishes its own audio track back, and the whole path runs on LiveKit's realtime media server.

That choice dictates the shape of the framework. If the agent is a participant in the room, then "who is speaking, are they done, should we barge in, when do we reply" must be solved by the framework rather than by application code. LiveKit Agents answers with four abstractions plus a built-in conversation-control stack: Agent, AgentSession, entrypoint and AgentServer, backed by VAD, semantic turn detection, adaptive interruption detection, endpointing, background audio and filler scheduling.

Four abstractions: Agent, AgentSession, entrypoint, AgentServer

The README's "Core concepts" lists exactly four nouns, each with a clean responsibility boundary. An Agent is an LLM-based application with defined instructions, carrying prompts and tools. An AgentSession is the container that manages interactions with end users: wiring VAD, STT, LLM and TTS, and owning turns and interruptions. An entrypoint is the starting point of an interactive session, analogous to a request handler in a web server. An AgentServer is the main process that coordinates job scheduling and launches agents for user sessions. A minimal voice agent is just filling in one slot per layer:

@function_tool
async def lookup_weather(context: RunContext, location: str):
    """Used to look up weather information."""
    return {"weather": "sunny", "temperature": 70}

server = AgentServer()

@server.rtc_session()
async def entrypoint(ctx: JobContext):
    session = AgentSession(
        vad=inference.VAD(),
        stt=inference.STT("deepgram/nova-3", language="multi"),
        llm=inference.LLM("google/gemma-4-31b-it"),
        tts=inference.TTS("cartesia/sonic-3", voice="..."),
    )
    agent = Agent(
        instructions="You are a friendly voice assistant built by LiveKit.",
        tools=[lookup_weather],
    )
    await session.start(agent=agent, room=ctx.room)
    await session.generate_reply(instructions="greet the user and ask about their day")

Tools are plain async functions plus a @function_tool decorator: the docstring is the tool description and the type annotations become the parameter schema, with RunContext handing session context into the tool. After the session starts, generate_reply can drive the first turn with a one-off instruction, for example greet the user and ask about their day.

The model layer is a socket: mix STT, LLM, TTS and Realtime freely

The framework splits the voice path into four slots - STT, LLM, TTS and Realtime API - and the first feature in the README is flexible integration: mix and match the right providers per use case. The repository is a monorepo: beside the core livekit-agents package there are 83 plugin directories covering Deepgram, OpenAI, Cartesia, ElevenLabs, Google, Azure, Groq and Silero among STT/LLM/TTS/Realtime providers, plus video-avatar providers such as Tavus and Bithuman, telephony and turn detection. Install only what you need: pip install "livekit-agents[openai,deepgram,cartesia]".

Besides instantiating plugins directly (deepgram.STT(model="nova-3"), openai.LLM(model="gpt-4.1-mini"), cartesia.TTS(model="sonic-3", voice=...)), the framework offers LiveKit Inference as a unified entry point: write the model as a string such as "deepgram/nova-3", "google/gemma-4-31b-it" or "cartesia/sonic-3:voice-id" and reach it through LiveKit Cloud's inference gateway without holding every vendor key. The example's google/gemma-4-31b-it is a low-latency Gemma hosted on LiveKit.

The hard parts of conversation are built in: turn detection, interruption, answering-machine detection

Nearly all of the experience gap in voice interaction comes from knowing when a speaker is done. LiveKit Agents answers with semantic turn detection: a transformer model decides whether the user's turn has ended, which cuts both false interruptions and sluggish replies. In code that is inference.eot.TurnDetector, with model tiers turn-detector-v1 and the smaller turn-detector-v1-mini, served through the LiveKit Inference gateway. On the interruption side, AdaptiveInterruptionDetector stops the playing reply when the user barges in; the session layer adds endpointing, background audio and a filler scheduler that covers wait time with natural "uh, let me check" feedback.

Telephony gets its own component: AMD (Answering Machine Detection) for outbound calls. It starts before the SIP participant is created, listens to the greeting and uses an LLM to classify it into five categories, so the agent does not read a whole paragraph to a voicemail box.

AMD verdict	Meaning
human	A real person answered
machine-ivr	An IVR / DTMF menu prompt
machine-vm	A voicemail greeting where a message can be left
machine-unavailable	Mailbox full or not set up; no message possible
uncertain	Transcript ambiguous, cannot classify

Multi-agent handoff: let a tool return the next Agent

Real conversations rarely stay in one role. The framework's handoff pattern is direct: a @function_tool returns "the next Agent instance plus a handoff line" and the session switches character. In the README's storytelling example, IntroAgent gathers name and location, writes them into context.userdata, then returns StoryAgent with "Let's start the story!"; StoryAgent can override the model per agent, swapping the standard LLM for a Realtime API model such as openai.realtime.RealtimeModel(voice="echo"). The session container is generic over userdata (AgentSession[StoryData]), so state shared across roles stays typed.

Testing: assert on the session as an event stream, then let an LLM judge

LLM behaviour is non-deterministic and voice agents are harder still. The framework ships native test integration: inside pytest you drive a session with await sess.run(user_input=...) and assert on the resulting event stream - skip a message, expect the next event to be a function call named start_order, expect its output, then ask an LLM judge whether the assistant message matches an intent (.judge(llm, intent="...")). Judge calls are pinned to the low inference class (inference_class="low") so they never compete with live traffic for gateway capacity. The repo's test markers mirror this emphasis: unit, audio_eot, plugin, stt, tts, realtime, evals and docs, with per-plugin integration tests running in CI under provider credentials.

async with AgentSession(llm=llm) as sess:
    await sess.start(MyAgent())
    result = await sess.run(user_input="Hello, I need to place an order.")
    result.expect.next_event().is_function_call(name="start_order")
    await (result.expect.next_event()
           .is_message(role="assistant")
           .judge(llm, intent="assistant should be asking the user what they would like"))

Three run modes and production dispatch

Command	Behaviour	External dependencies
python myagent.py console	Terminal mode with local audio input and output for quick validation	No server or credentials needed
python myagent.py dev	Starts the agent server with hot reloading; one process hosts multiple concurrent agents efficiently	LIVEKIT_URL / API_KEY / API_SECRET
python myagent.py start	Production mode with production-ready optimizations	Same as above

On the production side, built-in scheduling and the dispatch APIs connect end users to agent jobs, and LiveKit's telephony stack (SIP) lets agents make and receive real phone calls; RPCs and Data APIs exchange structured data with clients. On the client side sits LiveKit's open-source SDK ecosystem: Browser, Swift, Android, Flutter, React Native, Rust, Node.js, Python, Unity, Unity WebGL and ESP32, with AgentsJS for TypeScript; the Agents Playground is the fastest way to feel the result.

Building agents with agents: the docs MCP and the official skill pack

The README devotes a section to AI coding assistants: install the LiveKit Docs MCP server (up-to-date documentation and cross-repo code search for your coding agent), then the official Agent Skill (npx skills add livekit/agent-skills --skill livekit-agents), which teaches workflow design, handoffs, tasks and testing patterns. The division of labour is stated plainly: the skill teaches your agent how to approach building with LiveKit, while the MCP server provides the current API details. MCP is also a first-class citizen inside the framework - tools from MCP servers integrate in one line of code.

Ecosystem position and licensing

LiveKit Agents sits at the junction of realtime transport and voice agents: below it is the LiveKit server, one of the most widely used WebRTC media servers; above it are the client SDKs and the telephony stack; this middle layer turns conversation control and model slots into framework capabilities. The examples directory sketches outbound calling, frontdesk, hotel reception, healthcare, drive-thru, warm transfer, multi-user transcription, video avatars and Gemini Live vision. Licensing: the framework is Apache-2.0 while the turn detection models carry the LiveKit Model License; development uses uv for dependencies, ruff for formatting and linting, and pdoc for docs. For teams that need a self-hosted realtime voice front door, it is one of the few open-source options that hands you transport, conversation control, models, testing and telephony in one place.

flowchart LR
    A["User mic / camera tracks
WebRTC room"] --> B["VAD + STT
deepgram/nova-3"]
    B --> C{"TurnDetector
turn-detector-v1-mini
is the turn over?"}
    C -- not yet --> B
    C -- done --> D["Agent / LLM
function_tool + MCP"]
    D --> E["TTS
cartesia/sonic-3"]
    E --> F["Publish audio track
back as a participant"]
    B -. parallel .-> G["AdaptiveInterruptionDetector
barge-in stops playback"]
    H["SIP outbound call"] -. pre-answer .-> I["AMD answering-machine detection
human / ivr / vm / unavailable"]
    I -- human --> A