Speech Engine

Overview

ElevenLabs Speech Engine adds voice capabilities to any chat agent. ElevenLabs handles speech-to-text and text-to-speech while your server provides the LLM logic. The SDK manages connection lifecycle, turn-taking, and interruption detection so you can focus on your agent’s behavior.

How it works

Speech Engine connects your server to ElevenLabs over WebSocket. Each connection represents one conversation.

1. A user speaks in the browser. ElevenLabs captures the audio and transcribes it.
2. The transcript is sent to your server along with the full conversation history.
3. Your server passes the transcript to your LLM and streams the response back.
4. ElevenLabs converts the text to speech and plays it in the browser.

When to use Speech Engine

Speech Engine is designed for developers who want to bring their own LLM and control the conversation logic on their own server. Use it when you need to:

• Add voice to an existing text-based chat agent
• Use a specific LLM, fine-tuned model, or custom inference pipeline
• Keep full control over conversation routing, context management, and tool calling
• Integrate voice into an existing server application (Express, FastAPI, etc.)

If you want a fully hosted solution where ElevenLabs provides the LLM, knowledge base, and tools, use ElevenAgents instead.

Key features

• Any LLM - use OpenAI, Anthropic, Google Gemini, or any model that produces text. The SDK auto-extracts text from OpenAI, Anthropic, and Gemini stream formats.
• Interruption handling - when the user speaks mid-response, the SDK cancels the in-flight LLM request automatically via an AbortSignal (TypeScript) or task cancellation (Python).
• Streaming - responses are streamed to the browser as they are generated. Pass a string, an async iterable, or a native LLM stream object.
• Turn-taking - the SDK manages conversation turns, so your server only needs to respond to transcripts.

Conversation duration

Speech Engine supports conversations of up to 40 minutes (2,400 seconds). By default, a conversation ends after 600 seconds (10 minutes). To allow longer conversations, raise conversation.max_duration_seconds in the API or Max conversation duration in the dashboard’s Advanced tab. See Configuring the maximum conversation duration. If you need conversations longer than 40 minutes, contact our team.

IP allowlisting

If your server is behind a firewall or uses IP-based access controls, you can allowlist the static egress IPs from which ElevenLabs WebSocket connections originate. See IP allowlisting for the complete list of IP addresses.

FAQ