Designing a real-time voice AI architecture with WebSockets and LLMs

Introduction

Most LLM applications start with a simple question-and-answer flow: a user sends text, the backend forwards it to an LLM, and a complete response is returned. This works well for text because both input and output are discrete and finite.

Voice breaks this model. Audio is continuous, not atomic. Waiting for a user to finish speaking before processing adds noticeable delay, and waiting for a full response before playback makes conversations feel slow and unnatural. Traditional HTTP request-response architectures are not designed for this kind of interaction.

WebSockets fit the problem. A persistent, bidirectional connection lets audio, transcripts, model responses, and synthesized speech flow incrementally instead of in isolated requests.

A real-time voice AI system needs a different architecture. At a minimum it has a streaming-capable client, a WebSocket gateway, a Speech-to-Text (STT) service, an LLM orchestration layer, and a Text-to-Speech (TTS) system.

This article covers how these components fit together and why streaming and low-latency design are fundamental to real-time voice-based LLM systems.

The problem: high latency

High latency is the biggest reason most voice-based LLM systems feel unnatural.

A typical naïve architecture follows a strict, sequential flow: record audio → transcribe → send to LLM → generate response → synthesize speech → play audio. Each step waits for the previous one to finish, pushing the first audible response several seconds late.

LLM speed is not the main cause. The system treats audio, text, and responses as static payloads instead of streams.

• Audio is processed only after recording completes
• STT runs on full clips, not partial speech
• LLMs respond only once the full prompt is ready
• TTS waits for the entire response before playback

The result is a batch-oriented pipeline that cannot support real-time conversation.

Solving this requires a streaming-first architecture where STT, LLM, and TTS operate concurrently, minimizing end-to-end latency instead of optimizing individual components.

Why WebSockets over REST and SSE

Real-time voice AI systems need continuous, low-latency, bidirectional communication. That rules out traditional REST APIs.

REST is request-response by design. Each interaction requires a new HTTP request, which makes it unsuitable for streaming audio, partial transcriptions, or incremental model responses. Repeated connections and the lack of real-time bidirectional flow add unnecessary latency.

Server-Sent Events (SSE) improve on REST by letting the server push updates to the client over a persistent connection. However, SSE is unidirectional. It works for streaming text responses, but it cannot efficiently carry upstream audio from the client to the server.

WebSockets solve both problems. They establish a single, long-lived connection that supports full-duplex communication, so the client can stream audio while simultaneously receiving partial transcripts, LLM responses, and synthesized speech. That makes it possible to overlap STT, LLM, and TTS processing instead of running them sequentially.

In a real-time voice architecture, WebSockets are a requirement, not an optimization. They provide the communication model needed to treat voice interactions as continuous streams instead of isolated requests.

High-level architecture overview

A real-time voice AI system is best understood as a streaming pipeline, where components operate concurrently instead of in isolation. Each layer is designed to handle partial data, which reduces end-to-end latency and keeps the conversation natural.

Core components

• Client: captures microphone audio, plays synthesized speech, and maintains a continuous connection to the backend
• WebSocket gateway: provides a persistent, bidirectional channel and acts as the system's real-time coordination layer
• Speech-to-Text (STT): transcribes incoming audio streams and produces partial and final text output
• LLM orchestration layer: handles conversation state and turn management, and streams generated tokens downstream
• Text-to-Speech (TTS): converts model output into audio and streams synthesized speech back to the client

Understanding the diagram: data flow in action

The diagram above shows how data flows through the system in real time. A typical interaction goes like this:

1. The user speaks, and audio frames flow from the client to the WebSocket gateway
2. STT emits partial transcripts while the user is still speaking
3. The LLM streams tokens as soon as it has enough context
4. TTS produces audio chunks from the incoming tokens
5. The client starts playing speech before the LLM finishes generating

The timelines overlap. Unlike a sequential pipeline where each stage waits for the previous one to finish, this architecture lets multiple operations run concurrently. When the user finishes speaking, the system has already begun synthesizing a response.

The WebSocket gateway acts as the coordination point, routing messages between subsystems and managing control signals for interruptions and turn-taking.

End-to-end data flow

Once the architecture is in place, data moves through the system as a continuous stream rather than discrete requests.

1. The client streams audio frames to the backend over WebSockets
2. STT transcribes speech incrementally and emits partial text
3. The LLM consumes transcripts and streams generated tokens
4. TTS converts tokens into audio chunks
5. Audio is streamed back to the client for immediate playback

Because each stage processes data as it arrives, transcription, reasoning, and synthesis overlap in time. This streaming flow is the key architectural decision that keeps conversational latency within human expectations.

Designing the WebSocket event protocol

With a persistent WebSocket connection in place, the next challenge is defining how data moves across that connection. A real-time voice system streams audio and also coordinates several asynchronous subsystems, which makes the event protocol as important as the transport itself.

A well-designed WebSocket protocol should be:

• Explicit about intent (audio, text, control)
• Stream-friendly, supporting partial and incremental data
• Extensible, allowing new events without breaking clients

Event categories

WebSocket messages fall into three categories:

• Audio events carry raw or encoded audio frames from the client. They are sent at a high frequency and represent an ongoing stream rather than discrete actions.
• Text events contain partial and final transcripts from STT, plus streamed LLM tokens or text chunks. They can flow in both directions depending on the system design.
• Control events signal boundaries such as start and stop speaking, handle interruptions, cancellations, and resets, and coordinate turn-taking across the pipeline.

Separating events by intent avoids overloading a single message type and keeps the protocol understandable as the system grows.

Streaming semantics

The protocol must treat all major data types (audio, text, and speech output) as streams, not single payloads. This means:

• Audio frames are sent continuously until a stop signal is emitted
• Transcripts are updated incrementally
• LLM responses are streamed token-by-token or chunk-by-chunk
• TTS output begins before the full response is available

Control events let the system interrupt or cancel downstream processing when new audio arrives, which prevents stale responses from being synthesized or played.

Designing the protocol around streams and control signals is what enables low-latency, interruption-aware voice interactions, instead of a request-response exchange over a persistent connection.

Example protocol structure

Here is an example of how WebSocket events might be structured:

// Client → Server: Audio stream
{
  "type": "audio.input",
  "data": "base64EncodedAudioChunk",
  "format": "pcm16",
  "sampleRate": 16000
}

// Server → Client: Partial transcript
{
  "type": "transcript.partial",
  "text": "Hello, I need help with",
  "isFinal": false,
  "timestamp": 1234567890
}

// Server → Client: Final transcript
{
  "type": "transcript.final",
  "text": "Hello, I need help with my account.",
  "timestamp": 1234567895
}

// Server → Client: LLM token stream
{
  "type": "llm.token",
  "token": "I'd",
  "isComplete": false
}

// Server → Client: Audio output
{
  "type": "audio.output",
  "data": "base64EncodedSynthesizedAudio",
  "format": "pcm16",
  "sampleRate": 24000
}

// Client → Server: Interrupt current response
{
  "type": "control.interrupt",
  "reason": "user_speaking"
}

// Server → Client: Response complete
{
  "type": "control.response_complete",
  "tokenCount": 45,
  "audioLengthMs": 3200
}

This structure keeps events lightweight while providing all necessary metadata for debugging, monitoring, and coordinating the pipeline.

Real-world reference: OpenAI Realtime API

OpenAI's Realtime API (released in 2024) uses many of these patterns in production. It uses WebSockets for bidirectional communication, supports function calling during voice conversations, and runs server-side Voice Activity Detection (VAD) to handle turn-taking automatically.

Similarly, providers like Deepgram and AssemblyAI offer streaming STT APIs that emit partial transcripts over WebSocket connections, allowing applications to begin processing before a user finishes speaking. On the TTS side, services like ElevenLabs and PlayHT stream audio chunks incrementally, enabling near-instantaneous playback.

Streaming-first design is increasingly standard for voice AI systems.

Latency optimizations

In a real-time voice AI system, latency comes down to a series of small, compounding decisions across the pipeline. Each layer has to reduce waiting and maximize overlap.

Small audio buffer sizes

Audio should be captured and transmitted in small frames rather than large chunks. Smaller buffers reduce the delay between speech and transcription, allowing downstream components to begin processing almost immediately. The goal is to balance responsiveness with network overhead, favoring lower latency over throughput.

// Example: Client-side audio buffering
const BUFFER_SIZE = 4096; // Small buffer for low latency
const SAMPLE_RATE = 16000;

navigator.mediaDevices.getUserMedia({ audio: true }).then((stream) => {
  const audioContext = new AudioContext({ sampleRate: SAMPLE_RATE });
  const source = audioContext.createMediaStreamSource(stream);
  const processor = audioContext.createScriptProcessor(BUFFER_SIZE, 1, 1);

  processor.onaudioprocess = (e) => {
    const audioData = e.inputBuffer.getChannelData(0);
    // Send immediately - don't wait for large chunks
    websocket.send(
      JSON.stringify({
        type: "audio.input",
        data: encodeAudioData(audioData),
        timestamp: Date.now(),
      }),
    );
  };

  source.connect(processor);
  processor.connect(audioContext.destination);
});

Partial STT usage

Streaming STT services emit partial transcripts before speech segments are complete. Forwarding these partial results to the LLM allows response generation to start early, instead of waiting for final transcripts. Even imperfect partial text is often sufficient to prime the model and reduce perceived response time.

# Example: Server-side partial transcript handling
async def handle_stt_stream(audio_stream, websocket):
    async for transcript_event in stt_service.stream(audio_stream):
        if transcript_event.is_partial:
            # Send partial transcript to client
            await websocket.send_json({
                'type': 'transcript.partial',
                'text': transcript_event.text,
                'isFinal': False
            })

            # Start LLM processing early with partial text
            if len(transcript_event.text) > 20:  # Sufficient context
                asyncio.create_task(
                    prepare_llm_context(transcript_event.text)
                )
        else:
            # Final transcript - trigger full LLM response
            await websocket.send_json({
                'type': 'transcript.final',
                'text': transcript_event.text,
                'isFinal': True
            })
            await generate_and_stream_response(
                transcript_event.text,
                websocket
            )

Token-to-TTS piping

Rather than waiting for a full LLM response, generated tokens should be streamed directly into the TTS system. This lets speech synthesis begin while the model is still reasoning, which cuts the time before the first audio is heard.

# Example: Streaming tokens directly to TTS
async def generate_and_stream_response(transcript, websocket):
    buffer = []

    async for token in llm.stream(transcript):
        buffer.append(token)

        # Send token to client for display
        await websocket.send_json({
            'type': 'llm.token',
            'token': token
        })

        # When we have enough tokens for natural speech, synthesize
        if len(buffer) >= 10 or is_punctuation(token):
            text_chunk = ''.join(buffer)
            buffer.clear()

            # Stream to TTS immediately
            async for audio_chunk in tts.synthesize_stream(text_chunk):
                await websocket.send_json({
                    'type': 'audio.output',
                    'data': base64.b64encode(audio_chunk).decode()
                })

Early stream cancellation

Voice interactions are interrupt-driven. When new audio arrives, ongoing LLM inference or TTS synthesis should be canceled immediately. Early cancellation prevents wasted computation and avoids playing outdated responses, keeping the system responsive to user intent.

# Example: Cancellation handling
class ConversationSession:
    def __init__(self):
        self.current_llm_task = None
        self.current_tts_task = None

    async def handle_interrupt(self, websocket):
        # Cancel ongoing operations immediately
        if self.current_llm_task and not self.current_llm_task.done():
            self.current_llm_task.cancel()

        if self.current_tts_task and not self.current_tts_task.done():
            self.current_tts_task.cancel()

        # Notify client
        await websocket.send_json({
            'type': 'control.cancelled',
            'reason': 'user_interrupt'
        })

        # Clear audio playback buffer on client
        await websocket.send_json({
            'type': 'control.clear_audio'
        })

Minimal payload size

WebSocket messages should carry only what is necessary. Compact audio encoding, lightweight event metadata, and minimal JSON structures reduce serialization overhead and network latency. In streaming systems, even small payload savings matter when multiplied across hundreds of messages per interaction.

Together, these optimizations move the system from a batch-oriented pipeline to a streaming architecture where useful work happens continuously.

Final thoughts

Building real-time voice AI depends on architecture at least as much as on model choice. Most latency and reliability problems come from how STT, the LLM, and TTS are connected and coordinated, not from the components themselves.

A streaming-first design with WebSockets as the backbone for bidirectional communication makes voice interactions feel immediate and natural instead of delayed and mechanical.

Real-time voice AI is a systems problem. Small architectural decisions, such as buffer sizes, streaming boundaries, cancellation semantics, and protocol design, compound into large differences in user experience. If the architecture is wrong, no amount of model quality will hide the latency.