> For the complete documentation index, see [llms.txt](https://developer.tiledesk.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.tiledesk.com/widget/voice_streaming.md).

# Voice Streaming

Voice streaming lets the user talk to the chatbot directly from the widget, in real time: the microphone is streamed to a dedicated service, the speech proxy, which transcribes the speech, delivers it to the bot as a regular message, and returns the bot's reply already spoken.

This document explains how it works and the steps required to enable it on a Tiledesk project.

***

## Table of contents

1. [Requirements](#1-requirements)
2. [Step-by-step activation](#2-step-by-step-activation)
3. [How it works](#3-how-it-works)
4. [How provider, voice, and language are chosen](#4-how-provider-voice-and-language-are-chosen)
5. [Barge-in: interrupting the bot](#5-barge-in-interrupting-the-bot)

***

## 1. Requirements

* A Tiledesk project with a **chatbot** that replies to widget conversations.
* An **ElevenLabs** account with an API key. By default, ElevenLabs handles both transcription of the user's voice (STT) and speech synthesis of the bot's reply (TTS).
* A browser with microphone access. The widget must be served over **HTTPS**: browsers do not grant the microphone to insecure pages.
* The speech proxy reachable from the browser. On Tiledesk cloud it is already available.

***

## 2. Step-by-step activation

```mermaid
flowchart TD
    A[Tiledesk dashboard] --> B["1. Settings → Widget → Advanced<br/>enable Voice streaming"]
    B --> C["2. Integrations → ElevenLabs<br/>enable and enter the API key"]
    C --> D["3. Design Studio<br/>the chatbot replies with voice messages (TTS)"]
    D --> E["4. (Optional) Chatbot voice settings<br/>voice, models, language, barge-in"]
    E --> F["5. Open the widget and press<br/>the Stream audio button"]
    F --> G{The browser asks for the microphone}
    G -->|Allow| H[Voice session active]
    G -->|Deny| X[Streaming does not start]
```

### Step 1 — Enable voice streaming in the widget

1. Sign in to the Tiledesk dashboard and open the project.
2. Go to **Project settings → Widget → Advanced**.
3. Enable **Voice streaming** and save.

This setting is stored in the project's widget settings as `showAudioStreamFooterButton: true`. On the next load, the widget reads it and shows the **Stream audio** button (equalizer icon) in the conversation footer.

> For testing you can force the button without going through the dashboard, using the JavaScript setting `window.tiledeskSettings = { showAudioStreamFooterButton: true }` or the URL parameter `tiledesk_showAudioStreamFooterButton=true`.

### Step 2 — Enable and configure ElevenLabs in Integrations

1. In the dashboard, go to **Integrations**.
2. Open **ElevenLabs** and enable the integration.
3. Enter the ElevenLabs **API key** and save.

The key stays associated with the project. When a voice session starts, the speech proxy reads the project's integrations and uses this key both to transcribe the user's voice and to synthesize the bot's replies. If the key is missing, the session is rejected with a provider-not-configured error.

The integration is only for the API key: voice and models are set on the chatbot (step 4).

### Step 3 — Make the chatbot reply with voice messages

The speech proxy speaks only bot replies that contain a **TTS** command. Text-only messages are ignored. After 30 seconds without a TTS reply, the proxy returns the `Bot reply timeout` error and goes back to listening.

In Design Studio the flow must therefore reply with voice content.

### Step 4 — (Optional) Customize the chatbot voice

In the chatbot settings, under **Design Studio → chatbot details → Voice settings**, choose how the bot should speak and listen. The values are saved as chatbot globals and read by the speech proxy on every new session:

| Setting        | Chatbot global       | Effect                                                                                                                             |
| -------------- | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| Voice provider | `VOICE_PROVIDER`     | A single provider, used for both transcription (STT) and synthesis (TTS). With ElevenLabs, the integration from step 2 is required |
| Voice          | `TTS_VOICE_NAME`     | The bot's voice. For ElevenLabs this is the **voice ID**                                                                           |
| TTS model      | `TTS_MODEL`          | Synthesis model (e.g. `eleven_v3`)                                                                                                 |
| STT model      | `STT_MODEL`          | Transcription model (e.g. `scribe_v1`)                                                                                             |
| Language       | `TTS_VOICE_LANGUAGE` | Language of the voice conversation, used both to transcribe and to speak                                                           |
| Barge-in       | `BARGE_IN`           | `true` to let the user interrupt the bot (see [section 5](#5-barge-in-interrupting-the-bot))                                       |

If you set nothing, the proxy uses ElevenLabs with the installation's default voice, models, and language.

### Step 5 — Use streaming from the widget

1. Open the widget and start a conversation with the chatbot.
2. Press the **Stream audio** button in the footer.
3. Allow microphone access when the browser asks for it.
4. The widget shows the session status:
   * **Connecting...**: connecting to the speech proxy.
   * **Listening...**: the microphone is open, you can speak. The transcript appears in the text area as you speak.
   * **Processing...**: the bot is processing or speaking its reply. The reply text is highlighted word by word while the audio plays.
5. Press the button again (the X) to close the voice session.

You do not need to press send: when you stop speaking, the speech proxy sends the sentence to the bot as a conversation message.

***

## 3. How it works

### Architecture

```mermaid
flowchart LR
    U((User)) -->|voice| W[Widget]
    W <-->|"WSS /ws/voice<br/>audio + JSON events"| P[Speech proxy]
    P <-->|audio / text| STT[STT<br/>ElevenLabs]
    P <-->|text / audio| TTS[TTS<br/>ElevenLabs]
    P <-->|"Chat21 messages<br/>(RabbitMQ)"| C[Chat21]
    C <--> B[Tiledesk chatbot]
    P -.->|"project integrations<br/>bot voice settings"| DB[(Tiledesk DB)]
```

The widget opens **a single WebSocket connection** to the speech proxy. On that connection:

* the widget sends microphone audio as binary frames;
* the proxy sends JSON events (session status, transcript, reply text) and the bot's reply audio as binary frames.

The widget does not call ElevenLabs directly and does not publish the messages: transcription, delivery to the bot, and speech synthesis all happen in the proxy.

The widget does not send the transcript as a message: the proxy publishes it at the end of the sentence. That way each sentence appears only once in the conversation.

***

## 4. How provider, voice, and language are chosen

On every new session the speech proxy combines several configuration layers; the most specific one wins:

```mermaid
flowchart BT
    L1["Installation defaults<br/>(proxy environment variables)"] --> L2
    L2["Project integrations<br/>(ElevenLabs API key)"] --> L3
    L3["Chatbot voice settings<br/>(provider, voice, models, language, barge-in)"] --> R[Session configuration]
```

* **API key**: comes from the project's ElevenLabs integration. If the project does not have one, the proxy uses the installation key, when present.
* **Provider**: the one chosen on the chatbot (`VOICE_PROVIDER`), which applies to both STT and TTS. Otherwise the installation default, which is ElevenLabs for both.
* **Voice and models**: those from the chatbot voice settings, otherwise the installation defaults.
* **Language**: the widget sends `lang: "multi"` (automatic detection). The proxy uses the chatbot language (`TTS_VOICE_LANGUAGE`) or the installation default language, because with automatic detection short sentences are often transcribed in the wrong language.

Settings read from the database stay cached for 5 minutes: after you change the integration or the voice settings, the new configuration can take up to 5 minutes to apply to new sessions.

***

## 5. Barge-in: interrupting the bot

By default the bot finishes speaking before the user can reply: during the reply the microphone does not send audio.

With the chatbot global `BARGE_IN = true`, the proxy tells the widget in `session_started` (`bargeIn: true`) and:

* the widget keeps sending audio while the bot speaks; the browser's echo cancellation keeps the bot from hearing itself;
* if the user says at least 2 words after the first second of the reply, the proxy stops synthesis and sends `barge_in`;
* the widget stops the audio immediately, and what the user is saying becomes the next sentence.

***
