WebSocket Server

A WebSocket server is the runtime piece on your side that the Sinch Voice API talks to whenever a call leg is routed to a STREAM (raw PCM audio) or VOICE_RELAY (text-in/text-out) destination. The other tutorials in this section focus on the Sinch side: the SVAML payload that points a call at your server. This tutorial is the server-side reference: what messages you must handle, in what order, how to reply with audio or text, and how to ship the thing to production.

The fastest way to understand it is to run an echo server and connect a tiny WebSocket client to it locally. No phone call, no ngrok, and no Sinch account required. You see the exact frames a real call would send, echoed straight back. Start there (below), then read the protocol reference and the production checklist.

There are two distinct WebSocket protocols, picked by to.type in your SVAML:

• STREAM: raw PCM audio frames interleaved with JSON control messages. Pairs with Stream Call Audio in Real-Time.
• VOICE_RELAY: text-only protocol; Sinch does STT and TTS for you. Pairs with Connect an AI Chatbot (Voice Relay).

This tutorial ships minimal echo servers for both, in Node.js and Python.

All sample code lives in the sinch/sinch-voice-tutorials repository. Clone it and change into this tutorial's folder:

git clone https://github.com/sinch/sinch-voice-tutorials.git
cd sinch-voice-tutorials/4.4-websocket-server

The scripts for this tutorial are under scripts/. Every relative link in this page (../4.1-voice-relay/, ../4.2-stream-audio/, and so on) resolves against the same repo when you browse it on GitHub or on your own clone, so you can follow the cross-references without leaving the tree.

You need one of these runtimes:

For the local first-success test below, that is all you need. To drive a server from a real call you additionally need:

• A Sinch account with API credentials and a virtual number (see Outbound Call or Inbound PSTN).
• A publicly reachable wss:// endpoint. ngrok gives you one for free: ngrok http 8765.

The scripts default to port 8765 (override with the PORT environment variable). Note: the shared .env at the repository root sets PORT=8081 for other tutorials, so either unset it for this one or pass PORT=8765 explicitly so the port matches the ngrok http 8765 commands below.

You can validate the entire WebSocket contract on your laptop before touching Sinch. Run an echo server, point a local client at it, and watch the frames come back.

# Python
pip install websockets
python scripts/ws-stream-server.py        # listens on ws://localhost:8765

# ...or Node.js
npm install ws
node scripts/ws-stream-server.node.js

You should see STREAM echo server on :8765.

Save this as test-stream.py and run it in a second terminal (pip install websockets if you haven't):

import asyncio, json, websockets

async def main():
    async with websockets.connect("ws://localhost:8765") as ws:
        # Pretend to be Sinch opening the call:
        await ws.send(json.dumps({"command": "connect", "callId": "test-1", "applicationId": "app-1"}))
        print("server replied:", await ws.recv())     # expect {"command":"answer"}
        # Pretend to be the caller's audio (4 bytes of fake PCM):
        await ws.send(b"\x01\x02\x03\x04")
        echo = await ws.recv()
        print("audio echoed back:", echo)              # expect b'\x01\x02\x03\x04'

asyncio.run(main())

server replied: {"command": "answer"}
audio echoed back: b'\x01\x02\x03\x04'

That is the complete STREAM handshake: you sent connect, the server answered, you sent a binary PCM frame, and it came straight back. On a real call those binary frames are the caller's voice and the echo is what the caller hears.

python scripts/ws-relay-server.py          # or: node scripts/ws-relay-server.node.js

import asyncio, json, websockets

async def main():
    async with websockets.connect("ws://localhost:8765") as ws:
        await ws.send(json.dumps({"command": "connect", "callId": "test-1"}))
        print(await ws.recv())   # {"command":"answer"}
        print(await ws.recv())   # greeting text
        await ws.send(json.dumps({"command": "text", "text": "hello world"}))
        print(await ws.recv())   # {"command":"text","text":"You said: hello world","isLast":true}

asyncio.run(main())

Once both pass, the plumbing is proven correct for these echo cases. Everything after this is wiring the same server up to a live call and replacing the echo with real STT/LLM/TTS.

1. Expose the running server: ngrok http 8765, then copy the wss://<id>.ngrok-free.app Forwarding URL.
2. Put that URL in the endpoint of a STREAM or VOICE_RELAY destination. See Stream audio (STREAM, outbound trigger) or Voice relay (VOICE_RELAY, inbound via dashboard).
3. Place the call. A STREAM server will echo your voice back to you; a VOICE_RELAY server will read back whatever you say.

The shared .env field WS_ENDPOINT holds the wss:// URL used by the call-trigger scripts in Stream audio/Voice relay. Update it to your current ngrok URL each session (ngrok URLs change on restart unless you have a reserved domain).

• Custom STT/TTS pipeline: You run your own ASR (for example Whisper) and TTS (for example Coqui). Use STREAM so Sinch never touches the audio's meaning.
• Embedded LLM with off-the-shelf voice: You do not want to host ASR/TTS. Use VOICE_RELAY and keep your server in pure text.
• Hybrid agent: Most turns go through VOICE_RELAY; rare "play this exact recording" turns hand off via a separate SVAML branch.
• Auditing or wiretap: Stream a copy of the audio to an auditing service alongside the live conversation (set up two STREAM legs on one call).

When Sinch dials to: { type: "STREAM", stream: { endpoint: "wss://..." } }, it opens a WebSocket to your endpoint and exchanges these messages:

The audio is linear PCM, codec fixed to PCM, at the sample rate negotiated in streamOptions.sampleRate (one of 8000 / 16000 / 24000 / 44100 / 48000 / 96000 Hz; default 8000). Frame size is implementation-defined: write whatever PCM bytes you have, when you have them.

Verify: the field and command names above (connect, answer, busy, reject, clear, heartbeat, headers, callId, applicationId) come from the 4.2 tutorial and the working sample servers, not from the OpenAPI spec. The spec defines only the SVAML side (endpoint, streamOptions.codec, streamOptions.sampleRate, streamOptions.version, callHeaders[]). The on-the-wire JSON protocol is documented in the Streams product docs. Confirm exact names there before relying on busy / reject / clear.

scripts/ws-stream-server.node.js, requires npm install ws:

import { WebSocketServer } from "ws";

const PORT = Number(process.env.PORT || 8765);
const wss = new WebSocketServer({ port: PORT });
console.log(`STREAM echo server on :${PORT}`);

wss.on("connection", (ws, req) => {
  console.log(`new connection from ${req.socket.remoteAddress}`);

  // Keepalive so corporate proxies don't drop an idle connection.
  const heartbeat = setInterval(() => {
    if (ws.readyState === ws.OPEN) ws.send(JSON.stringify({ command: "heartbeat" }));
  }, 30_000);

  ws.on("message", (data, isBinary) => {
    if (!isBinary) {
      let msg;
      try { msg = JSON.parse(data.toString()); } catch { return; }  // ignore malformed JSON
      console.log("text frame", msg);
      if (msg.command === "connect") ws.send(JSON.stringify({ command: "answer" }));
      return;
    }
    // Binary frame == raw PCM from the caller. Echo it back.
    ws.send(data, { binary: true });
  });

  ws.on("close", () => { clearInterval(heartbeat); console.log("connection closed"); });
});

scripts/ws-stream-server.py, requires pip install websockets:

import asyncio, json, os
import websockets

PORT = int(os.environ.get("PORT", "8765"))

async def heartbeat(ws):
    try:
        while True:
            await asyncio.sleep(30)
            await ws.send(json.dumps({"command": "heartbeat"}))
    except websockets.exceptions.ConnectionClosed:
        return

async def handler(ws):
    print(f"new connection from {ws.remote_address}")
    hb = asyncio.create_task(heartbeat(ws))
    try:
        async for msg in ws:
            if isinstance(msg, (bytes, bytearray)):
                await ws.send(bytes(msg))   # echo PCM
                continue
            try:
                payload = json.loads(msg)
            except json.JSONDecodeError:
                continue                    # ignore malformed JSON
            print("text frame", payload)
            if payload.get("command") == "connect":
                await ws.send(json.dumps({"command": "answer"}))
    finally:
        hb.cancel()
        print("connection closed")

async def main():
    async with websockets.serve(handler, "0.0.0.0", PORT):
        print(f"STREAM echo server on :{PORT}")
        await asyncio.Future()  # run forever

if __name__ == "__main__":
    asyncio.run(main())

Both servers handle the connection lifecycle (heartbeat cancelled on close), ignore malformed JSON instead of crashing, and never busy-loop.

When to.type is VOICE_RELAY, Sinch does STT and TTS on its end. Your server only ever sees and sends text:

Verify: as with STREAM, these command names are not in the OpenAPI spec. The spec defines only the SVAML voiceRelay object (endpoint, ttsVoice, sttLanguage, enableInterruptions, callHeaders[]). The JSON message shapes come from the Voice Relay product docs (an AsyncAPI definition) and the working reference server. The full reference implementation in Voice Relay (../4.1-voice-relay/server.py) handles every command above.

scripts/ws-relay-server.node.js, requires npm install ws:

import { WebSocketServer } from "ws";

const PORT = Number(process.env.PORT || 8765);
const wss = new WebSocketServer({ port: PORT });
console.log(`VOICE_RELAY echo server on :${PORT}`);

wss.on("connection", (ws) => {
  ws.on("message", (data) => {
    let msg;
    try { msg = JSON.parse(data.toString()); } catch { return; }
    console.log("rx", msg);

    if (msg.command === "connect") {
      ws.send(JSON.stringify({ command: "answer" }));                                   // accept
      ws.send(JSON.stringify({ command: "text", text: "Hi! Anything you say I will repeat back.", isLast: true }));
      return;
    }
    // Sinch delivers transcribed user speech as `text` (or `prompt`).
    if ((msg.command === "text" || msg.command === "prompt") && msg.text) {
      ws.send(JSON.stringify({ command: "text", text: `You said: ${msg.text}`, isLast: true }));
    }
  });

  ws.on("close", () => console.log("connection closed"));
});

scripts/ws-relay-server.py, requires pip install websockets:

import asyncio, json, os
import websockets

PORT = int(os.environ.get("PORT", "8765"))

async def handler(ws):
    print("new connection")
    try:
        async for msg in ws:
            if isinstance(msg, (bytes, bytearray)):
                continue                    # Voice Relay is text-only; ignore stray binary
            try:
                payload = json.loads(msg)
            except json.JSONDecodeError:
                continue
            print("rx", payload)
            cmd = payload.get("command")
            if cmd == "connect":
                await ws.send(json.dumps({"command": "answer"}))
                await ws.send(json.dumps({"command": "text",
                                          "text": "Hi! Anything you say I will repeat back.",
                                          "isLast": True}))
                continue
            if cmd in ("text", "prompt") and payload.get("text"):
                await ws.send(json.dumps({"command": "text",
                                          "text": f"You said: {payload['text']}",
                                          "isLast": True}))
    finally:
        print("connection closed")

async def main():
    async with websockets.serve(handler, "0.0.0.0", PORT):
        print(f"VOICE_RELAY echo server on :{PORT}")
        await asyncio.Future()

if __name__ == "__main__":
    asyncio.run(main())

Both echo servers reply to connect, greet the caller, and read back each transcribed turn. That is enough to confirm the relay plumbing before you swap the echo for a real LLM call (see 4.1 for a LangChain-backed version supporting OpenAI, Claude, and Gemini).

Rule of thumb: reach for VOICE_RELAY (4.1) unless you specifically need to own the audio path; reach for STREAM (4.2) when you run your own ASR/TTS or an audio-to-audio model.

The spec covers only the SVAML side: how you point a call at your server, not the bytes exchanged afterward.

• STREAM (stream schema): requires stream.endpoint (ws:// or wss://, public). streamOptions.codec is fixed to PCM. streamOptions.sampleRate is in {8000, 16000, 24000, 44100, 48000, 96000}, default 8000. streamOptions.version default 1. Optional callHeaders[] (max 16, key/value up to 255 chars).
• VOICE_RELAY (voiceRelay schema): requires voiceRelay.endpoint (URI), ttsVoice, and sttLanguage (BCP-47, for example en-US). Optional enableInterruptions (default true) and callHeaders[] (max 16).
• The on-the-wire JSON/binary protocols between Sinch and your WebSocket server are out of band of the OpenAPI spec; they live in the Streams / Voice Relay product docs. Every command name in this tutorial is marked "verify" above for that reason.

Each echoes back what it receives so you can sanity-check the WebSocket plumbing, using the local client snippets above or a live call, before plugging in a real LLM / ASR / TTS.