OpenAI Realtime fires on ambient speech. Selective Auditory Attention (SAA) filters it first.

The false-trigger problem

OpenAI Realtime's server VAD fires on any voice above a noise floor, with no addressee signal.

In a drive-thru lane, meeting room, or clinic, the VAD fires on every voice it hears. Each crossing bills a full model call: audio tokens, generation, TTS.

SAA (Selective Auditory Attention) is the hosted addressee gate between mic capture and the WebSocket. See what addressee detection is and how it differs from VAD.

The missing pipeline layer

SAA is the L5 addressee gate. VAD answers a different question entirely.

A standard voice pipeline has five layers. None of the first four decide whether a turn is addressed to the agent. SAA is the layer that does. See the pre-ASR pipeline layer and the 5-layer voice stack.

See the engagement-control layer SAA implements.

Architecture

AttentionClient streams mic and camera to SAA. turnReady delivers audio ready for the Realtime API.

AttentionClient streams mic and webcam audio to SAA. Addressed utterances emit turnReady as base64 PCM16, OpenAI's native input_audio format. Non-addressed speech never reaches the WebSocket.

turnReady handler

client.on('turnReady', ({ audio, frames }) => {
  // audio = base64-encoded PCM16 at 24000 Hz -- native Realtime input_audio format
  // No transcoding needed: pass the payload directly into conversation.item.create
  const content = [{ type: 'input_audio', audio }];

  // Optional: JPEG frames from the webcam map to input_image content parts
  if (frames?.length) {
    frames.forEach(f => content.push({ type: 'input_image', image: f }));
  }

  ws.send(JSON.stringify({
    type: 'conversation.item.create',
    item: { type: 'message', role: 'user', content }
  }));
  ws.send(JSON.stringify({ type: 'response.create' }));
});

The key integration step

Set turn_detection: null in session.update to hand turn detection to SAA.

Setting turn_detection: null in session.update disables OpenAI's server VAD entirely; SAA decides each turn boundary instead. Other session fields are unchanged.

ws.addEventListener('open', () => {
  // Disable OpenAI server VAD; hand the turn boundary decision to SAA
  ws.send(JSON.stringify({
    type: 'session.update',
    session: {
      turn_detection: null,       // critical: prevents OpenAI from acting on ambient speech
      input_audio_format: 'pcm16',
      modalities: ['text', 'audio']
      // other session fields (voice, instructions, tools) unchanged
    }
  }));
});

Echo suppression

Mute the mic during playback so the agent's own TTS never re-enters the addressee model.

Without this, TTS re-enters the mic and can restart the loop. Call client.mute() and client.markResponding(true) at playback start; hold ~400ms after it ends, then unmute.

ws.addEventListener('message', ({ data }) => {
  const ev = JSON.parse(data);

  if (ev.type === 'response.audio.delta') {
    // Playback started: suspend SAA addressee predictions
    client.mute();
    client.markResponding(true);
  }

  if (ev.type === 'response.audio.done') {
    // Playback ended: hold ~400ms to absorb room reverb and speaker tail,
    // then resume addressee detection
    setTimeout(() => {
      client.unmute();
      client.markResponding(false);
    }, 400);
  }
});

Barge-in

SAA's interrupt event wires to response.cancel and a gain fade for clean barge-in.

SAA keeps listening while the agent talks. On barge-in it emits interrupt with a fadeMs value: fade playback, send response.cancel, and unmute the mic for the next turnReady.

// SAA emits an interrupt event when it detects addressed speech mid-response
client.on('interrupt', (e) => bridge.interrupt(e.fadeMs));

function interrupt(fadeMs) {
  // Fade and stop Realtime audio playback over fadeMs milliseconds
  gainNode.gain.linearRampToValueAtTime(0, ctx.currentTime + fadeMs / 1000);

  setTimeout(() => {
    audioBufferQueue.length = 0;

    // Halt upstream model generation
    ws.send(JSON.stringify({ type: 'response.cancel' }));

    // Unmute mic so the user's new utterance enters the next turnReady turn
    client.unmute();
    client.markResponding(false);
  }, fadeMs);
}

What gets filtered

Every suppressed utterance is a model call that never runs.

Spend tracks user intent, not acoustic activity. A TV, a door, or a second speaker all clear VAD but never reach the WebSocket. See when your agent responds to the wrong person.

Getting started

Get an SAA token, install the Cloud SDK, wire prewarm() for near-instant first turns.

Sign up at the dashboard for a free, self-serve SAA token, then install @attenlabs/saa-js (npm) or attenlabs-saa (PyPI). prewarm() opens the WebSocket early for a near-instant first turn.

Install

# JavaScript (npm) -- for browser-side Realtime API adapter
npm install @attenlabs/saa-js

# Python (PyPI) -- for server-side addressee gating
pip install attenlabs-saa

Initialize and prewarm

import { AttentionClient } from '@attenlabs/saa-js';

const client = new AttentionClient({ token: SAA_TOKEN });

// prewarm() opens the Realtime WebSocket during SAA model warmup to absorb
// handshake and session.update latency before the first addressed utterance
let ws;
client.prewarm(() => {
  ws = new WebSocket(
    'wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview',
    ['realtime', `openai-insecure-api-key.${OPENAI_KEY}`, 'openai-beta.realtime-v1']
  );
  // Disable server VAD on open
  ws.addEventListener('open', () => {
    ws.send(JSON.stringify({
      type: 'session.update',
      session: { turn_detection: null, input_audio_format: 'pcm16', modalities: ['text', 'audio'] }
    }));
  });
});

// Acquire mic (and optionally webcam) and start the addressee model
await client.start({ video: true });

Common questions

What developers building on the OpenAI Realtime API ask before integrating SAA.

Why does OpenAI Realtime respond to background talkers?

OpenAI Realtime's server VAD triggers a response on any speech that clears an energy threshold, with no addressee signal: it cannot tell whether someone is talking to the agent or to someone nearby. In a drive-thru lane, a conference room, or anywhere ambient speech reaches the mic, every utterance that clears VAD fires an API call and charges tokens.

How does SAA replace OpenAI's built-in turn detection?

SAA sends turn_detection: null in the session.update handshake, disabling OpenAI's server VAD entirely, then evaluates each utterance itself and emits turnReady only when speech is confidently addressed to the agent. The application sends one conversation.item.create followed by response.create. SAA fails closed: when uncertain, it suppresses rather than forwards.

What audio format does SAA deliver, and does it need transcoding?

turnReady delivers the utterance as base64 PCM16 at 24000 Hz, OpenAI Realtime's native input_audio format, so no transcoding is needed: pass the payload directly into a conversation.item.create content array. Optional webcam JPEG frames map to input_image parts in the same message. See arXiv:2604.08412 for the addressee model's evaluation.

How do I stop the agent from hearing its own TTS?

Call client.mute() and client.markResponding(true) when playback starts, and unmute plus markResponding(false) when it ends. This suspends addressee predictions so the agent's own TTS never re-enters the model. Hold ~400ms after playback to absorb room reverb before detection resumes.

Can a user interrupt the agent mid-response?

Yes. SAA keeps monitoring for addressed speech while the agent talks. On barge-in it emits interrupt with a fadeMs value: wire client.on('interrupt', (e) => bridge.interrupt(e.fadeMs)) to fade playback, send response.cancel, and unmute the mic. SAA pre-rolls the barge-in audio so the next turn carries the user's actual question.

Does SAA require changes to my OpenAI account?

No changes to your OpenAI account are needed. SAA uses its own API key, self-serve from the dashboard. Pass the SAA token to AttentionClient and your OpenAI key to the WebSocket; the two services are billed independently. In production, proxy the WebSocket server-side so the OpenAI key never reaches the browser.

Next step

Get your API key and start building today.

Sign up at the dashboard for a free, self-serve API token, no approval needed. Want to test first? Request a free eval kit.

Related: what addressee detection is and how it differs from VAD · the pre-ASR pipeline layer and the 5-layer voice stack · when your agent responds to the wrong person · LiveKit integration guide · Pipecat integration guide

Last updated June 29, 2026