1 · How it works
A caller dials a client's business number. Twilio hits POST /voice, which returns TwiML that opens a bidirectional Media Stream to /media-stream. The engine bridges that stream to the OpenAI Realtime API - the caller talks to the AI live. Both legs are G.711 μ-law @ 8 kHz, so audio passes through with no resampling.
The AI hears the caller, speaks back, and can call three tools: capture_lead (record name/number/reason), transfer_to_human (request a live handoff - allowed or denied by your policy, not the model), and end_call (say a closing line and hang up when the caller says goodbye - no dead air on the meter). On hangup the capture, an AI-written 1–2 sentence summary, and the transcript are delivered, and usage is metered.
Brand-neutral by design. The caller only ever hears the client's business name - never "VocalizeCloud", "Twilio", or "OpenAI". The product name lives here in the operator tooling, not on the phone.
| Piece | What |
|---|---|
| Host | Railway - service vc, vc-production-517e.up.railway.app |
| Console | console.worldvc.business (and /console) |
| Telephony | Twilio (YoreVox account), Media Streams |
| AI | OpenAI Realtime API |
| Store | Postgres (Railway) - agent_config, call_capture, call_usage |
2 · Onboard a number
Everything is per-number and hot-swappable - a config change takes effect on the next call to that number, no redeploy.
- In Twilio, point the number's Voice webhook at
https://vc-production-517e.up.railway.app/voice(HTTP POST). - Open the console, paste the
ADMIN_TOKEN, click Connect. - + New, then fill the number's card:
| Field | Notes |
|---|---|
| Number | E.164, e.g. +19205330466. This is the primary key and the billable client. |
| Business name | What the caller hears. "Northside Dental", not a product name. |
| Voice | OpenAI voice - sage (default), alloy, echo, shimmer, coral, ballad, verse. |
| Greeting | The exact first line the AI speaks. Blank = a sensible default. |
| Instructions | Persona & behavior. Blank = default that tells the AI to capture leads. Add domain rules here (hours, services, what to never promise). |
| Handoff SMS | E.164 that receives a text when a lead is captured. Blank = off. |
| Transfer number | The human's phone. Presence of this is what offers transfer to the AI at all. |
Click Save. Call the number to verify.
3 · Transfer policy
The AI can request a transfer; the engine decides whether to allow it via evaluateTransfer(). You control this per number - the model never gets the final say.
| Mode | Behavior |
|---|---|
| always | Any transfer request connects the caller to the human number. |
| never | AI only. Requests are denied; the AI takes a message instead. |
| rules | Allowed only when the rules below pass. This is the nuanced mode. |
| (blank) | No explicit policy - falls back to rules defaults. |
Rules-mode fields
- Business hours - timezone (e.g.
America/New_York), start/end (09:00–18:00), and days. Outside hours → denied, message taken. Hours are evaluated in the given timezone viaIntl.DateTimeFormat. - Caller asks for a person - allow when the trigger is a direct request.
- Urgent matters - allow when urgency is high.
- Trigger keywords - comma-separated (
emergency, complaint). If the trigger matches, allow.
When a transfer is allowed, the AI tells the caller it's connecting them, then after ~3.5 s the live call is redirected (Twilio call-update to
<Dial>). The capture collected so far is delivered before the handoff, so nothing is lost.Whisper: when the human answers, they hear "Incoming call, a customer for <name>. Connecting you now." before the line opens - the caller holds until it finishes. The name is
announce_name when set, else business_name; set it per number when several businesses ring the same phone (admin API field announceName, or the Communications app's Call handling panel).If the human doesn't answer (
timeout="20"), Twilio falls to /voice/transfer-fallback: the caller hears an apology and can leave a recorded voicemail (persisted as call_recording.kind = 'voicemail', playable from the console and the Communications Calls tab). So "transfer" degrades gracefully to "message" - it never dead-ends.Self-serve: clients configure all of this themselves from the Communications app - each voice-enabled number's Call handling panel writes transfer mode/hours, the transfer number (any country - enable that country in Twilio Geo Permissions), the announcement name, and recording via this engine's admin API.
4 · Capture delivery
On hangup, deliverCapture() fires all three channels in parallel (Promise.allSettled - one failing never blocks the others):
| Channel | Detail |
|---|---|
| Postgres | call_capture row - structured lead, AI summary, transcript, seconds + tokens. Always on (when a DB is configured). This backs the Captures tab. |
| SMS | Sent from the client's own Twilio number to the Handoff SMS number; leads with the AI summary. Blank = off. |
| Webhook | Signed POST to the configured URL (Nexus/workspace/CRM ingest). Blank = off. |
Webhook contract
POST JSON: { type, business, call:{sid,to,from}, lead, summary, transcript, usage:{seconds,inputTokens,outputTokens}, at }. If a webhook secret is set, the request carries x-vc-signature: sha256=<HMAC-SHA256 of the raw body>. Verify it before trusting the payload. The CRM module's POST /api/ingest verifies this exact signature.
signature = "sha256=" + hmac_sha256(secret, rawRequestBody)
// constant-time compare against the x-vc-signature header5 · Usage & billing
Every call writes a call_usage row attributed to the called number (= the client). It records call seconds and the AI input_tokens/output_tokens summed from each response.done.
The Usage tab (and GET /admin/usage) rolls this up per number for the current month, or ?month=YYYY-MM. Minutes are rounded up per call (billing convention); tokens are raw sums.
Local metering is live now. Forwarding these rows to a control-plane
/usage endpoint (so metered spend lands on a tenant invoice) is the marked TODO in usage.ts - wire it when the control plane runs and numbers map to tenants.6 · Call forwarding (GSM)
To route an existing mobile line into a VocalizeCloud number, use the carrier's call-forwarding codes from the handset:
| Code | Action |
|---|---|
**21*<number># | Forward all calls unconditionally. |
*#21# | Check unconditional-forwarding status. |
##21# | Cancel unconditional forwarding. |
**61*<number># | Forward on no-answer. |
Gotcha seen in testing: on an unregistered PH SIM,
**21*…# can return "Setting Registration Failed" and international forwarding may be blocked carrier-side (Globe). That's a carrier limitation, not an engine fault - register the SIM or forward to a domestic number first.7 · Environment (Railway)
| Var | Purpose |
|---|---|
| OPENAI_API_KEY | OpenAI Realtime. At least one provider key (this or XAI_API_KEY) must be set or the media socket closes immediately. |
| XAI_API_KEY | Grok Voice (xAI) as an alternative realtime provider. Per-number choice: provider = openai | grok | elevenlabs on the agent config (admin PUT); AGENT_PROVIDER sets the default. An unconfigured provider falls back to the next available engine with a loud log, never dead air. Grok voices: ara, rex, sal, eve, leo (non-Grok voice names fall back to GROK_DEFAULT_VOICE, default ara). Tunables: GROK_REALTIME_MODEL (default grok-voice-latest). |
| ELEVENLABS_API_KEY / ELEVENLABS_AGENT_ID | ElevenLabs Agents as a third realtime provider. One-time dashboard setup: create a template agent with telephony audio (ulaw_8000 in/out), client tools capture_lead / transfer_to_human / end_call, and overrides enabled for prompt, first message and voice - every number then rides it with per-call overrides. Voices: rachel, adam, antoni, bella, josh (ELEVENLABS_DEFAULT_VOICE default rachel). |
| DEEPGRAM_API_KEY | Deepgram Voice Agent as a fourth realtime provider - STT + LLM + TTS over one WebSocket, RAW binary μ-law both ways, no dashboard agent to build. The "think" LLM is Deepgram-orchestrated (default OpenAI gpt-4o-mini - add your LLM provider key in the Deepgram dashboard, or override DEEPGRAM_THINK_TYPE/DEEPGRAM_THINK_MODEL; STT via DEEPGRAM_LISTEN_MODEL, default nova-3). Aura voices: thalia, andromeda, apollo, arcas, helena (DEEPGRAM_DEFAULT_VOICE default thalia). The three client tools run in the engine exactly as for the other providers. |
| DATABASE_URL | Postgres. Unset = env-only mode (no store, no console data). |
| ADMIN_TOKEN | Bearer token for /admin/* and the console login. Timing-safe compared. |
| TWILIO_ACCOUNT_SID | For outbound SMS + call-update (transfer). |
| TWILIO_AUTH_TOKEN | Twilio auth + inbound signature validation. Never paste in chat - set it directly in Railway. |
| PUBLIC_BASE_URL | Public https base; used to build the wss:// stream URL and callback URLs. |
| OPENAI_REALTIME_MODEL | Model id. preview in the name switches to the preview API shape. |
| TWILIO_VALIDATE | Set false to disable signature checks (debug only). Absent + auth token present = validation on. |
| RECORD_CALLS | true = record calls for numbers with no saved config (env-default agent). Per-number toggle in the console overrides. If on, disclose in the greeting. |
| OPENAI_SUMMARY_MODEL | Model for the per-call summary (default gpt-4o-mini). |
Health: GET /healthz → { ok, realtime, configStore, admin }. All-green means the key, Postgres, and admin token are all set.
8 · Troubleshooting
| Symptom | Likely cause → fix |
|---|---|
| Call connects then drops | /healthz shows missing OPENAI_API_KEY → set the key in Railway. |
| Connected but silent | Historically two causes, both fixed: greeting sent before the session's audio format applied, and a GA vs preview event-name mismatch. If it recurs, check logs for [openai] receiving audio - absence means the model side never produced audio. |
403 on /voice | Twilio signature mismatch. Confirm PUBLIC_BASE_URL matches the exact webhook URL Twilio calls (scheme + host), or the proxied host headers line up. |
| No SMS delivered | Check the client number can send SMS in Twilio; PH delivery can be finicky. The DB capture still lands - SMS is best-effort. |
| Transfer never rings | Confirm a transfer number is set and the policy allows it at call time (mode/hours). Logs show [transfer] … ALLOW/DENY with the reason. |
| Console shows no data | configStore is env-only → no DATABASE_URL. The console reads the store; without a DB there's nothing to list. |
| Caller hears endless ringing / busy, call never connects | Twilio error 12100 "Document parse failure" - our TwiML was invalid XML. Root cause seen live: an unescaped & between query params inside an attribute. Check Twilio Console → Monitor → Logs → Calls for the error code; validate any TwiML change is well-formed XML (escape & as &). |
| Agent doesn't hang up after goodbye | The end_call tool handles this (says a closing line, completes the call ~3s later). If it recurs, check logs for [call] agent ending call - absence means the model never invoked the tool; tighten the instructions. |
| No recording on the capture | Recording is per-agent opt-in - confirm the toggle is saved. Twilio finalizes recordings a few seconds after hangup; reload the Captures tab. Logs show [recording] started / [recording] ready. |
9 · Admin API
All routes take Authorization: Bearer <ADMIN_TOKEN>. The console is a UI over exactly these.
| Route | Does |
|---|---|
| GET /admin/agents | List every configured number. |
| GET /admin/agents/:number | One number's config (peers use this to merge-and-PUT). |
| PUT /admin/agents/:number | Create or update a number's config (E.164). Accepts tenantSlug and announceName (transfer-whisper name). Takes effect next call. |
| DELETE /admin/agents/:number | Remove a number's config. |
| POST /admin/numbers/attach | Point an existing account-owned number's VoiceUrl at this engine (returns previousVoiceUrl). The engine is the single writer of Twilio voice config - peer modules ask here. |
| POST /admin/numbers/detach | Restore a number's VoiceUrl (voiceUrl in body; empty clears). |
| GET /admin/captures[?to=] | Last 50 captures (with recording refs incl. voicemail kind), optionally filtered to one number. |
| GET /admin/usage[?month=YYYY-MM] | Per-number monthly rollup (calls, minutes, tokens). |
| GET /admin/recordings/:sid | Streams a call recording or voicemail (mp3) via our credentials - Bearer header or ?token=. The console's players use this; Twilio URLs never reach the browser. |
curl -sX PUT https://vc-production-517e.up.railway.app/admin/agents/+19205330466 \
-H "Authorization: Bearer $ADMIN_TOKEN" -H "Content-Type: application/json" \
-d '{"businessName":"Northside Dental","greeting":"Thanks for calling Northside Dental!",
"transferNumber":"+1512...","notifySms":"+1512...",
"transferPolicy":{"mode":"always"}}'