Basecamp 6: Voice Agents

The whole technology is a 5-step loop repeated every turn: the caller speaks; speech-to-text transcribes it; the text goes to an LLM that has been pre-prompted with who it is and what it does; the LLM answers in text; text-to-speech says it aloud. Nothing in the middle knows sound exists — which is why everything you learned about prompting LLMs transfers directly, and why transcription accuracy becomes a first-class concern (garbage transcription in, garbage everything out).

Three species share this anatomy: inbound agents (the receptionist and customer-service case this session builds), outbound agents (sales calls — a separate promised session), and interactive assistants living inside apps, Siri-style. The loop's total round-trip time is the latency you'll later tune — the seam where the illusion of conversation holds or breaks.

The economics stack three facts. Scale: UK and US businesses collectively lose on the order of $100B a year to missed calls. Speed: a lead contacted within 5 minutes is 21x more likely to convert than one contacted at 30 minutes — while the average UK business takes four hours to respond. Availability: a voice agent answers at 2 a.m., on holidays, during lunch rushes — 'the perfect employee', with no breaks and no sick days.

The subtle point is that a missed call isn't a neutral non-event: the caller found you through marketing you paid for, and their next act is dialing your competitor one Google result down. The trainer models good epistemics too — he cites Gartner's '90% adoption by 2027' while doubting the number himself: directionally right, precision dubious.

The prompt is how the agent is created — there is no other place its character comes from. The two ingredients are identity (who it is, how it speaks) and context (what situation it operates in, what it's for). The session's contrast pair makes the point cheaply: 'you are a bot that answers questions' produces a robot; 'you are a friendly assistant that helps customers track orders and give clear solutions' — barely a dozen more words — produces something callers can deal with. Small wording investments multiply.

The second half of the concept is how agents act: through APIs, connectors and bridges — 'the backdoor key to using a software or platform without physically being inside of it.' Identity and context make the agent someone; API connections to CRMs, calendars and databases make it useful. The rest of the session builds out both halves.

The economics first: free signup, $10 of credits, usage-based pricing — the demo agent runs about 11.5¢/minute. Then the triage. Matters for beginners: call history (filterable by agent, number, and post-call property values like call_outcome = appointment_booked, with each call's recording, cost, sentiment, latency and transcript), post-call data properties, knowledge bases, phone numbers, and agent-level webhooks. Skippable: workspace webhook settings, reliability/limits, pronunciation, boosted keywords — and batch calling, which died on arrival when Western countries banned AI cold-calling right after launch.

Phone numbers get the practical treatment: buy through Twilio (any country, cheaper than Retell's resold US/CA numbers), set up a SIP trunk, register it in Retell with the international-format number, termination URI and credentials — after which connecting number to agent is a two-click dropdown. Two experience verdicts round it out: Retell's text agents are 'not that good' (build text agents elsewhere), and the new alerting/AI-quality-assurance features arrived after his agency had already built its own with Claude Code analyzing call transcripts.

The temptation is to stuff everything into the prompt: all ten service descriptions, every policy, every price. The cost is triple. Expensive: pricing is flat (~4.5¢) under 3,500 prompt tokens and climbs steeply beyond. Slower: every turn re-processes the whole prompt, inflating latency. Dumber: instruction-following degrades as prompts bloat. The fix is architectural — big chunks of information go into an uploaded knowledge base, and the agent searches it on demand, mid-call, only when a caller's question requires it.

Retell's KB is a vectorized RAG system under the hood — BC5's retrieval concept in production — and it handles many pages comfortably (though it can't traverse folders or filesystems). Upload as markdown for best AI readability, and note the reuse economics: one knowledge base can serve every agent of the same business.

Structure first: single-prompt agents cover 99% of his builds; the conversational-flow (multi-path) builder earns its complexity only for methodical support trees. The welcome message is a control surface, not a nicety — AI-speaks-first with custom text sets the conversational frame. Then the tuned numbers: pause-before-speaking 0.4-0.6s; latency targeted at ~900-1500ms — the counterintuitive lesson being that TOO FAST breaks turn-taking, with the agent interrupting callers who are just actively listening ('yeah… uh-huh'); interruption sensitivity 0.75-0.85; a reminder message at 15 seconds of silence (twice), end-call around 90 seconds; and a sane max call duration — learned the hard way when a caller deliberately ran his agent's clock for a billed four hours.

Transcription always optimizes accuracy over speed (accuracy failures cascade through the whole loop), background call-center sounds add realism, and the edge-case switches — voicemail detection, iOS/Android call-screening response, IVR auto-hangup, keypad off — each close a real-world failure mode. On the LLM: Retell's menu suggests GPT-5.1/4.1, but tested practice says GPT-4.1 beats GPT-5.1 for voice, and his production default is Gemini 3 Flash — about half GPT-4.1's cost and faster; the price and latency visibly dropped on selection.

Retell's built-in voices are fine; ElevenLabs' library is better and vastly bigger, and the bridge is trivial — copy a voice ID, add it in Retell as a custom provider voice, with billing unified so there's no second subscription. Voice choice is branding: accent, age, warmth all signal who the business is before a word of content lands.

Multilingual support is nearly free but has one dependency: the voice itself must support the languages (the library tags each voice — look for the '+16'-style marker). Multi-select languages on the agent and it switches mid-conversation as the caller does. Voice cloning exists too — about 20 minutes of audio into ElevenLabs — with the ethical line drawn plainly: agents disclose they're AI in the welcome message; tricking callers is a GDPR and trust failure.

The five core sections, each with its craft. Role: who the agent is — Emma, receptionist at BrightSmile Dental. Context: who calls and why — patients booking, asking, sometimes in pain. Personality: traits WITH their reasons ('warm and reassuring, because callers often have dental anxiety') — the reason teaches the model when to apply the trait. Task: the objectives, deliberately split into ~5 small goals, because an agent given one big goal gets pushy about it. Conversation stages: the stage-by-stage script of how calls should flow — greeting, need discovery, data capture, booking, close — the section that makes you a conversational architect rather than a prompt writer.

Formatting is functional: markdown ## headers delimit sections, ** marks importance — structure the model actually parses. Around the core sit the supports: an inline mini knowledge base for facts too small to justify a KB upload; function-usage guidelines reinforcing when and how to call each function; important notes holding post-testing fixes (one question at a time, short responses, repeat details back, prioritize pain cases); and the load-bearing dynamic variable {{current_time_<IANA timezone>}} — without it, a caller's 'tomorrow' is unresolvable when the agent calls the calendar. Ask ChatGPT for your IANA timezone string.

Speech-to-text is the loop's weakest link, and these tactics are conversational error-correction. Names: ask the caller to spell it, repeat it back, and don't proceed until confirmed. Phone numbers: read them back digit by digit — never 'one twenty-six', which transcribes ambiguously. One question at a time, because two questions in one turn produce half-answers. And before any calendar check, force a single specific day — a whole week's availability payload is so large it confuses the agent into silence.

Two refinements complete the craft. Announce function calls naturally ('let me just check the schedule for that day') — it covers the retrieval latency AND is what a human receptionist would say anyway. And keep conditional instructions general — 'if multiple days, ask for one; if unsure, suggest the earliest' — because ten hyper-specific conditions bloat the prompt and confuse the model more than they help.

Retell ships cal.com functions natively: check-availability and book-appointment. Wiring them is pure configuration: a free cal.com account connected to Google Calendar; an API key (shown once — save it); an event type defining duration, location, availability windows and minimum notice; the event type ID lifted from the URL; then both values plus your timezone pasted into each function in Retell. From that moment the agent can consult a real calendar mid-call and write real bookings into it.

The subtle decision is the confirmation field: choosing phone vs email determines which caller details cal.com requires before accepting the booking. The demo chose phone — because the conversation stages already collect name and number — an example of the prompt design and the function config having to agree.

Post-call properties turn conversations into database rows. Four types — text, selector, boolean, number — extract fields like full_name, reason for calling, service requested, appointment_booked, call_outcome from every completed call. The mechanism behind them is the mental model that makes you good at this: imagine a blank ChatGPT canvas that receives ONLY the call transcript plus your description. Whatever the description doesn't specify, the extractor won't reliably do.

So each description gets written as a full prompt: the task, the constraints, the fallback ('return not provided' when the caller never said it), and worked examples of correct extractions. One lazy sentence ('the caller's name') yields inconsistent garbage; a real prompt yields data clean enough that the call-history filters and the n8n CRM pipeline downstream can rely on it.

Emma took a wisdom-tooth call end to end: empathy on the pain mention (the personality section firing), name spelled and confirmed, phone number read back digit by digit, one day forced for the availability check, slots offered, booking made. The choreography of the whole session, audible in ninety seconds.

The failures taught hardest. The first booking attempt failed because the collected number wasn't in international format — and the agent self-corrected, asking for the country code, exactly the resilience the function-usage guidelines exist to produce. Better still, a learner caught a congruence bug the trainer had missed: the prompt said 'you are Sarah' while the welcome message said 'my name is Emma' — the two configuration surfaces must agree. And the testing meta-lesson: test in-platform via audio or chat before wiring a phone number — the chat test exists 'so you don't look like a psychopath talking to your laptop in a café, having the same conversation over and over again.'

Retell fires three webhook events per call: call_started, call_ended, and call_analyzed — the last one carrying the full analyzed payload: summary, post-call properties, caller number, recording URL. The pipeline is n8n at its most archetypal: webhook trigger receives all three events, a filter node passes only call_analyzed (exact string match — character for character), and a Google Sheets append-row node maps payload fields to columns, one row per call. The date column needs one expression — UNIX timestamp to readable UK format — which Claude wrote on request.

The Q&A revealed the advanced inversion worth remembering: Retell's INBOUND webhook fires before the agent answers, which means you can look the caller up in your CRM by number and inject their history into the prompt — the agent greets a known customer knowing who they are and what happened last time. Same webhook machinery, opposite direction.

Platform landscape: VAPI is Retell's nearest peer, aimed at developers — Retell recommended for starters. GoHighLevel's native voice agents are 'rubbish' — no speech, call, transcription or post-call tuning — though exporting call notes INTO GoHighLevel works fine (his agency's CRM; tutorial on his YouTube). Deployment discipline for client work: the client's own Twilio account, with you buying the number and SIP trunk on their behalf — they own their telephony; you configure it.

Human handoff has two shapes: a live-transfer function that forwards the call to a phone number, or a human_callback post-call property that triggers an n8n notification for a follow-up. External RAG beyond Retell's KB is possible — custom function → n8n → vector store — at a latency cost. And pricing scales with use-case and systems complexity: the flagship reference build was an agent for a £25M/yr UK e-commerce company, wired into product, order and ticketing databases.