Day 6: Voice Agents That Talk Like Humans — the 5-Step Pipeline, Retell Settings That Matter, the 5-Section Prompt, cal.com Booking, and Call Logging

Per turn: (1) the user speaks; (2) a transcriber converts speech to text; (3) the text goes to an LLM pre-prompted with a role; (4) the LLM returns text; (5) a text-to-speech model voices it. 'It does that full 5 step process in roundabout 1 second' - up to ~1.5. Latency is defined as the time between the user finishing and the agent replying; Retell shows it per agent, decomposed into ElevenLabs, GPT, and transcription shares (970-1300 ms on the demo).

Cost follows the same decomposition: ~11.5 cents/min = 0.7 cents TTS + 4.5 cents LLM (GPT-4.1) + platform; above ~3,500 prompt tokens the LLM share climbs (he has seen ~+5 cents). Hence: knowledge bases as separate documents, not pasted into the prompt. Three agent types: inbound, outbound, and 'interactive learning assistant' (his term for agents embedded in software or a device).

Three archetypes: customer service (inbound), outbound sales (call the web-form lead), inbound receptionist (today's build: BrightSmile Dental, after-hours). The numbers: UK and US businesses 'collectively lose out on 100000000000 dollars' a year to missed calls; leads contacted within 5 minutes are '21 times more likely to convert' than within 30; Gartner is cited for 90% adoption by 2027. The pitch is cost: '24/7, 365', no wages, bonuses, sick days or holidays for the lower-level calls.

And the honesty rule: aim for human-like, not deceptive - the demo agent opens 'I'm an AI agent, but I promise I'm not gonna hack your phone' and still sounds smooth. An agent that only talks well is 'pretty useless' to a business; it needs connectors (bridges, APIs - 'it all means the same sort of thing') to a CRM, a database, a calendar.

Retell is pay-as-you-go (~$10 credit on signup). Single Prompt Agent is his recommendation even for production - '90-95%' of his agency's agents - because multi-prompt/deterministic flows get stuck when callers go off-script. Model: GPT-4.1, 'still like the industry standard' despite GPT-5, Claude Sonnet and Gemini Flash 3 being listed. Voice: import an ElevenLabs voice by copying its Voice ID into Add custom voice; set a Retell-native fallback voice in case ElevenLabs is down.

His switches: AI speaks first with a custom welcome ('speaking first grants control'); pause before speaking 0.4-0.6 s so the pickup does not feel instant; responsiveness below max so it does not interrupt mid-breath; interruption sensitivity 0.75-0.8 (~2-3 words to interrupt); backchanneling OFF (overdone 'mm, yeah' sounds more robotic); speech normalization ON (fixes 'dollar 15'); pronunciation tool never worked - spell names phonetically instead; denoising ON; transcription 'optimize for accuracy'; voicemail detection and IVR hang-up ON for outbound; end call on ~1.5 min silence; max duration ~8 min; ring ~30 s; ignore default dynamic variables; leave security defaults. Knowledge bases: upload Word/Markdown as per-agent vector stores rather than inflating the prompt. Batch calling: he avoids it (AI cold-calling rules in Europe/US) and triggers outbound from n8n/CRM. Numbers arrive via Twilio elastic SIP trunk -> termination URI + SIP credentials -> Retell.

Bad: 'You are a bot that answers questions' - zero context, zero identity. Better: 'You're a friendly assistant that helps customers track orders and gives clear instructions.' Poor prompts produce robotic replies, agents that 'get lost' off-script, and no personalization. The template he uses 'for every single agent': ROLE (name + business + purpose: 'You are Sarah, a friendly receptionist at BrightSmile Dental...'), CONTEXT (who is calling and the scenario), PERSONALITY (warm, calm, professional - tuned to the domain: empathetic for dental, no jokes for a law firm), TASK (several concrete sub-goals, not one line), CONVERSATION FLOW (the largest section - 'conversational architects'): Greeting -> Identify issue (empathy for pain) -> Collect details (full name spelled and repeated back before proceeding; phone likewise; one question at a time) -> Get availability (ask for ONE specific day; bracket notation for if/else; filler line 'let me just check the schedule' before the function call; offer 2-3 options) -> Book (only when caller and calendar agree; confirm) -> End (thank, recap, confirmation, anything else, warm close).

Extras: Knowledge Base inline only when tiny (here: services, hours, address, 30-min slots); Function Usage Guidelines because 'the LLM does NOT automatically know when to invoke a function'; Important Notes as an iteration log of small fixes found in testing ('say 15 dollars, not dollars 15'); and the current date/time dynamic variable stated at the top AND repeated at the end so calendar functions resolve dates. Formatting: # headers and **bold** so the model parses structure the way a reader parses font sizes.

cal.com is free. Connect Google Calendar so bookings are visible; create an Event Type (BrightSmile Appointment, 30 min, in-person address, availability 9-5 weekdays); copy the Event Type ID (7 digits in the URL); create an API key with no expiration and copy it immediately - 'this is the only time I'm gonna be able to see this API key.' In Retell, add Check Calendar Availability and Book Calendar Appointment functions, paste key + event ID into each, set the timezone (Europe/London), assign a phone number to the agent.

The live call to 'Emma': name spelled and confirmed, phone confirmed, three days offered -> agent asked for one, two windows presented, 12 pm confirmed directly, email mis-heard once and corrected on repeat-back, booking recapped - and the appointment appeared on the connected Google Calendar.

Post-call properties extract structured fields from each call: Text (reason for call), Selector (fixed choices - call outcome: no answer / callback / didn't book / booked), Boolean (did they book?), Number. The craft rule, repeated: 'write your descriptions as if they are a prompt in and of itself' - task framing, what to focus on, a fallback ('if no reason given, return not provided'), example input/output. 'A huge mistake that I see so many people making.' Selector outcomes double as deterministic CRM pipeline-stage routing.

Logging: paste your n8n webhook URL into Retell's webhook settings. Retell fires call_ended (thin) and call_analyzed (summary, transcript, custom analysis). Workflow: Webhook -> Filter (event equals 'call_analyzed') -> Google Sheets Append Row, document by ID (the string between /d/ and the next / in the sheet URL), map columns manually from the payload - properties live in the custom_analysis_data object near the bottom, call summary just above, recording_url by search, from_number for the caller (roles reverse on outbound), Unix timestamps converted with an n8n expression.

Part one, the Discovery Form: business info (name, what it does, typical customer); agent basics (inbound/outbound, main goal, who it talks to); conversation questions (what to ask, what to collect, flow); personality (tone, required phrases, forbidden topics - a dental agent never gives medical advice or diagnoses); FAQ and objection handling; differentiation; next steps after qualification; the definition of a successful call. Part two, a Claude/ChatGPT meta-prompt that ingests the filled form and outputs a complete agent prompt in the Role / Context / Personality / Task / Conversation Flow / Knowledge Base / Function Usage / Important Notes structure. Fill -> export markdown or Word -> upload with a one-line instruction. 'I've never given this out to anyone.'