A voice agent can have a capable model and a good voice, yet still frustrate people. Often, it comes from small engineering choices that are easy to skip past while you're building, and that you only notice when a real caller is on the line.
Our team has seen these choices play out in production voice-agent work. Two of our engineers, Nazar Vinnichuk and Ihor Isakov, pulled together the lessons that made a real difference to agent quality, along with a few things that turned out to be distractions.
We first shared these lessons in a workshop with Sigma Software. Nazar and Ihor walked through the things our team has learned while putting voice agents to work.
Key Takeaways
- Give the model choices it can use. Create Support Ticket is more useful to an agent than Make an HTTP Request.
- Test retrieval on its own. First confirm that the answer is in the agent’s context; then tune how the agent responds.
- Build for someone listening. When the agent needs a moment, tell the caller. When it can’t help, pass the call to someone who can.
Lesson 1: Replace Generic API Calls With Purpose-Built Tools
We started with a single and highly flexible tool: Make an HTTP Request.It seemed ideal, but it forced the agent to waste context figuring out endpoints, headers, auth, and request bodies just to handle routine tasks like creating a support ticket.
We replaced generic requests with direct business actions: Create Support Ticket, Search Product, and Check Order Status. Now, the agent picks an outcome rather than constructing an API call, keeping its full focus on the conversation.
We applied the same logic to outputs and workflows:
- return direct identifiers: if a tool creates a ticket, return the ID immediately. Don't force the agent to query the system again for data it just created.
- combine paired actions: if two operations always happen together, merge them into one. One predictable tool is far easier for the model to reason about than two sequential steps it might forget.
|
Generic tool |
Business-specific tool |
|
Make an HTTP Request |
Create Support Ticket |
|
Model reasons about endpoints, headers, auth, media type |
Model picks the action that matches intent |
|
Returns a raw response, often needs a follow-up call |
Returns the ticket ID in the same step |
Lesson 2: Check Retrieval Before Rewriting the Prompt
When an agent gives a bad answer, the prompt is an easy target. It’s right there. You can add a rule, move a sentence around, underline something in all caps, and feel like progress has been made. But the prompt may be innocent.
First, inspect the retrieval results without the model in the loop:
- did the search actually pull the right documents?
- do those documents contain the answer?
- is the retrieved information accurate and up to date?
If retrieval fails, fix the source content, chunking, metadata, query logic, or ranking. A better prompt can’t repair retrieval that missed the answer.
Lesson 3: Don’t Make the Agent Read the Formatting
Imagine calling support and hearing: “Here are three steps. First, open bracket… then asterisks… for more information, visit…” Not ideal.This happens when you reuse text prompts for voice. Visual formatting like bold text, bullet points, and URLs makes sense on a screen, but may sound absurd over a phone line.
Voice prompts require distinct conversation design:
- specify speech style: direct the model to speak in plain prose, use short sentences, and ask one question at a time.
- tailor by use case: a support agent needs to guide someone through a process, while a receptionist just needs to transfer the call. Customize the voice guidelines for each specific role.
Good tools and a well-written prompt only get an agent so far. Respeecher gives the agent a voice that keeps up with the rest of the experience.
See real-time TTS →
Lesson 4: Plan for the Bits Between the Answers
If an agent needs a few seconds to search a knowledge base or wait for another system, say so. “Let me check that for you” tells the caller the line is still alive and the agent hasn’t wandered off just yet.The same goes for silence on the other end. Follow up once or twice, then end the call cleanly. Nobody needs an endless standoff between a person making dinner and an agent asking, “Are you still there?”
Support calls bring a different problem. Ask, “Which department do you need?” and many callers will tell you the whole story. If a person will need to take over anyway, the agent should listen long enough to understand the issue, then transfer the call before the caller has to explain it all again.
Even background sound can change the feel of a phone conversation. In some tests, a faint office bed made the call feel less sterile. It’s not a universal fix, but it is a reminder that conversation design includes what happens around the words.
What This Adds Up To
The point is not to make a voice agent do everything. It is to make each call feel straightforward: answer what it can, be clear when it cannot, and get a person involved when that is the better option.
If you’re building a voice agent and want to talk through the voice, latency, or production side of it, our team is here. Tell us about your project.
