- decision
Gate every voice-agent release behind an automated eval harness
- assumption: the belief that made me confident
At production scale, manual QA cannot be the ship gate. Regression on prompts and flows has to be automated or quality decays silently
- confidence at the time
- 95% · very high
Indocord
A platform for building GenAI voice agents for BFSI and e-commerce support, scaled to millions of calls a month.
Founding team, Associate PM at Reverie (Reliance Jio Platforms)Retrospective
the call I made
what it looks like
millions
calls handled monthly
platform scale, context for the artifact, not the claim
BFSI
sector focus
plus e-commerce support
what I did
I was a founding team member on Indocord, Reverie's platform for building GenAI voice agents for BFSI and e-commerce support. The platform scaled to handle millions of calls a month across enterprise customers, but the number that matters here is platform-scale context. The thing I built is smaller and more specific.
What I actually built
I built and owned the eval gate: automated regression testing on prompts and flows, plus quality checks required before anything shipped to production. When a platform is taking live calls for banks, "we tested it manually and it seemed fine" is not a release process. The harness became the ship/no-ship mechanism. The record above is the reasoning as I'd have written it into Lore at the time.
What the role taught me about real-time AI
Voice is the least forgiving surface for LLMs: the latency/cost trade-offs of real-time conversation forced an LLM-NLU hybrid approach rather than the pure-LLM architecture everyone defaults to today. I also worked on speech-to-speech, tool calling, and early Model Context Protocol integration, and ran customer interviews to pressure-test flows before scale-up.
what happened
The gate became the platform's ship/no-ship mechanism as it scaled to millions of calls a month across enterprise customers
▸ the full record: evidence, alternatives, risks, what I traded away
- risks I named
- The eval gate could become a bottleneck the team routes around under delivery pressure
- how I'd know it worked
Nothing ships to production without clearing the harness; regressions caught before customers see them
what I knowingly traded away
- LLM-NLU hybrid over pure LLM: real-time voice lives and dies on the latency/cost curve, and the hybrid held it
- Regression evals over manual QA: slower to build, but the only thing that scales with call volume
built with: Voice agents · LLM-NLU hybrid · Eval / regression harness · Speech-to-speech, tool calling, MCP exploration