When RevOps AI Agents Break: How to Improve Accuracy Over Time

Everyone’s shipping AI agents.

Almost no one talks about what happens when they’re wrong.

‍

The hallucination nobody catches. A prompt that slowly stops producing the output you expected. Latency that makes a workflow unusable. An edge case that works perfectly in testing and falls apart the minute real users get involved.

Getting an agent live is one challenge. Making it more accurate, reliable, and effective over time is another.

‍

In this session, Alex Avila (Nooks) and Gerard Martelly (Vapi) will get into the less glamorous, but much more important, work that happens after launch.

They’ll share examples from agents and AI workflows they’ve personally built, broken, measured, and improved, including what went wrong and how those failures changed the systems around them.

‍

At the center of the conversation is the learning loop: how you continuously evaluate agent output, identify where things are breaking, and feed those lessons back into prompts, architecture, guardrails, and workflows.

Because better agent performance isn't just about writing a smarter prompt.

It means deciding what “good enough” accuracy actually looks like. Building fallback logic for when things go wrong. Knowing when a human needs to stay in the loop. Measuring drift instead of discovering it from an angry user. And accepting that tuning isn't something you finish once and move on from.

‍

🔑 Key Takeaways

• Why reliable agent output depends on system architecture, fallback logic, and output parsing, not just better prompting
• How to decide what level of accuracy you're willing to trade for speed
• A practical framework for deciding when an agent can operate autonomously vs. when a human should stay in the loop
• What to measure across accuracy, latency, adoption, and drift
• How to turn failures and user feedback into a continuous learning loop
• Lessons from AI agents that actually broke in production, and what Alex and Gerard changed as a result

‍

📣 Speakers

• Alex Avila, Founding Solutions Engineer, Nooks
• Gerard Martelly, Senior Manager, RevOps, Vapi

‍

‍

Not a member of the RevOps Co-op yet? Join here to connect with 20,000+ pros who love revenue operations!