What I learned building a benchmark for professional voice agents

I kept coming back to the same question

Over the last year, I spent a lot of time looking at full-duplex voice systems: models that can listen while speaking, handle interruptions, overlap naturally with a user, and call tools during a live conversation. The demos are impressive. In many interactions, the model feels much closer to a person than the turn-based voice assistants we were used to.

But I kept feeling that we were evaluating the easiest part of the problem. We had good ways to ask whether a system could yield when interrupted, respond quickly, or produce a sensible answer. Those properties matter, but they do not tell me whether I would trust the system with actual work.

The example I kept using was deliberately mundane: suppose I ask a voice agent to prepare a leave request. It checks my available dates, starts filling the form, and while it is talking I say, “Actually, change the dates — and don’t submit yet.” A few seconds later I approve the final version. If the agent sounds fluent throughout but leaves the old dates in the form, or submits before approval, I do not consider that task successful.

That became the core question behind APEX-Voice: can a full-duplex voice agent translate conversational competence into dependable professional work?

My first design choice: evaluate workflows, not isolated turns

Once I framed the problem this way, single-turn question answering stopped being a useful abstraction. Professional tasks have persistent state. Information arrives gradually. The user can revise earlier decisions. Tools change the environment. Some actions require approval. And the final output usually exists outside the conversation as a concrete artifact: a form, ticket, memo, schedule, plan, record, or negotiation outcome.

So I designed each benchmark item as an executable workflow rather than as a prompt-response pair. Every workflow runs inside what I call the Voice Workbench: an environment that can contain task-specific knowledge, typed tools, a versioned work artifact, authorization constraints, and a user policy that reacts to the agent as the interaction unfolds.

I ended up with 120 workflows spanning ten work archetypes. I intentionally varied more than just the task surface: some workflows require almost no external knowledge, others require policy or document lookup; some are nearly prepare-only, others require confirmation or explicit approval; some users are cooperative, while others correct themselves, become ambiguous, or interrupt at inconvenient moments.

The hardest part was making the benchmark interactive and still reproducible

This was the part I worried about most during the design. If I simply scripted one fixed audio file for every task, the benchmark would be reproducible, but it would not really test an interactive voice agent. The user would never respond to what the agent actually did.

At the other extreme, I could put another language model on the user side and let it improvise freely. That would feel realistic, but now two evaluations of the same agent might expose it to materially different users. Worse, two different agents might be tested on different dialogue trajectories, making comparisons much harder to interpret.

I settled on a middle ground. I generate and validate the user-side speech realizations offline, but a runtime policy decides which realization to play based on the evolving workflow state. In other words, the interaction can branch, but the speech itself comes from a controlled bank.

That design was important to me because I wanted repeated runs to mean something. When I later measured Pass@3 and Reliable@3, I wanted differences to come from the agent’s behavior rather than from an unconstrained simulator inventing a new task on every run.

I also stopped treating the transcript as the ground truth

Another choice that changed how I thought about evaluation was to record the whole execution as a causal timeline. The conversation transcript is only one view of what happened. For these tasks I also need to know when the user changed a value, when a tool was called, which artifact version existed at that point, whether approval had been granted, and whether the agent acted on stale state.

This led to the semantic-act observer and synchronized event stream in the workbench. A correction such as “change the dates” is represented as a state-changing event, not merely a sentence in a transcript. The environment can then check whether the old value was superseded, whether dependent state changed, and whether a later action used the authorized version.

Then I ran the agents — and the first result was worse than I expected

I evaluated five frontier real-time voice agents on the same workflow suite. My initial expectation was that the strongest systems would still fail in difficult cases, but that end-to-end success would track their apparent conversational quality reasonably closely.

It did not. None of the five systems exceeded 25% Pass@1. The strongest Pass@1 result was 23.6%, and the strongest Reliable@3 — succeeding on the same workflow in all three independent runs — was only 10.8%.

The gap between Pass@3 and Reliable@3 became one of the most interesting results for me. A model can sometimes find a successful trajectory without being able to reproduce it. Grok-Voice-Think-2.0, for example, succeeds at least once across three runs on 41.7% of workflows, but succeeds in all three runs on only 5.0%.

That changed the way I interpreted “capability.” For an assistant doing professional work, occasional success and dependable success are very different things.

The number that made me re-check the evaluation

GPT-realtime-2.1 gets 91.4% Artifact Field Accuracy but only 23.6% Pass@1. When I first saw that gap, I went back through the evaluation because it looks counterintuitive: how can a system be right on more than nine out of ten fields and still fail most workflows?

The answer is that professional tasks are conjunctive. A nearly correct artifact can still be unusable. One stale date, one missing required field, one unauthorized action, or one uncompleted tool call can invalidate the task.

That is why I score workflow success through four gates rather than averaging local correctness:

The workflow only counts as successful if all four are satisfied. Across the voice agents, Artifact Validity was the lowest gate and topped out at 35.3%. For me, this was a strong signal that current systems are much better at getting pieces of a workflow right than at keeping the whole workflow coherent.

The most useful surprise: interruptions were not the main problem

Going into the experiments, I thought a major failure mode would be basic full-duplex control: the user interrupts and the model keeps speaking, fails to hear the correction, or takes too long to yield.

That hypothesis was only partly right. Four of the five frontier systems yielded on 99.8–100% of barge-ins, with median stopping delays between 20 and 156 ms. So for those systems, the mechanics of yielding were already quite good.

The failure happened one step later.

When the user changed a value mid-speech, field accuracy on those corrected fields was 20–37 percentage points lower than on other fields. And 71–89% of final Artifact Validity failures were linked to corrections.

This became one of my favorite takeaways from the paper because it separates two abilities that are easy to conflate: detecting that the user interrupted and updating the internal task state correctly after the interruption. Current agents can often do the first while still struggling badly with the second.

I added text agents because I wanted to know whether the benchmark itself was simply too hard

At that point I had another concern: perhaps these workflows were difficult for any agent, and the poor voice results had little to do with the speech interface. So I ran frontier text agents in the same environment as a control.

The text agents were substantially stronger, reaching 54.3–62.0% Pass@1 and roughly 95–98% field accuracy, but they still did not solve the benchmark. I found this result useful because it separates two effects.

First, long-horizon professional execution is genuinely hard even without audio. Second, putting that workflow inside a live, full-duplex spoken interaction introduces another layer of fragility — especially when task state changes while the agent is already speaking or acting.

What I came away believing

Before doing this work, I tended to think of professional voice-agent quality as a stack of capabilities: good speech understanding, low latency, strong reasoning, good tool use, and natural turn-taking. I still think all of those matter, but the experiments made me much more focused on the glue between them.

A useful professional agent needs a continuously updated working state that connects what the user said, what the agent believes, what has already been written into the artifact, which tools have executed, and what the user has actually authorized. When any of those representations drift apart, a conversation can remain perfectly fluent while the task quietly becomes wrong.

That is why I think the next generation of voice-agent research needs to care much more about state synchronization: not just hearing a correction, but invalidating superseded information; not just calling a tool, but making sure its result is reflected in the work product; not just asking for approval, but binding that approval to the exact action and artifact state the user saw.

APEX-Voice is my attempt to make those failures measurable. The benchmark is not meant to argue that current systems are bad at conversation — many of them are strikingly good at it. The point is that sounding competent and completing professional work reliably are still very different milestones.

Why I am excited about this direction

I like this problem because progress is easy to imagine and hard to fake. If a system really gets better, it should leave behind a better artifact, preserve the right state after corrections, obey authorization boundaries, and reproduce that success across repeated runs. Those are outcomes we can inspect directly.

My hope is that APEX-Voice pushes evaluation a little further away from “does this interaction feel good?” and toward “did the system actually complete the delegated work correctly?” I think we need both. Natural interaction is what makes voice agents useful; dependable execution is what will make them trustworthy for real professional workflows.