TurnBench
A multi-domain benchmark for evaluating conversational turn-taking. We hand-annotate end-of-turn and interruption events in dual-channel human conversations, and measure how accurately and how quickly models detect them.
Leaderboard
The ideal model is capable of high recall while maintaining a low false-positive rate and low latency. The following models are evaluated against our held-out test set and ranked by recall.
#Model Recall↑FPR↓Latency↓
1 Voice Activity Projection 0.845 0.055 368 ms
2 ESPnet Turntaking 0.826 0.078 862 ms
3 WavLM Large Anchor 0.800 0.054 1076 ms
4 Mimi Endpointer 0.782 0.078 645 ms
5 Kyutai Semantic VAD 0.773 0.059 1007 ms
6 Smart Turn v3 0.752 0.047 1017 ms
7 ESPnet Turntaking Perchannel 0.711 0.081 730 ms
8 Gemini 3.1 Live 0.657 0.022 1234 ms
9 WavLM Large Causal 0.408 0.054 683 ms
10 WavLM Base Causal 0.403 0.061 701 ms
11 OpenAI Realtime (Semantic VAD)0.303 0.018 793 ms
12 Moshi 0.233 0.044 702 ms
13 OpenAI Realtime (Server VAD)0.955 0.525 282 ms
14 RMS Energy VAD 0.718 0.632-117 ms
Recall:share of true events detected (higher is better).
FPR:false-positive rate on test (lower is better); above 0.15 does not qualify.
Latency:median delay after the event, p50 in ms (lower is better).
Click any model to open its full report: dev-set scores per conversation, with its predictions overlaid in the viewer.
Dataset
TurnBench evaluates models on a 30-hour corpus of studio-recorded, dual-channel dyadic speech. The corpus is provided byMundo AIand features 154 dialogues, 106 actors, and is balanced across 6 conversation types1. Every dialogue is labeled by three independent annotators (Fleiss's κ = 0.78) where the ground truth label is derived from 2/3 consensus.
Additionally, we include a training dataset, otoSpeech, hand-labeled under the same protocol as the evaluation set. All three splits are available on Hugging Face.
train 104 hotoSpeech
collected by Oto
audio + annotations
dev 7.3 hturn-benchmark-dev
collected by Mundo AI
audio + annotations
test 22.9 hturn-benchmark-test
collected by Mundo AI
audio only
We created aninteractive data viewerfor the public dev set. It is an audio player that displays each conversation with the annotated labels drawn on the waveform. You can compare these labels with a published baseline's predictions or submit your own.
Evaluation
Per conversation, TurnBench requires models to output a discrete list of times2for the following events:
End-of-turn When a speaker has finished speaking and the floor is open for the other speaker to take.FP:declaring end-of-turn during a mid-turn pause.
Interruption When a speaker takes the floor while the other is still talking.FP:declaring an interruption during a backchannel or because of noise, channel bleed, or echo.
We open source the scorer, submission format, and baseline implementations on GitHub.
Results
We find end-of-turn recall is invariant to conversation type1, while interruption false positives are not, tending to concentrate in casual conversations with high backchannel density. As expected, we find that acoustic detectors fire over-eagerly on silences, saturating recall but blowing past the false-positive budget. Semantic systems on the other hand are more selective but slower, staying in budget by waiting for more evidence.
VAP is the strongest model on both the EOT and INT tracks, but**no system is simultaneously fast, high-recall, and low on false positives.**For reference, humans in the TurnBench corpus begin turn transfers a median 151 ms _before_ the turn ends3. We hope TurnBench is a valuable resource for the community to create more effective spoken interaction systems.
Submissions
We invite the community to improve upon these results, and provide thedev-set scoreras a resource. To create a submission, please emailturnbench@sesame.comwith:
1. 1.Model display name for the leaderboard 2. 2.Organization name 3. 3.`predictions-dev.json` and`predictions-test.json`
Please refer to the full submission instructions here:SUBMISSION_FORMAT.md.