Voicecore: Rust media backend, road to infinite scalability with no idle GPUs

A voice-agent server spends most of its hardware budget on the GPU, and the next cards are several times faster. A faster card pays off only when the rest of the server hands it more conversations at the same time.

That rest includes the media path: the code that takes the audio of every call in frames of 20 ms, levels it, listens for speech and mixes the agent’s voice back in. On our previous path it ran in Python, 50 times a second for every call, through one interpreter. At about 96 simultaneous conversations the processor of one bot process was used up, at the same load today’s GPU carries. A card twice as fast would have had nothing more to serve.

Voicecore is the media core we wrote in Rust to remove that limit. This post describes what it does, what one frame of audio costs on each path and how many conversations one bot process carries with resources unbound, meaning the GPU and the models do not limit it. With Voicecore the answer is 224.

In short:

• One frame costs less. On the same audio and one core, Voicecore uses 2.4 times less processor per 20 ms frame (32 µs against 79 µs) and holds Python’s interpreter lock 21 times less (1.4 µs against 30 µs).
• One process carries 2.3 times more. On 8 physical cores the old path used up the processor of a bot process at about 96 conversations. Voicecore carries 224, and a straight-line extrapolation puts its next ceiling near 245.
• The processor stays free. At 224 conversations the process uses 2.6 cores, about a third of the 7.0 the old path used at 96.
• Reply time stays flat. The reply-time index holds near 1.0 from 96 to 224 conversations. On the old path it passes 2.0 at 192.

We measured everything here on our test stand on 9 to 11 November 2026.

A frame every 20 ms

A call reaches the server as a stream of RTP packets, and the server works on it in frames: 20 ms of audio each, 160 samples at 8 kHz or 320 at 16 kHz. Twenty milliseconds is the usual packet size on a phone line. It is short enough that a reply loses nothing while a frame fills, and long enough that the work per frame stays affordable. Each call produces 50 frames a second, so a server with 200 calls handles 10,000 frames a second.

The frame is the unit the whole media path counts in. Working frame by frame keeps the delay low and gives every operation a fixed deadline: a frame that arrives late is a click or a gap the caller hears. Each frame goes through four operations:

1. Receive. The SIP stack takes the RTP packets, puts them in order in a jitter buffer and cuts the stream into frames.
2. Gain control. The server brings the voice to a steady level.
3. Voice-activity detection (VAD). A neural model listens to windows cut from the frames and decides whether the caller is speaking, so the agent knows when speech starts and stops.
4. Mixing. The mixer lays the agent’s voice, with the background under it, over the outgoing frame, and the SIP stack sends it back.

The speech models (recognition, the language model and speech synthesis) run on the GPU. The frame work above stays on the processor.

On the old path all four operations ran in Python. For every frame the process took the audio from the SIP stack through a Python wrapper while holding the interpreter lock, computed the level and gain with numpy, ran the VAD model, updated the speaking state and mixed a frame back. That is 10,000 events a second at 200 calls, all through one interpreter, one lock and one event loop, which also streams language-model tokens, speech-synthesis chunks and recognition requests.

Today’s GPU is busy 91–95% of the time from 80 to 96 conversations. To make a faster card useful, the processor side has to carry more than that.

What Voicecore is

Voicecore is the media core of the voice engine, written in Rust. It installs as a regular Python package, keeps an established SIP stack written in C for SIP signaling and RTP, and runs its own threads around it:

The border with Python is built so the interpreter lock is taken once per tick, not once per frame. The core collects the events of all calls, wakes Python’s event loop once and hands over the whole batch in one call. For a call on the Python side the result arrives finished: here are 20 ms of audio after gain control, the speech probability is 0.93, speech has started.

We introduced it in phases, with the old path always beside it:

• Phase 1, transport: SIP, RTP, frames, recording.
• Phase 2, signal: gain control, VAD and the mixer moved into the core.
• Phase 3, planned: end-of-turn decisions in the core.

Earlier changes to the media path, one shared 20 ms clock for all calls and cheaper frame handover, are on the Technologies page. The Roadmap lists what comes next.

How we tested it

An error in a media core is something a caller hears, so the rules were strict.

• Performance budgets are part of the specification. Missing a budget is a defect. In CI every pull request is timed against the baseline, and the build fails when the lower bound of the slowdown’s confidence interval is above the threshold.
• Bit-for-bit parity. A corpus of 277 calls (12.2 hours, 2.2 million frames) went through the old Python path and through the core. The audio after gain control matches to the last bit, the VAD probabilities are bit-identical, and all 3,451 speech starts land on the same frames.
• Fuzzing of the SIP INVITE parser, the SDP parser and the packet demultiplexer.
• A chaos lab: RTP drops, a re-INVITE every 2 seconds, Replaces, a password change under 10 calls, an engine stop under 50 calls, a 3-second freeze of the event loop. The core passes 31 of 31.
• Size: about 78,000 lines of Rust, 1,319 tests, 48 architecture decision records and three code audits.

The core’s own timing tests ran on a modest virtual machine (4 vCPU at 2.1 GHz), so a production server is faster:

At 1,000 calls the cost per call stays the same: the work grows in step with the number of calls.

The cost of one frame

To compare the two paths on equal terms we fed both the same audio, the left channel of the corpus at 16 kHz, on one physical core of the server:

• Python ran what the old path ran: its own gain-control and VAD code on each frame, the VAD in a thread pool as in production.
• Voicecore ran virtual calls with gain control and VAD. Python only collected and acknowledged the events.

Each run had 100 or 200 simultaneous calls of 20 seconds, with four runs in all.

• Python, old path
• Voicecore

One physical core, 100 and 200 simultaneous calls of 20 seconds, the same recorded audio, mean of four runs. In production the audio reaches Voicecore over RTP without passing through Python; the lock time here excludes feeding the recording into a virtual call.

Show the numbers

Cost of one 20 ms frame of one call, microseconds.

Measure	Python, old path	Voicecore	Ratio
Whole process, processor time	79.4	32.4	2.4×
Main thread, interpreter lock	30.4	1.4	21×
Voice-activity detection	48.9	27.4	1.8×
Gain control, level, window cutting	(main thread)	1.6	—

A second of conversation is 50 frames. In Python that is 4 ms of processor and 1.5 ms of lock time per call per second. One core carries the signal work of about 250 calls, and the lock alone is used up by about 650 calls, before recognition, the language model, synthesis or any dialogue logic. In Voicecore the figures are 1.6 ms of processor and 0.07–0.15 ms of lock: about 600 calls per core, and the signal work no longer competes for the lock.

Most of the cost of a frame is the neural VAD, 27 of the 32 µs. The Rust code around it takes about 1.6 µs. The gain on the VAD itself comes from batching: one model run covers windows from many calls.

How many conversations one process carries

The cost of a frame is only part of the picture. The question that mattered is how many conversations one bot process carries when the GPU stops being the limit. To measure that we ran the stand with resources unbound: the language model, speech synthesis and recognition answer at once and keep the shape of the real protocols, so the only thing that loads the server is the bot.

Load comes from pairs of an agent and a simulated caller, with both ends served by the same process. One pair is a conversation, and it costs the process two sessions. A step ramps up at 2 conversations a second, settles for 90 seconds and runs for a measurement window of 4 to 8 minutes. We draw a step that fails our checks (steady state, matching of turns, saturated limiters) hollow. On the old path the saturated queue of the VAD pool is itself the result, so its steps from 128 conversations on are hollow.

These are ceilings of the bot process. The capacity of a Server Unit stays rated at 100 simultaneous calls, set by the models on the GPU.

Reply time here is an index: 1.00 is Voicecore at 96 conversations, in paired runs of the same build. A lower index is a faster reply.

• Old media path
• Voicecore
• Extrapolated, not measured
• Hollow: the step did not pass the stand’s checks

Each dashed end continues the last three measured steps of its path in a straight line. At 96 conversations the two paths ran in pairs.

Show the numbers

Reply-time index by load, resources unbound (1.00 = Voicecore at 96 conversations). † The step did not pass the stand’s checks. ≈ Extrapolated, not measured.

Conversations	Old media path	Voicecore
96	1.08	1.00
128	1.39 †	—
160	1.78 †	1.00
192	2.13 †	1.00
224	≈ 2.50	1.02
245	—	≈ 1.02

• On the old path the processor limit arrives at 96 to 128 conversations. At 96, in paired runs, Voicecore already answers about 8% faster. After that, reply time on the old path rises through one stage, the wait for the end of the caller’s turn, which grows by half between 128 and 192 conversations. Voicecore’s wait stays within 2% from 96 to 224.
• Voicecore stays flat to 224 conversations, more than twice the old path’s 96, with reply time within 2% of its value at 96.

The number of replies tells the same story.

• Linear growth from 96 conversations
• Old media path
• Voicecore
• Extrapolated, not measured
• Hollow: the step did not pass the stand’s checks

Counted over each step’s measurement window and converted to a minute. The dotted line is the 96-conversation rate of Voicecore multiplied by the load.

Show the numbers

Replies per minute by load, resources unbound. † The step did not pass the stand’s checks. ≈ Extrapolated, not measured.

Conversations	Old media path	Voicecore
96	283.50	311.70
128	271.10 †	—
160	252.60 †	517.10
192	237.20 †	621.60
224	—	726.00
245	—	≈ 794.55

On the old path the replies per minute fall as the load rises, from 283 at 96 conversations to 237 at 192: the process does less work as it is pushed harder. With Voicecore they grow along the line of proportional growth, to 726 a minute at 224 conversations.

Where the processor went

Why does the old path use up its processor so early? We read the processor time of each thread of the bot process at 128 conversations:

Of the 6.3–6.7 cores the process uses on the old path, 4.9–5.4 belong to 15 threads without a name, each at 33–36% of a core. They are the thread pool of the math library under numpy, which starts on import and is sized to the available cores minus one. Some operation in the hot path wakes the pool from time to time, and afterwards its threads spin, waiting for the next job. Spinning threads take cores, and their hyper-threading neighbors, from the main thread, the signal clock and the VAD threads.

Behind the processor stands the interpreter lock. The VAD pool, whose four threads are each busy about 5% of the time, queues for the lock that the main thread and the SIP stack’s callbacks also want. Raising the VAD pool from 4 to 8 threads gave the same result, because the lock sets the pace, not the thread count.

Voicecore keeps that pool to one thread and moves the signal work out of Python. At 160 conversations the process then uses 1.8 cores where the old path used 6.5, and at 224 conversations 2.6, about a third of the 7.0 the old path used at 96.

• Idle thread pool of the math library
• Conference-bridge clock of the SIP stack
• Main thread (all dialogue logic)
• Voice-activity detection
• End-of-turn detector
• RTP, signal clock, the rest

At 96 conversations, the last step the old path passes, the process uses 7.0 cores.

Show the numbers

Processor time by group of threads, in cores.

Group of threads	Old path, 160	Voicecore, 160	Voicecore, 224
Idle thread pool of the math library	5.15	0.00	0.00
Conference-bridge clock of the SIP stack	0.42	0.61	0.88
Main thread (all dialogue logic)	0.51	0.49	0.74
Voice-activity detection	0.20	0.29	0.41
End-of-turn detector	0.20	0.28	0.40
RTP, signal clock, the rest	0.03	0.10	0.16
Total	6.50	1.77	2.59

What stays is the work we expected to see. At 224 conversations the bridge clock takes 0.88 cores (34%), the main thread 0.74 (29%), the VAD threads in the core 0.41 (16%), the end-of-turn detector 0.40 (15%), and RTP, the core’s signal clock and everything else 0.16 (6%).

Two more limits we removed

Garbage collection. Every 45 to 50 seconds the event loop froze for 200 to 870 ms. A watchdog that captured the main thread’s stack during a freeze, and a histogram of collector pauses, pointed to the garbage collector walking hundreds of thousands of long-lived objects created at startup. The bot now moves those objects out of the collector’s reach once startup is done. Over one ladder of load steps there were 3,559 collections, all shorter than 100 ms (1.7 ms on average), and not one freeze.

The open-files limit. At 192 conversations reply time rose several times over from the first minute, with the processor free. The log held the cause: the container allowed 1,024 open files, and a session holds about 2.5 descriptors (sockets to the models, cache files), so 384 sessions ran into it. The bot now raises the limit to 65,536 at startup. The old limit would have stopped any configuration above about 400 sessions.

Where the next ceiling is

Three resources decide how many conversations one process carries: what the GPU carries, what the processor allows, and what the main thread allows. The smallest of the three sets the ceiling.

The two remaining single threads grow in a straight line, so we can extend them:

• Conference-bridge clock of the SIP stack
• Main thread (dialogue logic)
• Extrapolated, not measured

The 80% line is our judgment of where the event loop starts to fall behind, not a measurement.

Show the numbers

Share of one core by thread, Voicecore. ≈ Extrapolated, not measured.

Conversations	Bridge clock	Main thread
160	60.9	48.7
192	74.8	61.3
224	88.1	74.3
238	—	≈ 80.0
252	≈ 100.0	—
288	—	≈ 100.0

The bridge clock gains 0.43 points per conversation and reaches 100% near 252 conversations. The main thread gains 0.4 and reaches our practical limit of about 80% near 238. These are extrapolations through three measured steps. A step at 256 conversations would test them.

Now lay a faster GPU over this. Today’s GPU carries about 96 conversations at 90–95% busy, which is close to the 100 calls a Server Unit is rated for. Take a hypothetical GPU k times faster on our load. This sets a scale and says nothing about any particular model.

• What the GPU could carry
• Voicecore
• Old media path
• Extrapolated, not measured

Today’s GPU carries about 96 conversations at 90–95% load. A faster GPU is a hypothetical: the chart sets a scale, not a promise about any particular model. The 245 for Voicecore is the extrapolation of the previous chart.

Show the numbers

Conversations per server by GPU speed, a model. ≈ The ceiling of 245 is extrapolated, not measured.

GPU speed	×1	×1.5	×2	×3
GPU could carry	96	144	192	288
Old media path	96	96	96	96
Voicecore	96	144	192	≈ 245

With the old path the processor limit sits at the load today’s GPU already carries, so a faster GPU would add no conversation. With Voicecore a process on 8 cores has 2.3 times the headroom of today’s GPU as measured and about 2.5 times by extrapolation: it would keep a GPU two and a half times as fast busy.

The next steps follow the two threads:

• The bridge clock of the SIP stack arrives first, at 88% of a core at 224 conversations. It is C code that walks the ports of all calls every 20 ms. Planned: a port clock of the core’s own, without the bridge, or several bridges. The core’s own clock is busy about 7% at 224 conversations.
• The main Python thread follows, at 74% at 224 conversations. The next step there is several bot processes per server, and the core allows it: each process has its own engine.

Stability

• Duration2 hours50 simultaneous conversations
• Process memory+1% an hourabout 6.4 GB, no steps
• Reply time, last 10 minutes against first 10−1%no drift
• Core faults0
• Dropped frames0
• Playback underruns0

Show the numbers

Two-hour run at 50 conversations.

Measure	Result
Process memory growth	about +1% per hour (6.42 to 6.54 GB over 1.9 hours)
Reply time, last 10 minutes against the first 10	0.99
Core faults	0
Dropped frames	0
Playback underruns	0

• Two hours under load, 50 conversations: memory grew about 1% an hour with no steps, reply time in the last ten minutes was 1% lower than in the first, and the core reported no fault, no dropped frame and no playback underrun.
• The SIP traces of the core and the old path agree, and in the audio lab the latency of the two paths matches within ±0.4 ms.

What comes next

Voicecore takes the processor out of the equation, so a faster GPU can shorten the model part of a reply. The other large part is the wait for the end of the caller’s turn, the rule that decides the caller has finished. Moving that decision into the core is phase 3, planned. Knowing when the caller has finished describes the model behind it, and Listening while talking describes early start, which prepares the reply inside the pause and cut reply time by about 17% at 64 and 80 conversations in our load test.

What we took from it

1. Rewriting the hot path in Rust is half of the work, and the border with Python is the other half. The core hands events over in batches, one wake-up per tick, so the interpreter lock is taken once for all calls.
2. Look at the processor by thread, not by process. Most of the old path’s processor went to a pool nobody had thought about.
3. The bottleneck moves. The old path ran into the interpreter lock and an idle thread pool, and Voicecore runs into the single-threaded bridge clock. Each time a measurement found it.
4. Measure the ceiling without the GPU. A run with resources unbound shows what the next generation of cards will meet, before anyone buys one.

How we measured

• Server. A 16-core, 32-thread machine with one GPU. The bot process had 8 physical cores (16 threads); the model side and the load bridge ran on other cores.
• Resources unbound. The language model streams answers with 6% unique replies, speech synthesis gives first audio after 80 ms and recognition speaks the real protocol, so no wait for the GPU enters the result.
• Paths compared. The old path with its default settings against Voicecore with the math library on one thread. At 96 conversations the two ran in pairs, on the same build.
• Load profile. A recruiting agent, simulated callers with a constant conversation profile, G.711 A-law, 16 kHz audio, the speech-synthesis phrase cache on.
• Steps. Ramp-up at 2 conversations a second, 90 seconds of warm-up, a measurement window of 4 to 8 minutes. A step is flagged when it fails the checks for steady state, matching of turns and saturated limiters.
• Frame cost. One physical core of the server, 100 and 200 calls of 20 seconds, 95,000 and 190,000 frames per run, four runs. The core’s own timing tests ran on a virtual machine.
• Sources. Stand runs of 9 to 11 November 2026, including a two-hour run at 50 conversations.

TagsCapacityLoad