perf(gateway): reverse proxy throughput optimizations#825
Draft
kvinwang wants to merge 23 commits into
Draft
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
With TCP_NODELAY enabled, an 8 KiB copy buffer emits many small tail segments and caps bulk throughput. A 64 KiB buffer keeps writes at full segment size; measured ~2x passthrough/terminate throughput on a 4-core gateway with no latency regression. 128 KiB adds <5% for double the per-connection memory, so 64 KiB is the chosen knee.
TLS 1.2 resumption already works via the in-memory session-ID cache, but TLS 1.3 resumption needs a ticketer. Without one, every TLS 1.3 reconnect pays a full handshake (RSA-2048 signing). Measured on tdxlab (4-core gateway, wrk Connection: close): TLS 1.3 reconnect CPS 6.2k -> 9.6k (+53%) with the ticketer, and TLS 1.2 CPS unchanged (~20k). Default TLS version is unchanged.
The accept/handle hot path logged three info-level events plus an info-level span per connection. At the default info log level this cost ~4-8% of TLS-terminate CPS (measured ~19.2k vs ~20.8k req/s on a 4-core gateway). Downgrading them to debug (and the span to debug_span) removes the per-connection logging cost at info while keeping the detail available when debug logging is enabled.
The TLS-passthrough path is a pure TCP relay (the gateway never sees plaintext), so it can move bytes kernel-side with splice(2) through a pipe instead of copying through a userspace buffer. Gated behind the new proxy.tcp_splice_enabled flag (default off, Linux only). Measured on a 4-core gateway (tdxlab): bulk passthrough throughput 7.60 -> 8.52 GB/s (+12%) with lower tail latency under load (p99 38 -> 21 ms). Small-request passthrough latency regresses (extra pipe hop per tiny message), so it stays opt-in. Data integrity verified with a 100 MiB checksum round-trip.
Adds proxy.ktls_enabled (default off, Linux only). The handshake still runs in rustls; afterwards the negotiated keys are installed into the kernel TLS ULP so record crypto happens there, and when combined with tcp_splice_enabled the payload is relayed with splice and never enters userspace. Measured on a dedicated 44-vCPU GCP VM with the gateway pinned to 4 cores (isolated A/B, interleaved rounds): rustls (userspace) 32226 CPS 7.69 GB/s kTLS alone 20598 CPS 5.72 GB/s (-36% / -26%) kTLS + splice 22640 CPS 9.63 GB/s (-30% / +25%) kTLS alone is a loss: without NIC TLS offload (TlsTxDevice = 0) the kernel's software AES-GCM is slower than rustls with aws-lc-rs, plus per-connection setup cost. Combined with splice it gives the highest terminate throughput measured, so it is worth having for bulk-transfer deployments -- but it stays off by default because connection setup rate drops ~30%. Security note: enabling this extracts session keys into the kernel via dangerous_extract_secrets. Inside a CVM the kernel is part of the measured TCB; on a normal host this widens key exposure. Hence opt-in. Verified with a 100 MiB checksum round-trip and TlsDecryptError = 0.
kTLS' cost and benefit sit on opposite sides of the connection-reuse axis: setup is paid per connection (~34% of connection rate) while the win is per byte (+29% bulk throughput). Offloading at handshake time therefore penalises short request/response connections that never earn it back. With the new proxy.ktls_offload_after_bytes, a connection starts in userspace rustls and is handed to the kernel only once it has moved that many bytes: flush, let CorkStream stop at a TLS record boundary, extract the current traffic secrets (they carry the record sequence numbers, which is what makes a mid-stream handover sound), install them into the kernel, splice the rest. 0 keeps the previous immediate-offload behaviour. Measured on a dedicated 44-vCPU GCP VM, gateway pinned to 4 cores, threshold 1 MiB (2 interleaved rounds): rustls 32170 CPS 7.70 GB/s 164459 small-req/s kTLS immediate 21212 CPS 9.95 GB/s 193208 small-req/s (-34% CPS) kTLS adaptive 26303 CPS 9.93 GB/s 194720 small-req/s (-18% CPS) So it recovers about half the lost connection rate while keeping the full throughput gain, and long-lived small-request connections gain ~18% at lower CPU because they cross the threshold and then run zero-copy. Verified: with a 1 MiB threshold a small request does not offload (kernel TlsTxSw unchanged) while a 100 MiB transfer does (counter +1), checksum matches and TlsDecryptError stays 0.
Profiling the small-request workload showed the gateway leaving ~0.5 of 4
cores idle while HAProxy pinned all four. The cause was scheduling, not the
data path: the gateway performed ~0.617 context switches per request against
HAProxy's ~0.000. Two sources, both structural:
* accept runs on its own current_thread runtime and hands every connection
to a separate multi-threaded worker runtime -- one cross-runtime wakeup
per connection;
* tokio's work-stealing scheduler then migrates those tasks between worker
threads, costing further wakeups and cache locality.
With proxy.thread_per_core = true the proxy instead runs `workers` threads,
each with its own current_thread runtime and its own SO_REUSEPORT listener.
The kernel load-balances new connections across the listeners and a connection
is accepted and served entirely on one thread, so nothing is handed off.
Measured on a dedicated 44-vCPU GCP VM, 4 cores, vs the default model and
vs HAProxy given matching settings:
default thread-per-core HAProxy
terminate small-req 158k @3.55 267k @4.03 204k @4.03
terminate CPS 32.8k @3.65 45.0k @3.96 28.2k @4.00
passthrough small-req 175k @3.56 312k @4.02 355k @4.02
terminate throughput 7.5 GB/s 8.3 GB/s 6.0 GB/s
Context switches per request drop to ~0.000 and the proxy now saturates its
cores. That turns two of the three deficits against HAProxy into wins
(terminate small-request +31%, terminate CPS +60%) and closes most of the
third (passthrough small-request now -12%, was -63%).
Off by default: it changes the process model, needs SO_REUSEPORT (Linux), and
per-core accept queues mean a slow connection can only be served by the core
that accepted it. Verified with 100 MiB checksum round-trips on both paths.
Creating a pipe per direction per connection costs two pipe2 calls plus four descriptor closes. That is pure overhead for short connections and it showed: enabling splice raised passthrough connection-setup CPU from 134 to 142 us per connection, while HAProxy -- which pools its pipes -- moved the other way. Pipes now come from a thread-local pool (cap 64), and are only returned once the transfer has drained them; a pipe with unknown residue is closed rather than risk feeding stale bytes to the next connection. Thread-local means no locking, and under thread_per_core a connection stays on the thread that borrowed the pipe. Measured on the 4-core GCP bed (passthrough, 2 interleaved rounds): small-request rate 317.8k -> 335.9k rps (+5.7%, HAProxy 336.8k) bulk throughput unchanged at ~13 GB/s connection setup unchanged So the win is on keep-alive small-request traffic, where it brings the gateway level with HAProxy. Verified with repeated 100 MiB checksum round-trips and interleaved small/large requests, which is what would expose a dirty pipe.
connect_multiple_hosts always built a JoinSet and spawned a task per candidate address so it could race them. With a single candidate -- the common case for an app with one instance -- there is nothing to race, and the allocation plus task spawn was paid on every connection. Take the address directly in that case. Measured on the 4-core GCP bed (passthrough connection setup, 2 interleaved rounds): 139.7 -> 130.7 us of CPU per connection (-6.4%), narrowing the gap to HAProxy's 123.1 us from 13.5% to 6.2%.
The pool only recycled a pipe when its direction observed EOF, so the other direction's pipe was closed and recreated on every connection -- syscall counting showed a steady 1.00 pipe2 per connection instead of ~0. The pipe is in fact empty at every point between chunks, because each chunk is fully drained before the next splice from the socket. Track that directly: mark the pipe recyclable after each complete drain, and only un-mark it while a chunk is in flight. A pipe is still discarded rather than pooled if the connection dies mid-drain. Measured on the 4-core GCP bed (passthrough): pipe2 per connection 1.00 -> 0.00 connection-setup CPU 132.4 -> 120.1 us median (HAProxy 123.4) That closes the last connection-setup deficit against HAProxy; the two now overlap within run-to-run spread. Re-verified with repeated 100 MiB checksum round-trips and interleaved small/large requests, which is what a dirty pooled pipe would corrupt.
splice costs ~17 syscalls per connection to move a small response -- fill the
pipe, drain the pipe, plus readiness retries -- where a read/write pair needs
two. Its benefit is per byte but its cost is per connection, the same shape as
kTLS, so short request/response connections paid for zero-copy they never used.
With the new proxy.tcp_splice_after_bytes a passthrough connection is relayed
with plain reads/writes until it has moved that many bytes, then switches to
splice for the remainder. 0 keeps the previous behaviour.
The relay buffers come from a thread-local pool for the same reason the pipes
do: a first cut that allocated two buffer_size buffers per connection was
*worse* than the pipe setup it replaced (148.6 us vs 133.5 us per connection).
Pooled, and sized for this phase rather than for bulk copying, it wins.
Measured on the 4-core GCP bed (passthrough, medians of 3 interleaved rounds,
threshold 64 KiB):
always splice adaptive HAProxy
connection setup 123.0 us 117.2 us 123.5 us
small-request rate 300.0k rps 295.3k rps 344.6k rps
bulk throughput 12.6 GB/s 12.4 GB/s 10.2 GB/s
splice syscalls per connection drop from 16.86 to 0.00 on the connection-churn
workload, and connection setup costs slightly less CPU than HAProxy. Keep-alive
and bulk traffic are unchanged within run-to-run spread (~2%), since those
connections cross the threshold and end up spliced anyway.
Verified with checksum round-trips at 1 MiB and 100 MiB plus payloads sized
just under and just over the threshold, which is where the handover happens.
The thread-per-core path built its listener with socket2 and a 4096-deep accept queue; the default path used TcpListener::bind, i.e. tokio's default backlog of 1024. That is where SYN drops begin under connection bursts, and it meant the two modes behaved differently for no reason. Both paths now share one socket2-based bind, differing only in whether SO_REUSEPORT is set, with the backlog in a named constant.
A 40-minute soak produced 379 error-level lines and not one real fault. They were all the peer hanging up: connection reset while bridging, while splicing, or during the TLS handshake. That is routine -- a browser navigating away, a mobile network dropping, a load generator ending its run -- and logging it at error level buries the failures that are actually the gateway's fault. Classify the error chain: if any cause is a ConnectionReset, BrokenPipe, UnexpectedEof or ConnectionAborted io::Error, log at debug instead. Also records what the soak found about the splice pipe pool's descriptor cost (PIPE_POOL_MAX * 2 * workers, so ~512 for 4 workers and ~4096 for 32), which is why that cap is not larger.
… probe Thread-per-core measured 22-77% better on every metric, on loopback and again over a real network, and it is what turned the deficits against HAProxy into wins. The only thing holding it back was that it had never run longer than a benchmark. A soak of 8 x 60s of mixed load (small-request, bulk, connection-churn) says it is stable: RSS 21.9 MB cold -> ~44 MB steady, flat across all rounds fds 31 -> 543, plateauing at the pipe pool cap, then flat threads 11, constant drift small-request -2.3%, connection rate -1.3% (both within noise) Flipping the default needs a safety net, because the model cannot work without SO_REUSEPORT and failing to serve is much worse than losing the optimisation. `start` now probes a SO_REUSEPORT bind first and falls back to the shared-runtime proxy with a warning if it fails. Verified by occupying the port with a non-REUSEPORT socket: the gateway logs the fallback and keeps serving. Deployments that want the old process model can still set proxy.thread_per_core = false. The one behavioural difference to be aware of is that per-core accept queues mean a connection is served by the core that accepted it, so a slow core cannot be helped by its neighbours.
…bridge
Two costs the small-request passthrough profile put at the top of userspace,
both in io_bridge:
* `tokio::io::split` wraps the stream in a `BiLock`, and its `poll_read`
showed at 1.1% of total time. Passthrough always has a plain socket on both
sides, so it can use `TcpStream::split`, which hands out borrowed halves
with no lock. Added `bridge_tcp` for that case.
* every read, write and flush was wrapped in `tokio::time::timeout`, i.e.
three timer arm/disarm pairs per request; `TimerEntry` drop plus
poll_elapsed came to 2.9%. Replaced with one watchdog per connection that
samples per-direction progress counters -- the hot path now costs an integer
increment instead of touching the timer wheel, and no clock is read.
The watchdog is coarser than the old per-operation timeouts: it fires within
`idle/4`, so a stalled connection is closed in up to 1.25x `idle` rather than
exactly at it. Verified with `idle = 3s`: old path closed at 3.0s, new at 3.7s.
Measured on the 4-core GCP bed with the corrected harness (physical-core
disjoint pinning, warmed up, n=5, sd<1%):
passthrough small-request, default config 257 151 -> 290 440 rps (+12.9%)
cpu per request 15.7 us -> 13.9 us
p99 221 us -> 197 us
Note this only moves the default path: with `tcp_splice_enabled` the relay is
`splice_bidirectional`, which never used either mechanism.
An earlier attempt to answer the same question by toggling
`data_timeout_enabled` reported "no effect" -- that was wrong, because the bench
config generator wrote the key under `[core.proxy]` instead of
`[core.proxy.timeouts]`, so serde silently ignored it and both arms ran with
timers on.
`splice_one` awaited `readable()` / `writable()` before every splice, even when the socket was already known to be ready. That builds, polls and drops a `Readiness` future each time; on the small-request passthrough profile `ScheduledIo::poll_readiness` plus `Readiness::drop` came to 2.4% of total time. tokio's `try_io` already decides with a single atomic read of the cached readiness flag and returns WouldBlock without calling the closure if the socket is not ready, so the wait is only needed on that path. Reordered to attempt the syscall first and await readiness only when it blocks. Measured (4-core GCP bed, corrected harness, n=5, sd 0.3%): passthrough small-request, splice enabled 291 546 -> 294 583 rps (+1.0%) cpu per request 13.9 us -> 13.7 us Smaller than the profile share suggested, but reproducible and above noise. 100 MiB checksum round-trip re-verified.
Each thread-per-core worker used to bind its own SO_REUSEPORT listener, so the order sockets joined the group was whatever the thread scheduler produced. Bind them all up front instead and hand listener i to worker i: the setup leaves the per-thread path, and group order becomes ours. That order matters for anything that steers connections by listener index, which is what motivated this. Recording the result of that attempt here, because it was a dead end worth not repeating: SO_REUSEPORT's default 4-tuple hash distributes moderate numbers of long-lived connections unevenly, and thread-per-core cannot rebalance afterwards. At 16 connections over 4 cores the per-core utilisation came out like [99, 42, 101, 101] and throughput swung 46% between restarts purely on how the hash fell. SO_ATTACH_REUSEPORT_CBPF was tried to replace the hash. Classic BPF has no maps, so round-robin is not expressible; the only useful signal it exposes is the CPU handling the packet, so the program steered by `cpu % n`. Measured over 6 restarts each at 16 connections: default hash 245 539 rps 4.0 of 4 cores active lowest core 73% cpu steering 167 331 rps 2.3 of 4 cores active lowest core 100% It collapses the distribution instead of evening it out: connections are established in a burst, so only the one or two client CPUs live at that moment get sampled, and every connection lands on the same one or two listeners. The knob was removed rather than shipped -- a setting that costs 32% in the only scenario measured is worse than no setting. A real fix needs per-connection state, i.e. SO_ATTACH_REUSEPORT_EBPF with a counter map, or accepting from a shared queue as HAProxy does.
SO_REUSEPORT picks a listener by hashing the connection's 4-tuple, and thread-per-core cannot move a connection afterwards, so a core that draws few connections idles while its neighbours saturate. At 16 connections over 4 cores utilisation came out like [99, 42, 101, 101] and throughput varied 46% between restarts on nothing but how the hash fell. Steering the kernel's choice with SO_ATTACH_REUSEPORT_CBPF was tried first and measured worse (see the previous commit). This fixes it on our side instead: the accepting core compares its own connection count against the least loaded core and, if it is ahead by more than a small slack, sends the connection over a channel for that core to serve. The cost is one channel send per *rebalanced connection* -- never per connection, never per request -- so everything already in flight keeps the thread-per-core locality that made this model fast. Measured over 6 restarts per arm, 4-core gateway, passthrough small-request: connections off on 16 236 110 rps, worst core 72% 251 840 rps, worst core 85% (+6.7%) 50 291 932 rps 287 440 rps (-1.5%) CPS 17 636 17 557 (-0.4%) The worst case improves more than the average: no core dropped below 76% with rebalancing, against a low of 47% without it. At high connection counts the hash is already even, so the balancer only costs its own bookkeeping. Off by default (`connection_rebalance`) since it trades a little at the high end for a lot at the low end, and which matters depends on the deployment.
…fer-pool wiring `Balancer::slot_for_self` became dead once handoffs carried their slot with them, and `PooledBufs::pair` once the relay borrowed the buffer fields directly.
…s reactor
`TcpListener::poll_accept` hands back a `TcpStream` already registered with the
accepting core's reactor. Moving that object to another core left the
registration behind, so every read and write on a rebalanced connection went
through the accepting core's epoll and woke the owning core across the channel
-- for the whole life of the connection, not just at handover.
That turned the rebalance into a net loss whenever the hash was already even.
Measured on passthrough small-request at 50 connections, where only ~50 accepts
happen and migrations should be invisible:
rebalance off 295 444 rps
rebalance on, code path but no 295 033 rps (+0.2%)
migrations (slack raised to 999)
rebalance on, migrating 288 825 rps (-1.9%)
The middle arm is what isolates it: every line of the balancer runs, and it
costs nothing. The cost was per *migrated connection*, and scaled with how
aggressively the threshold migrated -- which is what a permanently misplaced
reactor registration looks like, not what a channel send looks like.
Handing over the plain `std::net::TcpStream` deregisters on the way out and
re-registers on the way in. The penalty disappears: +0.4% at 50 connections
(was -2.5%), +0.8% on TLS terminate (was -2.1%), and the gain at 16 connections
grows from +4.0% to +10.7%.
…ng a fixed slack
A fixed slack of 2 is the right amount of hysteresis at 4 connections per core
and far too little at 12: it keeps migrating on differences that are noise at
higher load. Comparing thresholds on passthrough small-request:
connections 16 50
slack 2 236 646 290 542
slack 1 240 190 285 047
slack 0 239 335 284 847
proportional 246 216 287 714
and the worst-loaded core at 16 connections goes 81% (slack 2) -> 95%
(proportional). `least + 1 + least/4` migrates on relative imbalance, so it
still reacts when one core has 6 connections and another has 1, and stops
reacting when they have 13 and 12.
Also tried and dropped: capping accepts per scheduler turn the way HAProxy's
`maxaccept` does, and pushing every connection through the handoff channel even
when it stays local (HAProxy does this deliberately, for cache locality). Both
measured within noise here once the threshold was right.
This was opt-in because handing a connection over cost 2-3% wherever the `SO_REUSEPORT` hash was already even. That cost was a bug, not a trade -- the migrated socket kept its readiness registration on the accepting core's reactor -- and with it fixed there is no longer a workload where the rebalance loses. Measured on a 4-core gateway, 3-5 runs per arm after warmup, with bulk throughput re-run interleaved (off/on/off/on, n=5 each) because a single pass had suggested a 2% regression that turned out to be noise: | workload | off | on | | |---------------------------------------|-----------:|-----------:|-------:| | passthrough small-request, 8 conns | 113 350 | 143 341 | +26.5% | | passthrough small-request, 16 conns | 231 043 | 258 787 | +12.0% | | passthrough small-request, 50 conns | 294 937 | 295 311 | +0.1% | | TLS terminate small-request, 50 conns | 244 304 | 246 228 | +0.8% | | TLS terminate, connections/s | 41 024 | 43 019 | +4.9% | | passthrough, connections/s | 17 431 | 17 319 | -0.6% | | passthrough, bulk throughput | 12.10 GB/s | 12.16 GB/s | +0.5% | The gain concentrates where the hash has fewest connections to spread: at 16 connections the quietest core goes from 63% to 97% busy. At 8 and 16 connections this also moves us from behind HAProxy to level with it (143 341 vs 140 278, and 258 787 vs 261 407).
kvinwang
force-pushed
the
gateway-perf-clean
branch
from
July 26, 2026 08:34
84c0238 to
8be5386
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Performance work on
dstack-gateway's reverse proxy path, developed and benchmarked in session a31279f8.Rebased onto
masterwith only the 23 gateway proxy commits (no test-plan / full-plan history).Gateway proxy changes
splice(2)zero-copy relay for TLS passthrough (Linux), with optional byte thresholdJoinSetwhen only one upstream candidateConfig (
[core.proxy])buffer_size65536thread_per_coretrueconnection_rebalancetruethread_per_coretcp_splice_enabledfalsetcp_splice_after_bytes0ktls_enabledfalsektls_offload_after_bytes0Benchmark notes (from the session)
Validated on dedicated GCP machines against HAProxy where applicable; both TLS passthrough and terminate paths were exercised. Highlights:
epoll_ctlcost is ~0 in keep-alive windows (placement at accept only); ~0.04–0.15% of gateway CPU in CPS worst case.Test plan
cargo check -p dstack-gateway/ gateway unit teststhread_per_core+ rebalance)tcp_splice_enabled/ktls_enabledand re-run smoke