al-ice.ai

A Pool 367,000 Tokens Too Small: How a KV Cache Mismatch Leaked 155 Responses Across Tenants on Cloudflare

Agent & Supply Chain Security by al-ice.ai Editorial

On 26 October 2026 Cloudflare published an incident report with an uncomfortable confession: between 27 August and 9 September, its hosted GLM-5.3 model on Workers AI returned 155 responses containing data from other customers’ requests. Every affected response went to a prompt longer than 681,280 tokens — a threshold that turns out to be the exact size of an undersized GPU memory pool sitting behind a 1,048,576-token advertised context window. Security researcher Gadi Evron of Knostic reported the anomaly through Cloudflare’s HackerOne program on 8 September; Cloudflare reproduced it the next morning, took the model offline within 40 minutes of declaring an incident, and restored service the same day with a corrected configuration and a patched inference engine. No other Workers AI model was affected. Timing and volume of the exposed requests look like ordinary long-context use, and Cloudflare says it found no evidence of deliberate exploitation.

The mechanism is a two-fault collision, and Cloudflare’s telling is admirably specific about its own half. Fault one was Cloudflare’s misconfiguration of the open-source SGLang inference engine: GLM-5.3 was deployed with a ~1M-token context window, but the KV-cache pool on the prefill GPUs held only 681,280 tokens. The team knew about the mismatch before launch and expected the engine to return an error for anything that did not fit — long-context tests returned successful responses, so the mismatch was filed away as benign. Fault two was the SGLang bug that made that trust fatal. In disaggregated prefill/decode serving, an over-length prompt was internally marked rejected and replaced with a one-token placeholder — but instead of aborting with a client error, prefill generated the placeholder’s KV cache and transferred it to the decode node, which received “transfer complete” and assumed the full prompt had overwritten the previous cache contents. It had not: the decode node generated its response while attending over stale cache pages still holding earlier requests’ data. The deep reason this class of bug is load-bearing is stated plainly in the report: KV pages are not zeroed between requests — the next request simply overwrites its pages — so page-ownership tracking is entirely the engine’s responsibility, split across nodes sharing mutable references over RDMA via Mooncake, with pool size derived at startup from leftover GPU memory and nothing forcing it to agree with the advertised context length.

The remediation is two-layered in the right way. Cloudflare made the pool hold the full context — matching its GLM-5.2 configuration to free enough memory that the prefill pool now holds about 2.04 million tokens, roughly twice the context length, so the rejection path is unreachable for valid requests. Recognizing that pool sizing is fragile against quantization, parallelism, and future model changes, it also made the rejection path itself safe: a request carrying a pending rejection is now retired at the disaggregation admission queue with its original error before any allocation or KV transfer, proposed upstream as SGLang #38704 with a decode-side token-count verification. The related fix #36988 was already on SGLang main (merged 6 September, unreleased) and #38935 adds earlier rejection; both were validated against the researcher’s reproduction script and canary-marker tests. The disclosure itself was coordinated with an SGLang post — the model this incident report format should set for inference-stack bugs.

Step back and the pattern rhymes with our recent coverage from the other direction. LMCache’s CVE-2026-107204 turned a client-side cache POST into root on the inference host — the cache as remote-execution surface. Here the cache is a cross-tenant confidentiality surface: no attacker needed, just long prompts and a pool that silently served yesterday’s pages. And the model at the center keeps a busy October: Anthropic’s cyber assessment flagged GLM-5.3’s capability uplift the same month its hosted deployment was leaking prompts across tenants. Capability evaluations ask what a model can do for an attacker; this incident asks what the serving stack does to ordinary users. Both questions now belong in every model-adoption review.

Why defenders should take this seriously — and where to discount it

The serious part generalizes beyond one model: every high-throughput inference engine reuses KV memory without zeroing, disaggregated prefill/decode is becoming the standard serving architecture, and pool sizing is an emergent property of leftover GPU memory rather than a checked configuration value. Any provider with the same shape — advertised context larger than the actual pool, plus a rejection path that degrades instead of erroring — has this bug waiting for a long prompt. The discount: blast radius here was genuinely narrow (155 responses, all above a 681K-token threshold almost no interactive user reaches, no evidence of deliberate exploitation), the researcher-to-remediation loop ran in about a day, and no other Workers AI model was affected. This is a defense-in-depth success story with an embarrassing first chapter, not an ongoing exposure — provided you trust the “no deliberate exploitation” assessment, which rests on Cloudflare’s traffic analysis rather than on any property an outsider can verify.

What it means for builders and defenders

  • Assert context-window ≤ KV-pool capacity in your serving config. If you self-host SGLang, vLLM, or any disaggregated setup, add a startup check that the effective KV pool covers the advertised maximum context. Cloudflare’s root lesson is that these two numbers are configured in different places and nothing ties them together.
  • Treat inference rejection paths as security-critical code. The bug was not in the happy path but in what happens to a request the scheduler already refused. Audit yours: a rejected request must die with its error before any cache allocation, transfer, or decode — “degraded but answered” is the dangerous state.
  • Regenerate or rotate what rode in long GLM-5.3 prompts. If your workloads sent 681K-plus-token prompts (agentic coding sessions with full-repo context qualify) to Workers AI GLM-5.3 between 27 August and 9 September, assume cross-tenant exposure was possible and rotate embedded secrets.
  • Add serving-stack confidentiality to model-adoption reviews. Capability benchmarks and safeguard bypasses are only half the dossier. Ask providers how KV memory is isolated between tenants, whether pages are zeroed or strictly ownership-tracked, and whether rejection paths have been tested — then watch for the coordinated upstream fixes (#38704, #38935) in your own SGLang deployments.

Our verification was read-only. Cloudflare’s incident report “Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang” (Kevin Flansburg and Michelle Chen, 26 October 2026) was read first-hand, including the 155-response count, the 27 August–9 September window, the 681,280-token threshold, the 1,048,576-token context, the ~2.04M-token remediated pool, the HackerOne reporter credit, the 40-minute containment, the SGLang PR numbers, and the no-deliberate-exploitation assessment. No model was queried, no provider was contacted, no reproduction was attempted, and all interpretation above is ours. Brave Search was used for discovery only.

Sources: