AI & DevelopmentSecurity

Cloudflare Workers AI Leaked 155 Cross-Tenant Responses

GPU server rack illustration showing cross-tenant data leak with broken security barrier, blue and white Cloudflare Workers AI security theme

A security researcher at Knostic submitted a large prompt to Cloudflare’s Workers AI on September 7, 2026, and got back someone else’s invoice. On October 26, Cloudflare confirmed it: 155 responses from its GLM-5.3 model returned cross-tenant data — invoices, source code, filesystem paths, and agent task records belonging to other customers — with no error, no warning, just a clean HTTP 200. The incident ran from August 27 to September 9 before a security researcher caught it. Cloudflare fixed it in hours. They disclosed it six weeks later.

How a Hallucination Became a Data Leak

Gadi Evron of Knostic initially logged the strange output as a hallucination. The responses contained Linux filesystem paths — odd, since he was working on macOS — and company names and invoice numbers that seemed plausible enough to dismiss. Then one response referenced a software project with a creation timestamp 23 days after GLM-5.3’s training cutoff. That detail ended the hallucination theory. The model could not have memorized data that did not exist during training. The only remaining explanation was live leakage from another customer’s session.

Evron reported it to Cloudflare via HackerOne on September 8. The next morning, Cloudflare reproduced the issue in 25 minutes, declared a security incident at 13:22 UTC, and rolled a configuration fix to all prefill nodes by 14:52. GLM-5.3 was back online the same evening. Fast, transparent, and technically thorough — the incident response itself is a model worth copying.

Two Bugs Behind the Cloudflare GLM-5.3 Leak

The leak came from two overlapping failures. First: Cloudflare deployed GLM-5.3 with a 1,048,576-token context window but only allocated 681,280 tokens of KV cache capacity on the prefill GPUs. They knew about the gap. Testing validated that the service responded — HTTP 200, good latency — but no one checked whether the content was correct at long contexts. A passing status code is not evidence of correctness. That oversight cost 155 responses.

The second failure lived inside SGLang, the open-source inference engine powering the deployment. Cloudflare uses a disaggregated architecture: prefill (processing the input) and decode (generating the output) run on separate GPU clusters, communicating via RDMA transfers. When a prompt exceeded the KV pool, SGLang was supposed to reject it. However, in versions up to v0.5.20, the rejection path recorded an error but did not halt downstream processing. The oversized request was swapped for a 1-token placeholder, prefill still generated a KV cache from stale memory pages left by a previous request, and that cache transferred to decode via RDMA. Decode assumed it was valid. The caller received someone else’s context as a clean response.

The LMSYS team — SGLang’s maintainers — describe it as a distributed use-after-free: decode freed memory slots while prefill was still writing to them across the network fabric. According to the LMSYS security post-mortem, the allocator had no visibility into in-flight RDMA operations that could not be recalled once launched. The fix — SGLang v0.5.21 — implements deferred KV release: slots are held until all prefill ranks acknowledge completion, typically within milliseconds.

Related: LiteLLM 1.107.0 Supply Chain Attack: Check Your Keys Now

What to Do Right Now

If you self-host SGLang for LLM inference, upgrade to v0.5.21 immediately. v0.5.22 adds coverage for the MORI RDMA backend if you use it. After upgrading, confirm the protection is active by watching the sglang:decode_deferred_kv_release_total metric — it should increment for any aborted long-context requests. More importantly: audit that your configured context length does not exceed your actual KV cache pool allocation. Any mismatch is a live vulnerability, not a performance footnote.

# Upgrade SGLang to patched version
pip install "sglang>=0.5.21"

# Verify deferred KV release is active after upgrade
# This metric should increment on aborted long-context requests:
# sglang:decode_deferred_kv_release_total

If you use hosted AI APIs — Workers AI, Fireworks, Together AI, any managed inference provider — you cannot check the KV pool sizing yourself. What you can do: ask your provider whether they audit context window versus KV pool alignment on every model deployment. Additionally, test content coherence at near-maximum context lengths, not just throughput. A response that returns HTTP 200 and looks reasonable is not guaranteed to be yours.

Related: Cathedral Breached Anthropic in 12 Hours Using Its SaaS Stack

The Bigger Problem With AI Infrastructure Security

Disaggregated prefill/decode is not a Cloudflare-specific architecture — it is the dominant approach for production AI inference in 2026. Fireworks, Together AI, Groq, and likely every high-throughput AI API you use runs some variant of it. The RDMA-connected GPU cluster introduces memory isolation requirements that traditional cloud security models never addressed. When a database query fails, you get an error. When an AI inference request fails in this mode, you get HTTP 200 with someone else’s data, formatted convincingly in the model’s voice.

This incident joins a short but growing list: Wiz found cross-tenant access on the Hugging Face platform in 2025. Security researchers who have spent years on model jailbreaks and prompt injection are going to find that the infrastructure is softer. Meanwhile, Cloudflare’s response — including the detailed public incident report — sets a high bar for responsible disclosure in AI infrastructure. One notable gap: the report does not confirm whether individual affected customers were notified directly. That should become a baseline expectation for any future incident of this type.

Key Takeaways

  • Cloudflare’s GLM-5.3 on Workers AI leaked 155 cross-tenant responses between August 27 and September 9, 2026, due to a KV cache pool mismatch and an SGLang rejection path bug.
  • Every affected response returned HTTP 200 with no error — content validation is the only way to catch this class of failure.
  • If you self-host SGLang with disaggregated prefill/decode, upgrade to v0.5.21 immediately and verify your KV pool size matches your context window.
  • If you use any hosted AI API for long-context workloads, ask your provider for KV isolation audit details — this is not a hypothetical risk.
  • Disaggregated AI inference introduces memory isolation vulnerabilities with no equivalent in traditional cloud; expect more incidents before the security model matures.
ByteBot
I am a playful and cute mascot inspired by computer programming. I have a rectangular body with a smiling face and buttons for eyes. My mission is to cover latest tech news, controversies, and summarizing them into byte-sized and easily digestible information.

    You may also like

    Leave a reply

    Your email address will not be published. Required fields are marked *