Evaluation Jobs Held the Keys: Fireworks AI's October Secrets Breach

2026-10-25 Agent & Supply Chain Security by al-ice.ai Editorial

On 6 October 2026 at 9am PDT, inference provider Fireworks AI identified a security incident involving unauthorized use of internal Fireworks credentials — and the intruder went straight for the seam where customer secrets meet provider infrastructure. An unauthorized third party used those internal credentials to access environment variables used in evaluation jobs in a limited number of customer accounts, and Fireworks believes API keys and tokens stored as secrets were captured. Containment began the same day; affected customers were notified starting 7 October, employee API keys were rotated on 9 October, all identified affected customers were notified that evening, and the bulletin was published at 10pm. On 10 October Fireworks revoked additional user API keys and rotated service-account tokens found in a completed credential review, with further review continuing.

Fireworks' scoping statement is careful and load-bearing: based on its investigation to date, it has not detected evidence of unauthorized access to customer models, datasets, training data, personal information, or inference traffic including prompts and outputs — adding that the platform is zero-data-retention by default and does not store prompts or outputs. Affected customers received direct email naming the exact credentials requiring action, and the bulletin is explicit about the part most vendors blur: removing a stored secret does not invalidate the underlying credential — third-party credentials require rotation with their issuing providers, plus review of access logs and data reachable through them. Fireworks says it is working with an external cybersecurity firm and will share remediation measures as verified. That is a fast (same-day containment, three-day public bulletin), specific disclosure — and it still leaves the entry vector, the dwell time before 6 October, and the identity of the affected accounts undisclosed outside direct notices.

October's credential-harvest pattern hardens

Fireworks did not happen in isolation; it is the earliest of at least four October incidents converging on the same asset class. Railway disclosed on 21 October that an intruder may have reached encrypted environment variables and some decrypted contents, assessing on the record that the attack is “part of a series of attacks recently targeting platform companies with the aim of compromising AI inference providers.” The very same week, identity-for-agents vendor Andromeda told The Hacker News it had identified a credential-theft incident on 13 October, disabled the compromised accounts that day with no further activity observed, and believes “the activity began as part of an industry-wide attack on or about October 9th” affecting a limited number of customer accounts — with, again, no evidence claimed of access to customer models, datasets, or training data. And Zeabur's 14–15 October breach read 3.2 million environment-variable records from its platform database, with the attacker's container reads starting on LLM-gateway projects. Four vendors, one month, one target: the credentials that buy and operate inference — provider API keys, eval-job secrets, platform-stored environment variables.

Two cautions before anyone merges these into a single campaign. First, the “industry-wide attack” framing is vendor attribution carried as claimed: Andromeda's statement reaches us single-sourced via press reporting — we found no public Andromeda disclosure of its own — and Railway's “series of attacks” assessment cites no corroborating disclosure. The timelines overlap (Fireworks identified 6 October, Andromeda's activity dated to ~9 October, Zeabur 14–15 October, Railway detected 21 October), and the target class matches, but overlap is not attribution. Second, Fireworks' clean-bill items deserve the standard discount: “no detected evidence” of model, dataset, or traffic access is a statement about current telemetry, and the company says its review of calls made with exposed credentials is still in progress. Treat the credential exposure as confirmed and the containment of blast radius as provisional — which is exactly what Fireworks' own rotate-everything guidance does.

The eval-job seam is the finding, not the footnote

The architectural detail that should travel furthest is where the secrets were taken: evaluation jobs. Evals are the workload teams treat as internal QA — they run against production-grade credentials because that is what makes the measurement realistic, they execute third-party or experimental evaluators, and their secret scoping gets a fraction of the review that serving infrastructure receives. An attacker holding provider-internal credentials correctly identified eval jobs as the place where customer keys sit closest to the surface. Our read, labelled as ours: any workload that mixes provider-internal access with customer-supplied secrets needs the same credential hygiene as the serving path — short-lived, narrowly scoped, per-job secrets with usage alerting — or it becomes the softest path to the customer's key wallet. The same logic covers agent sandboxes, batch-inference queues, and fine-tuning jobs: everywhere the provider touches customer secrets on the customer's behalf.

What to do

Fireworks customers: act on the notice, not on the absence of one. Replace revoked Fireworks API keys, rotate every third-party credential named in your notice with its issuing provider (deleting the stored secret is not rotation), and review provider-side logs for reads, downloads, or changes through the exposed keys. If you received no notice, no incident-specific action is required — but confirm your account status rather than assuming it.
Put inference keys at the front of every October rotation queue. Across Fireworks, Railway, Andromeda, and Zeabur, model-provider and cloud-credential theft is the common thread. Revoke and reissue provider keys first, then check usage dashboards and billing at each provider for consumption you did not generate.
Re-scope eval, batch, and sandbox workloads. Move evaluation and experiment jobs to short-lived, per-run credentials with the minimum provider permissions, alert on first use outside the job window, and verify that evaluators — especially third-party ones — cannot exfiltrate the secrets they are handed. Review for unrecognized API keys, evaluators, training jobs, and datasets, as Fireworks advises.
Do not merge the four incidents into one campaign in your threat model. Hunt each exposure on its own dates and indicators; shared targeting of inference credentials justifies a common defensive posture (key inventory, provider-side audit), not a shared-attribution conclusion the vendors themselves have not evidenced.

Our write-up is analysis of Fireworks AI's October 2026 security bulletin, read in full. The Andromeda incident is reported via The Hacker News' ThreatsDay bulletin quoting a company statement; we found no standalone Andromeda disclosure and carry the “industry-wide attack” characterization strictly as the company's attributed claim. We did not contact any affected system or verify any intrusion independently.

Sources: