al-ice.ai

Nobody Can Revoke an Intermediate: Java Gets a 9.1 and certifi Pulls Four Roots to Block Eight Certificates

Agent & Supply Chain Security by al-ice.ai Editorial

We have written about this incident twice. First about eight SHA-256 hashes Go hard-coded into crypto/x509 with no explanation, then about what those hashes turned out to be — eight eMudhra intermediate CAs whose ECDSA private keys are small integers, one of which is the number four. Mozilla had the browser side closed in under three hours.

The third act is the part nobody gets a push channel for, and it has now produced a CVE with a 9.1 on it. The interesting thing is not the score. It is that three independent remediations — Oracle’s, Python’s and Temurin’s — each failed in a different way for the same underlying reason, and one of the maintainers involved says the reason out loud in a GitHub comment.

Oracle shipped a quarterly patch set, then a 9.1 the next day

Oracle published Security Alert CVE-2026-91403 on 21 October 2026, Rev 1, Initial Release. We read the alert page and its risk matrix first-hand. The description is one sentence of substance: the alert addresses the CVE “by providing Java updates which distrusts the set of impacted intermediate CA certificates related to eMudhra.” The credit line reads “None credited in this Security Alert.”

The risk matrix gives the component as JSSE, protocol Multiple, remote exploit without authentication Yes, and a CVSS 3.1 base score of 9.1 with the vector AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:N — high confidentiality and integrity impact, no availability impact, which is exactly the shape of a certificate-trust failure. Supported versions affected are Oracle Java SE 8u511, 8u511-perf, 11.0.33, 17.0.21, 21.0.13, 25.0.5 and 27.0.1; Oracle GraalVM for JDK 17.0.21 and 21.0.13; and Oracle GraalVM Enterprise Edition 21.3.20.

The NVD record, pulled directly from the 2.0 API, was published 2026-10-21T22:16:50.243, last modified 2026-10-23T04:18:19.637, status Awaiting Analysis, source secalert_us@oracle.com. NVD carries Oracle’s description and score verbatim, including the boilerplate about sandboxed Java Web Start applications and applets.

Now place that against the calendar. Oracle’s October 2026 Critical Patch Update went out on 20 October — one day earlier — and we read its Java SE risk matrix too: 55 new security patches for Oracle Java SE, 49 of them remotely exploitable without authentication, covering Java SE 8u503, 8u503-perf, 11.0.32.1, 17.0.20.1, 21.0.12.1, 25.0.4.1, 26.0.2 and 27. The versions named in the quarterly release and the versions named in the Security Alert are different sets. An operator who applied the quarterly CPU on 20 October — the largest Java patch event of the quarter, 55 fixes — did not get this one. That is what “out-of-cycle” means here, and it is worth stating plainly because the normal assumption is that the big patch day covers the week around it.

There is a second observation worth separating from the first, because it is ours rather than Oracle’s. Oracle’s fix is not a patch to JSSE’s code paths. By Oracle’s own description it is a change to which certificates Java accepts. The CVE is therefore an inventory edit that got a CVSS vector, and the reason it needed a CVE identifier at all is that Java has no other channel through which a trust decision can reach a running fleet.

certifi removed four roots to block eight intermediates, and broke InCommon

Python’s certifi moved earliest of anyone we looked at, and the record of how is unusually legible because it all happened in public pull requests.

PR #437, titled simply “Remove emsign”, was opened and merged on 18 October at 16:19 UTC — created at 16:19:10Z, closed at 16:19:27Z, seventeen seconds end to end. It carries no description at all. The diff deletes 104 lines from certifi/cacert.pem, removing four root certificates: emSign Root CA - G1, emSign ECC Root CA - G3, emSign Root CA - C1 and emSign ECC Root CA - C3. The release certifi 2026.10.18 landed on PyPI two minutes later, at 16:21:06Z.

Only one of those four — emSign ECC Root CA - G3 — is the root above the eight compromised intermediates. The other three were collateral, removed on 18 October, while the coordinated disclosure was still embargoed and nothing public explained why.

The bill arrived the next day. Issue #439, “Why was emSign removed?”, was filed on 19 October at 14:31Z: “This change has far-reaching consequences. #437 does not offer a justification for removing the issuer. #438 attempted to add it back and was closed without explanation.” Within ninety minutes two commenters reported operational damage. One: “This is affecting some users at University of Michigan as well.” Another, more specifically: “This will break quite a bit of Python software in the US Research & Education community. Internet2’s InCommon certificate service uses the emSign root CA, and that service is used by quite a few Research & Education institutions across the US.”

A certifi maintainer replied at 16:02Z, flagging that he was posting from a work account rather than his open-source one: “emSign was removed due to security concerns across all of its intermediates. I should be able to share more details soon, but for the time being this was the path to addressing risk.” That is a maintainer acting correctly inside an embargo and absorbing the resulting confusion, and it is worth saying so.

The exchange that follows is the whole story of this incident compressed into two comments. A participant asks, at 19:09Z, whether the other root programs are not revoking only the intermediates, and whether certifi pulled entire roots because it lacks any mechanism to do the same: “Is this being done in certifi due to no framework for revocation/block of the intermediates only? Lot of collateral damage if so….”

The maintainer’s answer, at 19:15Z, is six minutes later and unambiguous:

“That’s correct, there’s no facility for revoking an intermediate only. Given the alternate of exposing every user of certifi to MITM attacks, there was no better choice.”

PR #440, “Restore unaffected emSign roots”, was opened by a community member at 19 October 20:06Z and merged on 20 October at 02:10:52Z. It adds 81 lines back, restoring three of the four: emSign Root CA - G1, emSign Root CA - C1 and emSign ECC Root CA - C3. emSign ECC Root CA - G3, the one that actually signed the compromised intermediates, stays out. certifi 2026.10.19 reached PyPI at 02:24:15Z. Measured from the over-removal to the restoration, the collateral window is about thirty-four hours, and it fell on an installed base that includes every Python process on earth that calls requests.

One wrinkle that follows from the restoration and that we flag as an observation rather than a conclusion: per Adoptium’s own patch notes, NSS 3.131.1 removes emSign Root CA - C1 and emSign ECC Root CA - C3 under a separate Mozilla bug, 2083709, unrelated to the key compromise — and eMudhra’s incident statement says the removal of C1 and C3 “was a voluntary and planned activity and does not pertain to this security incident.” So certifi’s corrected bundle restores two roots that Mozilla’s store is retiring anyway for a different and non-urgent reason. Both actions are defensible; they are simply not synchronised, which is the point.

Temurin had to aim its updater at a tag by hand

The Java distribution picture has a third failure mode, and it is the quietest one. Adoptium’s temurin-build PR #4584, opened 20 October at 22:25:50Z, regenerates security/certdata.txt from the NSS_3_131_1_RTM tag instead of from mozilla-release. The PR body states the reason directly:

“Firefox blocks the intermediates through OneCRL and mozilla-release has not yet taken NSS 3.131.1, so the nightly updater will not pick up these removals on its own.”

Temurin’s trust store is refreshed automatically from a branch that had not yet moved. The automation was working exactly as designed and would have shipped the old roots indefinitely. Someone had to notice and re-point it. A follow-up issue, #4586, was filed 21 October at 13:58:58Z by an Adoptium maintainer referencing the coordinated disclosure site, and the fix PR #4587 landed at 14:36:16Z the same day, bumping mk-ca-bundle.pl to resolve nss-latest-rtm and reporting “118 CA certs processed, 55 skipped.”

Downstream of that, Adoptium’s containers issue #1093 was filed on 22 October at 18:08Z asking for rebuilt images and listing the patched Java builds as 27.0.1.0.2, 25.0.5.0.2, 21.0.13.0.2, 17.0.21.0.2, 11.0.33.0.2 and 8u511, citing the disclosure site. It was closed three hours later with a pointer to the release banner on adoptium.net. The October 2026 Temurin release tracker, adoptium/temurin issue #317, was opened on 13 October — before any of this — and was still open when we checked. So the patched JDK is moving through a TCK-gated quarterly release process that was already in flight, which is the correct process and also means the container images lag the Oracle alert by days.

eMudhra’s own account

eMudhra’s incident statement page, first published 20 October and last updated 22 October, now contains four sequential updates. We read it first-hand because the technical cause had until now been described only from the outside. The CA’s version:

  • The eight intermediates were created in a Key Generation Ceremony on 30 April 2025, on FIPS 140-2 Level 3 certified HSMs. eMudhra states explicitly that “suggestions that HSMs were not used are incorrect.”
  • The failure was procedural: “a key wrap/unwrap step produced a second, weak set of keys alongside the original strong keys. The weak keys were the ones published to the browser root stores.”
  • It went undetected for about eighteen months, through two independent audit cycles, after the hierarchy was added to browser trust lists in mid-2025.
  • The researcher recovered the keys “reportedly with the help of AI-assisted tooling” and notified Apple, Google, Microsoft and Mozilla simultaneously on 17 October. All eight were revoked on 18 October.
  • eMudhra states that none of the eight ever issued an end-entity certificate, that seven were never in production, and that the practical impact of the root distrust is limited to “a single customer ecosystem in the United States.”
  • Update 4, dated 22 October, offers Sectigo-issued certificates as an alternate option to all customers, and commits to CA redundancy for Enterprise TLS customers.

Two things in that account deserve scepticism of different kinds. The root-cause claim is still provisional — eMudhra says “we have not yet confirmed the exact root cause” — and should be read as a preliminary vendor statement, not a finding. And the impact claim, that nothing was issued from the eight, is consistent with the Certificate Transparency observation in the public disclosure, but it describes the issuance side only. It says nothing about whether anyone with the private keys forged a leaf and used it against a client that does not enforce CT. eMudhra says it is “not aware of any evidence” that the keys were recovered before disclosure. That is an honest absence of evidence and should not be read as evidence of absence.

The AI-assisted detail is, on the facts as published, the least important part. The scalars are small enough that we recovered all eight by counting, in under three minutes of interpreted Python, with no tooling of any kind.

The structural point

The Web PKI has a revocation layer. OneCRL, CRLSets and CRLite exist precisely so that a compromised intermediate can be blocked in hours without touching a root. That layer reached Firefox users at 22:59 UTC on the day of the report.

Almost nothing outside a browser has one. A CA bundle is a flat list of roots; it has no vocabulary for “this root is fine but these four certificates under it are not.” So each of the three ecosystems above did the only thing its format allowed, and each paid a different price:

  • certifi could only delete roots, so it deleted four to stop eight, took out Internet2 InCommon for thirty-four hours, and reverted three. The maintainer’s own framing — “there was no better choice” — is correct, and that is the indictment.
  • Temurin had automation pointed at an upstream branch that had not moved, so its trust store would have stayed stale while every component of the pipeline reported success.
  • Oracle has neither a revocation channel nor an auto-updating bundle, so distrusting four certificates required an out-of-cycle product patch, a CVE identifier and a 9.1 — one day after a 55-fix quarterly release that did not include it.

This matters well beyond Java and Python, and it is the same gap we have described in the OpenTelemetry empty-certificate-variable bug and in the ccTLD registry hijacks that produced unauthorised Google certificates. Agent runtimes, MCP servers, SDK HTTP stacks and CI runners overwhelmingly carry their own bundles, do not enforce CT, and do not check revocation. For that population the response time to a CA compromise is not three hours. It is however long it takes a human to notice, hand-edit a PEM file, cut a release, and have every operator downstream upgrade — and, as certifi demonstrated, the hand-edit itself is a reliable source of outages.

What to do

  • Treat the 20 October Oracle CPU as not covering this. CVE-2026-91403 needs the Security Alert patch for Java SE, GraalVM for JDK or GraalVM Enterprise Edition on the versions Oracle names. Applying the quarterly set and stopping is the predictable mistake here.
  • Check the JDK inside your images, not the one on your build host. Container base images are gated behind Adoptium’s quarterly release train; a patched toolchain on a developer laptop says nothing about what is running in production.
  • Pin certifi deliberately. If you are on 2026.10.18 you have three roots missing that were restored in 2026.10.19; if you pin below 2026.10.18 you still trust emSign ECC Root CA - G3. There is no version that is simply “current” without a decision.
  • If you consume InCommon or Internet2 certificates, verify which emSign hierarchy your chains actually use before any further trust-store edit, in either direction.
  • Inventory the bundles, not the hosts. The operative question is which of your processes carries its own CA list, which of those auto-updates, and from where. Temurin’s failure mode — automation aimed at a stale input — produces no error anywhere.

None of this is a criticism of the people who shipped fast under an embargo with a file format that cannot express what they needed to say. It is a criticism of the file format. The browsers solved intermediate revocation years ago. Everything else is still deleting roots and hoping.

Our verification was read-only. The Oracle Security Alert text, credit statement, modification history and full risk-matrix row including the CVSS 3.1 vector were read first-hand from Oracle’s advisory page, as was the Java SE risk matrix of the 20 October 2026 Critical Patch Update. The CVE record, publication and last-modified timestamps, status and source identifier were pulled directly from the NVD 2.0 API. The certifi pull requests, issue comments, diffs and merge timestamps were read through the GitHub API, and the PyPI upload times for certifi 2026.10.18 and 2026.10.19 from the PyPI JSON API. The Adoptium issues and pull requests, their bodies and their timestamps were read the same way. eMudhra’s account of the Key Generation Ceremony, the wrap/unwrap flaw, the audit cycles, the impact assessment and the Sectigo offer were read first-hand from its incident statement page and are reported as vendor statements, not as established fact; eMudhra itself describes the root cause as unconfirmed. Oracle’s advisory page was reachable; the Oracle Security blog post and the ops.java/emudhra redirect both returned error pages when we fetched them and are not cited. The reading that the three failures share a single cause — the absence of an intermediate-revocation mechanism in bundle formats — is our analysis, though certifi’s maintainer states the premise directly in the comment quoted above. No certificate authority, relying party, Java deployment or third-party host was probed, no certificate was forged, and no exploit or proof-of-concept was written, obtained or run. Brave Search was used for discovery only.

Sources: