CISA Says “Open Source” AI Without Training Data Needs Proprietary-Risk Treatment

2026-08-02 Security by al-ice.ai Editorial

CISA has drawn a hard procurement boundary around a term the AI industry often uses loosely. In federal guidance published July 30, the agency says an AI model that does not provide access to its training data and training pipeline should be treated as though it is not open source—even when its license satisfies the Open Source Initiative’s Open Source AI Definition.

The operational classification CISA recommends is more important than the label: treat such a model as proprietary software with incomplete provenance. Before an agency applies ordinary open-source risk-management assumptions, it should confirm access not only to weights and code but to the full training dataset, methodology, training code, and tuning parameters.

This requirement appears in Open Source Software: Security Principles and Practices, a 31-page guide for federal agencies. Most of the document covers conventional OSS selection, patching, contribution, publication, and lifecycle management. Its AI section makes a narrower—and consequential—argument: the transparency that makes repository evidence useful for software assurance does not automatically exist for a downloadable model.

A valid open license is no longer enough for the risk decision

CISA distinguishes a licensing determination from a security determination. The OSI definition grants freedoms to use, study, modify, and share an AI system. But CISA notes that the definition does not require the original training data to be accessible; sufficiently detailed information that lets a skilled person build a substantially equivalent system may satisfy the definition.

That may be enough to classify distribution rights. CISA says it is not enough for agencies to evaluate trustworthiness. Dataset descriptions cannot expose every poisoned record, hidden correlation, licensing problem, or transformation that shaped the released weights. Nor does the ability to run or fine-tune weights recreate the evidence needed to determine how a suspicious behavior entered the model.

The guidance therefore creates two separate questions for model intake:

Is the system distributed under terms that qualify as open source? That remains a licensing and freedoms question.
Is the system transparent enough to receive open-source risk treatment? For CISA’s federal use case, that requires access to training data and the complete training pipeline.

A procurement form that records only “open,” “open-weight,” or an OSI-compatible license collapses these questions and can grant a model more trust than the available evidence supports.

Weights are executable supply-chain artifacts, not source-code substitutes

CISA identifies model corruption, weight manipulation, and dataset poisoning as distinct security considerations. These risks live at different layers. A conventional source scan can inspect the loader and inference service. A model scanner may detect unsafe serialization or some anomalous tensors. Neither necessarily reconstructs the training examples, labels, ordering, tuning stages, or optimizer behavior that produced a backdoor.

The guidance cites research on computationally undetectable machine-learning backdoors to explain why this evidence gap matters. Its conclusion is categorical: without the training data, defenders lack information needed to find and fix some model vulnerabilities. The document does not claim that complete data access guarantees detection. Rather, it says the absence of that access makes the ordinary OSS transparency model insufficient.

This is also why an updated software bill of materials cannot settle the question alone. An SBOM can bind packages and executable artifacts to versions, hashes, producers, and build context. CISA’s separate OSS guidance asks for evidence inside the model-production process. Teams need both software composition and model provenance; substituting one for the other leaves a blind spot.

CISA’s C4 framework moves trust from identity to observable process

For ordinary OSS, the guide introduces a five-step assessment built around four evidence groups: Codebase, Community, Conduct, and Configuration. Codebase evidence includes commit recency, CVEs, and outdated dependencies. Community covers maintainers and organizational support. Conduct covers review rules, vulnerability disclosure, licenses, and whether maintainers approve their own commits. Configuration asks whether defaults are secure and what hardening the project supports.

CISA explicitly says assessment should focus on how software was produced rather than who produced it. Maintainers can be pseudonymous, globally distributed, and outside contractual obligations. Repository transparency makes development behavior measurable even when identity-based assurance is weak.

The five steps are to select measurable criteria, set risk tolerance and weights for the intended deployment, collect observations, evaluate each criterion, and compare observed risk with tolerance. A criterion such as “follows transparency practices” is too vague; “publishes an SBOM for each version” is measurable. Some criteria can be designated red flags whose failure blocks adoption regardless of the aggregate score.

Tools including Hipcheck, CHAOSS, and OpenSSF Scorecard can collect parts of this evidence, but the guidance warns against confusing presence with quality. A tool may confirm that an SBOM exists without checking whether its contents are complete. The same principle applies more strongly to model cards and dataset summaries: artifact presence is not provenance validation.

The intake decision has to follow the deployment’s authority

C4 is not intended to produce one universal trust score. CISA ties tolerance to the component’s environment, purpose, and access. That makes the framework directly relevant to agents. A local model used for low-impact text classification does not create the same exposure as the same model operating an email assistant, an incident-response shell, a code-writing agent, or an MCP client with production credentials.

For an agentic deployment, measurable criteria should extend beyond repository health to the model and runtime boundary:

Can evaluators access the exact training and tuning data, code, parameters, and provenance records for the released checkpoint?
Are model files cryptographically bound to the evaluated release, distributed in a safe format, and scanned before loading?
Does the serving stack publish complete transitive SBOMs and security advisories for each version?
Are tool permissions, network destinations, data classes, and credential scopes enforced outside the model?
Can operators reproduce evaluations against the actual quantization, adapters, system prompt, and tools used in production?
Is there a supported patch or replacement path when maintainers stop responding?

If data access fails the model-provenance criterion, teams should not compensate by assigning extra points for GitHub stars or a permissive license. Following CISA’s model, it should trigger a different treatment path: assume incomplete provenance, narrow authority, add safeguards, seek more evidence, or reject the component.

Patch ownership stays downstream

The wider guide reinforces a supply-chain reality that model consumers also inherit: OSS maintainers are not vendors contractually obligated to ship a fix. CISA’s two patching principles are “collaborate, don’t demand” and “push fixes upstream.” When a project does not respond to vulnerability information, agencies should re-evaluate whether it is effectively end-of-support, replace it when necessary, and maintain a fork only as a last resort.

CISA also anticipates that frontier AI will increase both vulnerability-report volume and patch-development speed. It recommends automated dependency and patch workflows, CI/CD testing, generated SBOMs, and project disclosure policies that state expectations for AI-supported or semi-automated reports. This is a useful balance: AI-assisted discovery can improve coverage, but maintainers need evidence standards and triage capacity rather than an unbounded stream of unverified findings.

What security and procurement teams should change

Split license status from assurance status. Record distribution rights separately from the evidence available to assess training, tuning, weights, code, and runtime dependencies.
Make “open-weight” explicit. Do not map downloadable weights to “open source” in asset inventories. Preserve whether training data and each pipeline stage are accessible.
Classify evidence gaps. When training artifacts are unavailable, record the model as proprietary with incomplete provenance for risk purposes, consistent with CISA’s recommendation.
Bind evidence to the deployed checkpoint. Hash weights, adapters, quantizations, tokenizer files, configuration, and serving images. A general model-family card is not evidence for every derivative artifact.
Scale authority to provenance. Models with incomplete evidence should not receive broad network access, durable credentials, unattended tool execution, or sensitive data merely because they can be self-hosted.
Use measurable gates. Turn C4-style criteria into reviewable controls with deployment-specific red flags, owners, expiry dates, and events that force re-evaluation.
Plan for abandonment. Track maintainer responsiveness and support status, automate dependency updates, and preserve a tested replacement route before a model or framework becomes operationally irreplaceable.

CISA’s guidance does not settle the philosophical definition of open-source AI. It does something more immediately useful for defenders: it refuses to let a distribution label stand in for evidence. In a software repository, transparency can expose code history, review practice, dependencies, and maintainer behavior. In a model supply chain, weights and a license may expose only the final artifact. Until the training pipeline is available, that difference should remain visible in procurement, architecture, and authorization decisions.

Sources: