• Posted by: Michal Zuberek
• October 8, 2025

AI-Generated Code and the Expanding Attack Surface of Software Supply Chains

Something has quietly shifted in how software gets written. A significant and growing fraction of production code now originates from large language models — whether through inline suggestions in an IDE, chat-based code generation, or automated PR fixes. For most development teams this has become routine. For attackers, it has created a new and largely underexplored avenue into the software supply chain.

The mechanism is straightforward. Language models are trained on code from the public internet, which means they learn from repositories, documentation, forum posts, and tutorials — including mentions of packages that exist, packages that existed and were removed, and packages that never existed at all. When a model suggests an import or a dependency that it has simply fabricated, and a developer installs it without verifying provenance, they are trusting a name the model generated rather than a package with a known author and history. Attackers have started registering those names in advance.

Hallucinated package names as an attack vector

Researchers have demonstrated that it is possible to query multiple LLMs for code suggestions in a specific domain, compile a list of package names the models invent with some regularity, and then register those names in npm, PyPI, or other registries with malicious payloads attached. The developer asks an AI assistant for help, the assistant suggests a package that sounds plausible, the developer runs the install command, and the attacker's code executes. No phishing, no exploit, no social engineering of a maintainer — just a name that the model made up and the developer trusted.

The code review problem

Beyond hallucinated packages, AI-generated code introduces a subtler problem: the volume of code under review is increasing faster than the capacity to review it carefully. When a developer writes a hundred lines of code, they typically understand each line. When an AI generates a hundred lines and the developer accepts it with light review, there is a higher chance that unexpected behavior — a dependency on an unusual domain, an undocumented call to an external service, an unusual network operation — goes unnoticed. These are exactly the kinds of behaviors that indicate a compromised dependency, but they are easy to miss when the surrounding code was also generated and not fully understood.

What changes and what stays the same

The attack vectors are new but the detection logic is not. A malicious package installed because an AI hallucinated its name behaves the same way at runtime as a malicious package installed through any other supply chain compromise — it makes network connections that the legitimate version of the application would not make. It reaches out to infrastructure that has no place in the software's expected communication profile. A system that understands the baseline of what an application legitimately talks to, and flags deviations from that baseline, catches AI-assisted supply chain attacks the same way it catches traditionally introduced ones.

Updating your threat model

If your organization has begun using AI coding tools — and at this point most have — it is worth revisiting whether your dependency vetting processes account for this vector. Installing a package suggested by an LLM without verifying it against a known registry, checking its download history, reviewing its source, or validating that it has a legitimate author is materially different from installing a package you found through documentation or a trusted recommendation. The convenience of AI-generated code should not be a reason to skip steps that exist for good reasons.

The entry points into your software supply chain are multiplying. The behavioral signals that indicate a compromise remain consistent. If you want to understand what your software is actually doing on the network — regardless of how it was written — we are happy to talk.