Why AI Hallucinates npm Packages (And How Attackers Exploit It)

You ask ChatGPT for help with a coding problem. It suggests installing a package: 'npm install flask-session-handler'. You copy the command, run it, and... congratulations, you may have just installed malware. The package exists, but it's not what the AI thought it was—because the AI made it up, and an attacker claimed the name first.

This is package hallucination, and it's becoming one of the most dangerous attack vectors in the AI-assisted development era. In this article, we'll explore why AI models hallucinate package names, how attackers exploit this behavior, and what you can do to protect your projects.

The Hallucination Problem

Large language models don't actually 'know' things—they predict what text should come next based on patterns in their training data. When asked about npm packages, they're essentially pattern-matching against millions of code examples, Stack Overflow posts, and documentation they've seen.

The problem is that these models often generate package names that sound plausible but don't exist. They might combine common naming patterns ('express-', 'react-', '-utils', '-helper') in ways that seem reasonable but point to non-existent packages. Or they might remember a package that was deprecated, renamed, or never existed in the first place.

Security researchers at Vulcan Cyber found that ChatGPT recommended non-existent packages in about 35% of code suggestions involving package installations. That's a massive attack surface.

How Attackers Exploit Package Hallucinations

The attack is deceptively simple. Researchers and attackers alike have started asking AI models common programming questions, collecting the package names they suggest, and checking which ones don't exist on npm. They then register those names with malicious code.

The Harvesting Phase

Attackers systematically query AI models with common programming questions: 'How do I handle sessions in Express?', 'How do I parse XML in Python?', 'How do I connect to PostgreSQL in Node?'. They collect every package name mentioned, then cross-reference against actual package registries.

Non-existent packages become targets. The attacker registers them on npm, PyPI, or other registries with names matching the AI's hallucinations.

The Payload Phase

The malicious packages are designed to look legitimate. They might even provide the functionality the AI described, making detection harder. But they also include malicious code that runs during installation: stealing environment variables, exfiltrating SSH keys, establishing reverse shells, or deploying cryptocurrency miners.

The postinstall script is the most common attack vector. It runs automatically when you npm install, before you ever import the package in your code. By the time you realize something is wrong, the damage is done.

The Waiting Game

Now the attacker waits. Every developer who asks the AI the same question, gets the same hallucinated package recommendation, and blindly installs it becomes a victim. It's a supply chain attack that scales with AI adoption.

Real-World Examples

In March 2024, security researchers demonstrated this attack by identifying packages that ChatGPT consistently hallucinated across different prompts. They found that certain naming patterns triggered reliable hallucinations—packages like 'aio-libs' (instead of the real 'aiohttp'), 'python-jwt' (instead of 'PyJWT'), and 'beautifulsoup' (instead of 'beautifulsoup4').

The researchers registered some of these hallucinated names and tracked installation attempts. Within weeks, they saw thousands of downloads—real developers, trusting AI suggestions, installing packages that shouldn't exist.

Why AI Models Hallucinate Packages

Several factors contribute to package hallucinations:

Training data staleness: The model's knowledge is frozen at its training cutoff. Packages get renamed, deprecated, or created after this date. The AI doesn't know about these changes.

Pattern overgeneralization: The model learns that 'express-session' exists, so 'express-session-handler' seems equally valid. It's pattern-matching, not fact-checking.

Confidence without verification: LLMs can't actually query npm in real-time. They generate plausible-sounding names without any mechanism to verify they exist.

Namespace confusion: The model might confuse packages across ecosystems (npm vs. PyPI), recommend internal/private packages it saw in training data, or merge memories of multiple similar packages into one non-existent hybrid.

Protecting Yourself

Verify Before Installing

Never blindly copy npm install commands from AI. Before installing any package: search for it on npmjs.com, check when it was created (brand new packages are suspicious), look at download counts (legitimate packages have history), read the README and verify it matches what you need, and check the publisher's other packages.

Use Lockfiles and Auditing

Always commit your package-lock.json or yarn.lock. Run npm audit regularly to check for known vulnerabilities. Consider using tools like Socket.dev or Snyk that analyze packages for suspicious behavior, not just known CVEs.

Disable Postinstall Scripts

For maximum safety, you can disable lifecycle scripts entirely: npm install --ignore-scripts. This prevents postinstall attacks but may break legitimate packages that need build steps. A middle ground is using npm's 'allow-scripts' feature to whitelist trusted packages.

Cross-Reference AI Suggestions

When AI suggests a package, search for it alongside official documentation. If you're adding session handling to Express, check the Express documentation for recommended packages. Official docs won't recommend hallucinated packages.

The Bigger Picture

Package hallucination is a symptom of a larger issue: we're trusting AI coding assistants without appropriate verification. These tools are incredibly useful, but they're not infallible. They don't understand security implications, they can't verify facts in real-time, and they confidently provide wrong information.

As AI becomes more integrated into development workflows, we need to develop new security habits. Trust but verify. AI suggestions are a starting point, not gospel. The few seconds it takes to verify a package name could save you from a serious security incident.

Conclusion

AI hallucinating package names isn't a bug that will be fixed—it's an inherent limitation of how language models work. Attackers have already weaponized this behavior, and attacks will only become more sophisticated as AI adoption grows.

The defense is vigilance: verify every package before installing, use security tooling, and never assume AI suggestions are correct. In the age of AI-assisted development, healthy skepticism is your best security tool.

References

• Vulcan Cyber Research - 'Can You Trust ChatGPT's Package Recommendations?'

• Socket.dev - 'AI Package Hallucination Attacks'

• npm Security Best Practices - npmjs.com/policies/security