LiteLLM Was Poisoned. Here's What It Reveals About AI Tool Supply Chains.

Yesterday, LiteLLM — the Python library that unifies LLM API calls across providers — was compromised. 40,000 GitHub stars. 95 million monthly downloads. 2,000+ dependent packages including DSPy, MLflow, and Open Interpreter.

Versions 1.82.7 and 1.82.8 contained a credential harvester. One pip install was all it took.

This isn’t a story about one package getting hacked. It’s a story about why the entire Python package ecosystem’s trust model is fundamentally broken for AI agent infrastructure — and what a real defense looks like.

What Happened

The attack was a four-step supply chain cascade:

Step 1 (March 19): Trivy v0.69.4 was poisoned. Trivy is Aqua Security’s open-source vulnerability scanner — a tool designed to protect you. The threat actor TeamPCP injected a credential stealer into it.

Step 2 (March 23): LiteLLM’s CI pipeline ran the compromised Trivy to scan its own code for vulnerabilities. During this “security scan,” Trivy silently exfiltrated the maintainer’s PYPI_PUBLISH_PASSWORD.

Step 3 (March 24, morning): TeamPCP published litellm 1.82.7 to PyPI using the stolen credentials. Malicious code was hidden in litellm/proxy/proxy_server.py, executing when developers imported the module.

Step 4 (March 24, hours later): TeamPCP published litellm 1.82.8 — an escalated version. This one added a litellm_init.pth file that executes automatically every time Python starts. No import needed. No function call needed. If Python runs, the malware runs.

The security tool became the attack vector.

The .pth Attack Vector

This is the most technically interesting part. Python’s .pth files are path configuration files processed by the site module at interpreter startup. If a line starts with import, it gets exec()’d — this is documented Python behavior, not a vulnerability.

The attacker exploited this:

import os, subprocess, sys; subprocess.Popen([sys.executable, "-c",
"import base64; exec(base64.b64decode('...'))"],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

This means:

• pip install anything → Python starts → .pth runs → credentials harvested
• python -c "print(1)" → same
• Your IDE starts a language server → same
• pytest runs your test suite → same

No user-visible action. Completely silent. The payload was triple-nested base64 to evade static analysis.

The Three-Stage Payload

Stage 1 — Credential Harvester: Systematically scraped /home, /opt, /srv, /var/www, /app, /data, /tmp for SSH keys, AWS/GCP/Azure credentials, Kubernetes secrets, cryptocurrency wallets, .env files, database passwords, shell history. Encrypted everything with a random 32-byte AES session key, then wrapped the AES key with RSA-OAEP (4096-bit public key embedded in the payload). Exfiltrated to models.litellm.cloud — a convincing impersonation domain.

Stage 2 — Kubernetes Lateral Movement: If it detected a K8s service account token, it deployed privileged pods to every cluster node, then installed persistence droppers.

Stage 3 — Persistent Backdoor: Registered ~/.config/sysmon/sysmon.py as a systemd user service. Polled checkmarx.zone every 50 minutes for new payloads. Downloaded to /tmp/pglog for execution. Had a 5-minute startup delay to evade sandbox analysis. Survived litellm uninstallation.

Why Existing Defenses Failed

pip install --require-hashes? Useless. The malicious files were properly listed in the wheel’s RECORD with correct hashes. Because the package was published with stolen legitimate PyPI credentials, everything was technically “authentic.”

Package signing? Same problem. The credentials were real. The signature was valid.

Security scanning? The attack started by compromising a security scanner. Trivy was supposed to protect LiteLLM. Instead, it became the entry point.

Community reporting? When the issue was filed on GitHub, the attacker used 73 stolen accounts to flood it with 88 spam comments in 102 seconds, then used the stolen maintainer account to close the issue.

The only reason the attack was discovered: the attacker’s own code had a bug. The .pth file spawned subprocess.Popen, and during child process initialization, Python’s site module re-scanned the same .pth, triggering exponential recursion — a fork bomb that crashed a Cursor IDE user’s machine. Karpathy commented: if the attacker had written better code, this might have gone undetected for weeks.

The Real Problem: Implicit Execution

The root issue isn’t LiteLLM. It’s that the Python package ecosystem has multiple paths for code to execute without explicit invocation:

AI agent infrastructure typically combines dozens of packages, each with their own dependency trees. Every dependency is a trust decision that most developers make unconsciously. The LiteLLM attack showed that even packages you never directly installed (transitive dependencies) can harvest your credentials silently.

What Sandboxing Actually Prevents

At Rotifer Protocol, we compile agent capabilities (called Genes) to WebAssembly and execute them in a wasmtime sandbox. This isn’t a theoretical defense — it’s a fundamentally different execution model that eliminates the attack surface LiteLLM was compromised through.

No filesystem access. A sandboxed Gene cannot read ~/.ssh/, ~/.aws/credentials, or any .env file. The WASM sandbox has no filesystem API unless explicitly granted.

No subprocess spawning. subprocess.Popen, child_process.exec, os.system — none of these exist in the WASM execution environment. The .pth attack chain (Popen → base64 → exec) is structurally impossible.

No implicit execution hooks. There is no .pth equivalent in WASM. Code runs when the runtime explicitly invokes it, not when an interpreter starts.

Declared network boundaries. Genes that need network access must declare allowedDomains in their Phenotype — a machine-readable capability manifest. An undeclared POST to models.litellm.cloud would be rejected before the request leaves the sandbox.

Binary-level enforcement. These restrictions aren’t policy rules that can be bypassed — they’re enforced by the wasmtime runtime at the system call level. A Gene compiled to WASM physically cannot issue the syscalls needed to read files or spawn processes, regardless of what its source code attempts.

In v0.8, we ran 22 adversarial tests specifically designed to break these sandbox boundaries: memory out-of-bounds attacks, infinite loops, recursive stack exhaustion, attempted filesystem access, unauthorized network calls. After patching two critical gaps found during testing, zero escape attempts succeeded.

V(g): Scanning for Exactly These Patterns

The V(g) security scanner we shipped in v0.7.9 detects the exact patterns used in the LiteLLM attack:

V(g) scans source code statically — no ML, no heuristics, just pattern matching on the things that matter. It grades tools A through D and generates shields.io-compatible badges that any developer can embed in their README.

When we scanned the Top 50 most-installed ClawHub Skills with V(g), 100% triggered at least one finding. Zero Grade A results. 14% contained dynamic code execution — the exact same technique used in the LiteLLM payload.

The Uncomfortable Conclusion

The LiteLLM incident isn’t an outlier. It’s the logical consequence of an ecosystem where:

1. Trust is transitive and invisible. You trust litellm, which trusts Trivy, which was compromised. You never made a decision about Trivy.
2. Execution is implicit. Code runs not because you called it, but because the interpreter started.
3. Authentication ≠ authorization. Valid credentials don’t mean valid intent. Hash verification and package signing are authentication measures. They tell you who published the package, not what the package does.

The defense isn’t better scanning of Python packages (though that helps). The defense is an execution model where untrusted code physically cannot access the resources it wants to steal.

Compile to WASM. Run in a sandbox. Declare network boundaries explicitly. Make the default “no access” instead of “full access.”

That’s what we’re building.

Immediate Actions If You’re Affected

If you installed litellm 1.82.7 or 1.82.8:

1. Assume all credentials are compromised. Rotate everything: SSH keys, cloud provider credentials, API tokens, database passwords.
2. Check for persistence: ls ~/.config/sysmon/ and ls /tmp/pglog. If either exists, your system has a backdoor.
3. Check for the .pth file: Search your Python site-packages for litellm_init.pth. Remove it.
4. Pin to safe version: pip install litellm==1.82.6
5. Run the community self-check script: gist.github.com/sorrycc/30a765…

Safe versions: litellm <= 1.82.6. Versions 1.82.7 and 1.82.8 are compromised and have been removed from PyPI.