WHEELS · STARTUP · RESOLVERS · LOCK FILES · PYPI · CI · RELEASES · SIGNATURES · PAYLOADS

Python supply chain attacks:
how a poisoned release reaches you

A hides malicious code inside software you already trust, so it arrives through an ordinary pip install. In 2026 that happened twice to LiteLLM, a Python library installed about 15–20 million times a week. This tutorial follows those releases link by link, from the maintainers’ accounts to the secrets on your machine.

The examples follow ask-docs, a small team’s FastAPI service that answers questions about their product docs through LiteLLM. Its CI job installs dependencies on every push. Its image runs in 3 Kubernetes pods, each with two model API keys and a database address in its environment, and one engineer runs a LiteLLM-based tool from a laptop. The team and its timings are illustrative; every LiteLLM fact comes from the published incident reports.

Here is the whole chain. Most boxes won’t mean much yet; each chapter explains one or two of them.

MapA release travels clockwise: along the top band (the publisher), down into PyPI and back along the bottom band (you). The chapters start at the bottom and work upstream.

You need to know what pip install, a virtual environment and a CI job are. Incident facts are as published by 20 November 2026; tool behaviour is for Python 3.15, pip 26.3 and uv 0.13.9. Organisations are named, people are not, and no code on this page runs the malware. Tap any figure to open it full screen and zoom. are tappable.

The wheel: what pip install puts on disk

ask-docs’s CI job runs pip install -r requirements.txt on every push, and the build goes green. You know it downloaded LiteLLM. You probably haven’t looked at what it wrote to disk, or which of those files can run code. A release on PyPI is a set of files, and the one pip prefers is a wheel. When you install a package, which of its files can run code, and when?

STEP 01 / 03

THE MAP SO FAR

Fig 1.1The accent box is where a wheel ends up. Faint parts arrive in later chapters.

Your side of the chain ends in an environment: a virtualenv on a laptop, a CI runner, a container image. Its installed packages live in one folder, , and every Python started from that environment reads it.

This chapter covers how a release lands there. The faint parts are the rest of the chain: the resolver that picks a version is chapter 03, and the publisher’s side along the top is chapters 05 to 08.

STEP 02 / 03

A WHEEL IS A ZIP FILE THAT PIP UNPACKS

Fig 1.2Each row is a path inside the wheel and where pip copies it; the accent row lands outside the package’s own folder.

A is a zip archive named name-version-pythontag-abitag-platform.whl, such as litellm-1.82.6-py3-none-any.whl. pip unpacks it. The package folder, litellm/, and the , which holds METADATA, RECORD and entry_points.txt, go into site-packages under their own names.

Anything under the wheel’s litellm-1.82.6.data/purelib/ folder goes to the site-packages root, beside every other package. Installing a wheel runs none of the package’s code; pip only writes small launcher scripts for its console commands.

Both 2026 attacks used these ordinary features. The 1.82.8 wheel had litellm_init.pth at its root, 34,628 bytes that pip copied straight into site-packages. 1.107.0 put cook.pth, cook.py and sitecustomize.py under its purelib folder, which lands in the same place.

LIST A WHEEL WITHOUT INSTALLING IT
# Download the file pip would install, without installing it
pip download litellm==1.82.6 --no-deps -d wheels/

# List what is inside, skipping the package folder itself
python -m zipfile -l wheels/litellm-1.82.6-py3-none-any.whl | grep -v '^litellm/'
#   litellm-1.82.6.dist-info/METADATA   ...
#   litellm-1.82.6.dist-info/RECORD     ...

# The removed 1.82.8 wheel had one more entry at the root, as researchers recorded it:
#   litellm_init.pth                    34628
STEP 03 / 03

RECORD MATCHES BECAUSE THE ATTACKER WROTE IT

Fig 1.3The accent row is the malicious file; its hash and size check out like every other row.

is the file in the dist-info folder that lists every file in the wheel with its SHA-256 and size. 1.82.8’s RECORD has the row litellm_init.pth,sha256=ceNa7w…,34628, and that row is correct for that file.

Whoever builds a wheel also writes its RECORD, so a malicious wheel’s RECORD checks out. In practice pip and uv don’t check RECORD at install at all (PEP 815). They check the downloaded file’s hash against the one the index lists, or the one in your lock file.

So the hashes inside a wheel can’t tell you whether to trust it. Who built a wheel, and from what source, is a separate question, and chapter 08 answers it.

COMMON QUESTIONS

Can I see a wheel’s files before installing it?
Yes. pip download litellm==1.82.6 --no-deps saves the wheel without installing it, and any zip tool lists it. PyPI’s inspector (inspector.pypi.io) shows a published file’s contents in the browser.
Does uninstalling remove everything a wheel added?
pip removes the files listed in RECORD. A file the payload created afterwards, such as a backdoor script or a systemd unit, isn’t listed, so it stays (chapter 09).
Are platform wheels riskier than py3-none-any ones?
Not in themselves. They can carry compiled extensions, which are harder to read. 1.107.0 shipped 7 platform wheels, and the payload in each was plain Python.

WHAT YOU NOW KNOW

  • A wheel is a zip that pip unpacks into site-packages; installing it runs none of its code (an sdist is different, chapter 03).
  • A wheel can place files at the site-packages root: 1.82.8 added litellm_init.pth, 34,628 bytes, beside the litellm/ folder.
  • RECORD’s hashes matched in 1.82.8, because whoever builds a wheel also writes its RECORD.
DEEP DIVE: wheel names and tags, the .data folders, console scripts, reading a release without installing it

Wheel names and tags

A wheel’s file name has five parts: name, version, Python tag, ABI tag and platform tag. litellm-1.82.6-py3-none-any.whl ends in py3-none-any: any Python 3, no compiled code, any platform. litellm-1.106.2-cp310-abi3-manylinux_2_28_x86_64.whl is built for one platform; cp310-abi3 means CPython’s stable ABI, from 3.10 on.

pip picks the most specific wheel that matches the machine, and falls back to the sdist when none does. 1.107.0 shipped 7 wheels; Socket reports the release was built with maturin, which produces one wheel per platform.

The .data folders

Under name-version.data/ a wheel can hold five folders, and pip moves each one somewhere else. purelib and platlib go to site-packages, scripts to the environment’s bin/, where its files become commands on PATH, headers to the include folder, and data to the environment’s root.

None of this runs anything at install. The files wait until something imports them, runs them, or, for a .pth file, starts Python (chapter 02).

Console scripts

entry_points.txt lists a package’s console_scripts, each a command name and the function it calls. pip writes a small launcher in bin/ for each. Running one starts Python, imports the package and calls that function, so any import-time code in the modules it imports runs too.

LiteLLM’s litellm command starts its proxy server, which imports litellm.proxy.proxy_server: the module 1.82.7 changed (chapter 02).

Reading a release without installing it

pip download litellm==1.82.6 --no-deps -d wheels/ saves the file and installs nothing, and python -m zipfile -l lists it (step 2’s sample). For a project that might ship only an sdist, add --only-binary :all:: pip may build an sdist to read its metadata, which runs its code (chapter 03).

inspector.pypi.io shows any published file’s contents in the browser. The PyPI JSON API (pypi.org/pypi/litellm/1.82.6/json) gives each file’s digests, the SHA-256 that a lock file should pin.

Interpreter startup: code that runs before yours

The wheel is unpacked and nothing has run yet. Most people assume a package’s code runs when they import it. On 24 March 2026, machines that never imported LiteLLM ran its payload anyway, including some that had it only as a dependency of a tool. Python does real work before your first line, and part of it reads files any package can drop into site-packages. How does a package run code when nothing imports it?

STEP 01 / 04

THE MAP SO FAR

Fig 2.1The accent box runs at every Python start, before any of your code.

Between “you run python” and “your script’s first line”, the runs. It sets up sys.path from site-packages, and in doing so it reads every .pth file there and imports sitecustomize if one exists.

This chapter adds that step to the map: interpreter startup, which reads whatever the wheel left in site-packages.

STEP 02 / 04

AN IMPORT LINE IN A .PTH FILE RUNS AT EVERY START

Fig 2.2Follow hops 1 to 5; the accent hop runs code from a file nobody imported.

site reads each in site-packages in alphabetical order. A line that is a path is added to sys.path. A line that starts with import followed by a space or tab is run with exec().

After the .pth files, site imports , then usercustomize, if it can find them. Only then does your script run. On a machine you trust, the sample below lists what each .pth file in an environment would do.

WHICH .PTH LINES RUN
import pathlib
import site

sp = pathlib.Path(site.getsitepackages()[0])     # this environment's site-packages
for pth in sorted(sp.glob("*.pth")):             # site reads them in this order
    for line in pth.read_text().splitlines():
        if line.startswith(("import ", "import\t")):
            print(pth.name, "RUNS AT EVERY START:", line[:60])
        elif line and not line.startswith("#"):
            print(pth.name, "adds to sys.path:", line)
STEP 03 / 04

THREE WAYS TO RUN BEFORE ANYONE ASKS

Fig 2.3Three columns, one per trigger; the accent row is where 1.82.7 differs from the other two.

1.82.7 added 12 lines at line 128 of litellm/proxy/proxy_server.py: , which reached LiteLLM proxy users only. 1.82.8 kept that and added litellm_init.pth. 1.107.0 used a .pth file too, cook.pth, whose one line is import cook, plus a sitecustomize.py.

litellm_init.pth is one line of 34,628 bytes. It starts import os, subprocess, sys; and then starts a second Python in the background, which decodes a base64 blob and runs it. The line returns at once, so startup looks normal.

That second Python reads the .pth file too, and starts a third. On at least one machine this repeated until about 11,000 processes crashed it, and that crash is how the attack was found.

THE WHOLE .PTH FILE, BLOB SHORTENED
# litellm_init.pth in litellm 1.82.8: one line of 34,628 bytes (the base64 blob is cut here).
# site exec()s it because it starts with "import ": at every Python start, in the background.
import os, subprocess, sys; subprocess.Popen([sys.executable, "-c", "import base64; exec(base64.b64decode('aW1wb3J0IHN1YnByb2Nlc3MK...'))"])
STEP 04 / 04

A .PTH RUNS FOR EVERY TOOL IN THE ENVIRONMENT

Fig 2.4Each chip is one Python start that ran the payload; the accent chips are the CI runner’s (illustrative timings).

In ask-docs’s CI job (illustrative timings), the runner installed 1.82.8 at 10:55. The .pth ran when pytest started, again at pip check, and again in the image build, which resolved 1.82.8 too. Each of the 3 pods ran it when uvicorn started.

On the laptop, built a temporary environment for the docs-search tool and resolved litellm fresh, so every Python start in it ran the payload.

Python 3.15 deprecates .pth import lines but still runs them. PEP 829 stops running them only from Python 3.18.

COMMON QUESTIONS

Can I turn .pth processing off?
python -S skips the site module entirely, and with it site-packages, so most programs break. No switch keeps site-packages and skips only the .pth import lines.
Are all .pth files suspicious?
No. Setuptools ships distutils-precedence.pth, and editable installs write their own. Read each one: a path line is harmless, and a long import line deserves a look.
Does isolated mode (-I) help?
It ignores PYTHON* variables and the user site folder, but it still processes the environment’s own site-packages, where the payload sat.

WHAT YOU NOW KNOW

  • The site module runs at every Python start and passes any .pth line beginning with import to exec(), before your code.
  • 1.82.8’s one-line litellm_init.pth ran in every Python in the environment; 1.82.7 ran only when the LiteLLM proxy was imported.
  • 1.107.0 used both a .pth file and sitecustomize.py. Python 3.15 still runs both.
DEEP DIVE: which folders site reads, sitecustomize and usercustomize, PEP 829 and .start files, auditing an environment

Which folders site reads

site builds its list from sys.prefix and sys.exec_prefix: lib/python3.X/site-packages on Unix. It adds the user site folder too, unless -s, PYTHONNOUSERSITE or a virtual environment turns it off. In each folder it reads the .pth files in alphabetical order, and since Python 3.13 it decodes them as UTF-8 first.

sitecustomize and usercustomize

After the .pth files, site runs import sitecustomize, then import usercustomize when the user site folder is on. Each is an ordinary import, so the first file of that name on sys.path wins.

A package can ship one into site-packages, as 1.107.0 did. If your image already has a sitecustomize.py earlier on sys.path, that one runs instead, which changes which file you need to inspect.

PEP 829 and .start files

PEP 829 adds .start files in Python 3.15: each line names an entry point, such as pkg.mod:callable, that site calls at startup. Import lines in .pth files keep running through 3.17 (with a diagnostic only under -v), are ignored in 3.18 and 3.19, and site is expected to warn about them from 3.20.

The PEP says it doesn’t remove the startup attack surface. A .start file still runs code at every start; what changes is that the code must be a named function in an importable module.

Auditing an environment

Use the shell, not Python, on a machine that might be affected: starting any Python there, pip included, runs a .pth payload again. The sample lists every .pth file with a line that runs code, and any sitecustomize.py or usercustomize.py.

FIND STARTUP HOOKS FROM THE SHELL
# On a machine you trust, ask Python where site-packages is.
# On a suspect one, give the path yourself: starting Python runs any .pth payload.
SP=/opt/app/.venv/lib/python3.15/site-packages

grep -l '^import[[:space:]]' "$SP"/*.pth            # .pth files with a line that runs code
ls -la "$SP"/sitecustomize.py "$SP"/usercustomize.py 2>/dev/null

The resolver: how pip and uv choose a version

Now you know where the payload sat and when it ran. Next is how it got there. ask-docs’s requirements say litellm>=1.80, and nobody on the team chose 1.82.8. The resolver chose it minutes after the upload, because a resolver tries the newest allowed version first. To pick a version it also has to learn each candidate’s dependencies, and for some files that means running their code. How does a resolver pick a version, and why did an illustrative build at 10:55 UTC get 1.82.8?

STEP 01 / 05

THE MAP SO FAR

Fig 3.1The accent box picks the version; the arrow from PyPI is where a new release reaches you.

The in pip or uv turns your requirements into one exact version of every package in the tree, then downloads one file for each. It is the moment a release on PyPI becomes a file on your machine.

This chapter adds the resolver and its download arrow from PyPI. The lock file under it is chapter 04.

STEP 02 / 05

THE RESOLVER TRIES THE NEWEST VERSION FIRST

Fig 3.2Follow hops 1 to 4; at the accent hop the resolver takes the first version that fits and looks no further.

The algorithm, in plain steps. Take an undecided package and list its versions newest first. Keep the first one that satisfies every seen so far. Read that version’s dependencies and add them as undecided. Repeat until nothing is undecided.

When two choices conflict, go back and try the next version: that is . pip uses resolvelib, a backtracking resolver and pip’s default since 20.3. uv uses PubGrub, which records why a conflict happened, so it never retries the same dead end. Both prefer the highest version unless told otherwise.

SEE WHAT WOULD BE PICKED
# What would pip install? Nothing is installed, but any sdist in the tree is still built.
pip install --dry-run --quiet --report - -r requirements.txt \
  | jq -r '.install[].metadata | .name + "==" + .version'

# The same question to uv: one pinned line per package in the tree
uv pip compile requirements.in
STEP 03 / 05

A FRESH ENVIRONMENT TAKES WHAT IS NEWEST NOW

Fig 3.3The accent slice is when the bad versions could be installed; the CI build lands inside it, the laptop doesn’t move.

1.82.6 came out on 22 March. On 24 March 2026, 1.82.7 appeared at 10:39:24 UTC and 1.82.8 at 10:52:19. PyPI got its first report 1 h 19 min after the first upload and quarantined the project 1 h 12 min later: 2 h 32 min in all. LiteLLM is installed about 1,700 times a minute, 40–50% of them unpinned.

ask-docs’s CI runner started empty at 10:55 (illustrative) and took 1.82.8. The engineer’s laptop venv already had 1.82.6, which satisfies litellm>=1.80, so pip install -r requirements.txt printed “Requirement already satisfied” and kept it.

pip upgrades an installed package only with -U, and uv prefers versions that are already installed or locked. A new environment has nothing to prefer.

STEP 04 / 05

RESOLVING CAN RUN CODE BEFORE ANYTHING IS INSTALLED

Fig 3.4Wheel and sdist side by side; the accent row is the only point where choosing a version runs the package’s code.

To check a candidate, the resolver needs its dependencies. A wheel lists them in its METADATA file, which PyPI also serves on its own (PEP 658), so nothing runs.

An has to be built first, which runs the project’s and any code its setup holds. That happens during resolution, under pip install --dry-run and pip download, and once for every version backtracking tries.

--only-binary :all: (pip) and --no-build (uv) refuse sdists. Neither LiteLLM attack needed this route: both shipped wheels.

STEP 05 / 05

YOUR DEPENDENCIES CHOOSE FOR YOU

Fig 3.5One bar per kind of dependent; the accent bar is the 88% whose range let 1.82.7 or 1.82.8 in.

You get what the whole tree allows, not only your own lines. Of the 2,337 packages on PyPI that depend on litellm, 2,054 (88%) declared ranges that allowed the bad versions. Another 209 pinned an exact version, and 74 had an upper bound below 1.82.7.

At FutureSearch, the company whose crashed machine exposed the attack, the engineer involved had never installed LiteLLM. A Cursor MCP plugin run with uvx pulled it in as a , along with 76 other packages.

COMMON QUESTIONS

Why did pip download three versions of one package?
Backtracking. It tried the newest, hit a conflict further down the tree, and went back. Each download is a candidate it checked, and every sdist among them was built.
Do pip and uv pick the same versions?
Usually. Both prefer the highest allowed version, though they search differently. uv’s --resolution lowest picks the oldest allowed instead, which is useful for testing your lower bounds.
Would --resolution lowest-direct have saved the build?
For LiteLLM as a direct dependency it would have picked an old 1.80 release. Transitive dependencies still get the highest, so it is a testing switch, not a defence.

WHAT YOU NOW KNOW

  • pip and uv try the newest allowed version first, so litellm>=1.80 took 1.82.8 minutes after it was uploaded.
  • The fresh CI runner took 1.82.8; the laptop venv that already had 1.82.6 kept it.
  • 2,054 of 2,337 dependents allowed the bad versions, and resolving an sdist can run its build code.
DEEP DIVE: resolvelib and PubGrub, upgrade strategies, markers and forks, metadata without downloads, build isolation

resolvelib and PubGrub

pip’s resolver is resolvelib, driven by a provider with three hooks. get_preference picks which requirement to look at next, with incomplete information. find_matches lists a requirement’s candidates, newest first. get_dependencies fetches a candidate’s metadata on demand, which may mean downloading or building it.

uv’s PubGrub gives priority to URL and == requirements, then looser ones in file order. When it fails, it explains why with the incompatibilities it learned, which is the message uv prints. It prefetches metadata in the background, so it rarely waits on the network.

Upgrade strategies

pip install -U upgrades what you name, and --upgrade-strategy decides the rest: only-if-needed (the default) touches a dependency only when the new version demands it, and eager upgrades everything.

uv prefers versions already locked or installed. A long-lived venv therefore drifts slowly, while a fresh one jumps to the newest release at once.

Markers and forks

A requirement can carry an environment marker, such as ; python_version < "3.10". pip evaluates markers for the one interpreter it runs under. uv’s universal resolution resolves for every supported Python and platform at once, and splits into forks where markers differ, so one uv.lock can hold two versions of a package for two Pythons.

Metadata without downloads

PEP 658 lets an index serve a wheel’s METADATA as a separate file beside it, and PyPI does. pip and uv fetch that small file instead of the whole wheel when they only need dependencies. PyPI’s JSON API (/pypi/<name>/<version>/json) gives the same requires_dist. pip’s docs still call metadata costly, because an sdist has none until it is built.

Build isolation

Building an sdist runs its PEP 517 build backend in a temporary environment that pip or uv creates first, with its own resolve of the build requirements, such as setuptools, hatchling or maturin.

Those come from the index too, at the newest version allowed. A build dependency is a supply-chain link with the same risks as a runtime one.

Lock files, cooldowns and indexes: fewer choices for the resolver

On 24 March 2026 the resolver did exactly its job: it found the newest version every specifier allowed. You can’t make it judge a release, but you can give it fewer choices: exact versions, exact files, a minimum age, and one place to look for each name. What would have kept the illustrative 10:55 build on 1.82.6, without anyone reading the news?

STEP 01 / 04

THE MAP SO FAR

Fig 4.1The accent box answers the resolver before it asks PyPI.

A sits beside the resolver and answers its question in advance: this version of each package, and only a file with this hash. The resolver then has nothing left to choose.

This chapter adds the lock file and its arrow, plus two settings that narrow what the resolver will take: a minimum age for new releases, and where it looks for each name.

STEP 02 / 04

A LOCK FILE WITH HASHES INSTALLS ONLY WHAT YOU REVIEWED

Fig 4.2Each row is one locked package; the accent row is what the build installs, whatever PyPI adds.

A lock file fixes every package in the tree to one version and lists the SHA-256 of each allowed file. In pip refuses any file whose hash is missing or different. Nothing new installs until someone regenerates the lock in a pull request.

With one, ask-docs’s build at 10:55 installs 1.82.6, whatever PyPI holds (illustrative). LiteLLM’s own Docker image pinned its versions and wasn’t affected.

A LOCK FILE WITH HASHES
# requirements.in says what you asked for: litellm>=1.80, fastapi, uvicorn
uv pip compile requirements.in --generate-hashes -o requirements.txt

# requirements.txt now pins every package in the tree, with each allowed file's hash:
#   litellm==1.82.6 \
#       --hash=sha256:164a3ef3e19f309e3cabc199bef3d2045212712fefdfa25fc7f75884a5b5b205

pip install --require-hashes -r requirements.txt   # refuses any file whose hash isn't listed
STEP 03 / 04

A COOLDOWN WAITS OUT THE FIRST HOURS

Fig 4.3The two short accent bars are how long each attack was live; the long bar is a 7-day cooldown.

A refuses versions younger than a set age. Both LiteLLM attacks were removed within hours: 2 h 32 min from upload to quarantine in March (PyPI’s figure), and about 1 h 43 min in October (11:39:51 to 13:23 UTC).

uv has it as exclude-newer = "7 days" (uv 0.9.17 and later), pip as --uploaded-prior-to P7D (pip 26.1 and later). Since July 2026 Dependabot waits 3 days by default, and PyPI’s own incident report recommends 3 days.

A cooldown doesn’t stop an attack nobody spots for a week. It gives the people who do spot one time to report it.

A 7-DAY COOLDOWN
# .github/dependabot.yml: version-update PRs wait until a release is 7 days old
version: 2
updates:
  - package-ecosystem: "pip"
    directory: "/"
    schedule:
      interval: "weekly"
    cooldown:
      default-days: 7

# The same rule in the resolver:
#   uv, pyproject.toml:  [tool.uv]  exclude-newer = "7 days"
#   pip, pip.conf:       [install]  uploaded-prior-to = P7D
STEP 04 / 04

ONE INDEX PER NAME

Fig 4.4Same two indexes, two tools; the accent row is where pip and uv choose differently.

pip checks PyPI, every --extra-index-url and every --find-links location with no priority between them, and takes the best version from any it finds.

Say a company’s private ask-docs-utils is at 1.4.0, and someone uploads ask-docs-utils 99.0.0 to PyPI (illustrative names). pip takes 99.0.0: that is . uv’s default index strategy, first-index, uses only the first index that has the name, and it checks extra indexes before PyPI.

LiteLLM wasn’t attacked this way, but it is the same link: the resolver deciding where a name comes from.

COMMON QUESTIONS

Doesn’t pinning mean missing security fixes?
Only if nobody updates the lock. Let an update bot open a pull request for each new version after the cooldown, and review it like code.
Could an attacker replace 1.82.6’s file with a bad one?
No. PyPI never lets a filename be uploaded twice, even after a deletion, and a locked hash wouldn’t match a different file anyway.
Is my cache a risk?
Yes. pip’s and uv’s caches keep downloaded wheels, so purge them after an incident (chapter 10).

WHAT YOU NOW KNOW

  • With a hash-pinned lock, the 10:55 build installs 1.82.6 whatever PyPI holds.
  • A cooldown of 3 days or more outlasts both attacks, which were live 2 h 32 min and about 1 h 43 min.
  • pip takes the best version across all indexes; uv’s default first-index stops at the first index that has the name.
DEEP DIVE: uv.lock, requirements hashes and pylock.toml, caches and mirrors, typosquatting

uv.lock, requirements hashes and pylock.toml

A hashed requirements.txt (pip-compile --generate-hashes, or uv pip compile with the same flag) is a flat list that pip reads in hash-checking mode. uv.lock is uv’s own format: every package for every platform and Python the project supports, each with its files’ hashes.

pylock.toml is the standard format from PEP 751. pip lock writes one (experimental since pip 25.1), pip install -r pylock.toml reads one (experimental since 26.1), and uv export -o pylock.toml converts a uv.lock.

uv sync --locked fails when uv.lock no longer matches pyproject.toml; --frozen installs from it without checking. Use --locked in CI, so a stale lock fails the build instead of being used as it is.

Caches and mirrors

pip keeps downloaded files in ~/.cache/pip and uv in ~/.cache/uv. A company proxy such as Artifactory or devpi keeps its own copy of every file it fetched. All three can still serve a file after PyPI has removed it.

After an incident, run pip cache purge and uv cache clean, and delete the bad versions from the proxy.

Typosquatting

A lookalike package name is the other common route: requets for requests, or a hyphen added or dropped. The resolver installs exactly the name you typed. The October attack used lookalikes for a GitHub organisation and a domain (chapters 07 and 09), not for the package name.

PyPI: who can upload, and what can be undone

Chapters 01 to 04 stayed on your side: what pip downloads, what runs, and how to choose fewer versions. Every one of those files came from the litellm project on PyPI, which served the two bad March versions over 119,000 times before PyPI quarantined the project. PyPI doesn’t review releases. It checks one thing about an upload: whether the credential may publish to that project. Who can put a new version of litellm on PyPI, and what can PyPI undo once it’s there?

STEP 01 / 04

THE MAP SO FAR

Fig 5.1The accent box is the only part both sides touch. Faint parts arrive in later chapters.

A is a name on pypi.org with owners and maintainers, a list of versions, and each version’s files. Your resolver downloads from it and the publisher’s CI job uploads to it, so it’s the one place both halves of the chain meet.

This chapter covers the project itself: who may add a version, and what PyPI can do to one afterwards. Chapters 06 and 07 walk further upstream, to how the March and October uploads were made.

STEP 02 / 04

ANY ALLOWED CREDENTIAL CAN PUBLISH A VERSION

Fig 5.2Each row is one way to upload; the accent row is the credential behind the March versions.

PyPI accepts a new file when the credential sent with it may publish to the project. Owners and maintainers log in with a password and 2FA, but an upload needs one of two other credentials: an , or a trusted publisher (chapter 06). Nobody at PyPI reads the code first.

Files are immutable. A filename can be uploaded once and never again, even after the file is deleted, so a bad version can be withdrawn but never fixed in place.

The March versions had no matching tag or release in LiteLLM’s GitHub repo. Someone uploaded 1.82.7 and 1.82.8 straight to PyPI with a valid token, and PyPI had no reason to refuse them.

STEP 03 / 04

YANK, QUARANTINE AND DELETE ARE DIFFERENT

Fig 5.3Same rows for each action; the accent row is what a pin to ==1.82.8 gets under each.

A (PEP 592) marks a version so resolvers skip it, unless a requirement pins it exactly with ==. The file stays on PyPI, and the owners can undo it.

is a PyPI admin action, for a whole project or one release. Its files are hidden from the index and can’t be installed, and its maintainers can’t change anything until the admins lift it. A lock file that records direct file URLs can still fetch a quarantined file from PyPI’s storage.

Deleting removes the files for good, and the filename can never be used again. None of the three reaches your side: a copy already in a pip cache, a mirror or a built image stays where it is.

STEP 04 / 04

OVER 119,000 DOWNLOADS BEFORE THE QUARANTINE

Fig 5.4The two slices make up the exposure; the accent slice runs from the first report to PyPI’s quarantine.

PyPI’s incident report gives the numbers: the bad versions were downloaded over 119,000 times. The first report reached PyPI 1 h 19 min after the upload, and the quarantine came 1 h 12 min after that, 2 h 32 min in all. 13 reports came in through “Report project as malware”.

A download isn’t an install. CI retries, mirrors and security scanners all count as downloads, so nobody knows how many machines ran the payload.

October went faster. Socket’s scanner flagged 1.107.0 7 minutes after its upload, and PyPI removed it 1 h 43 min after the upload.

COMMON QUESTIONS

Does PyPI scan uploads for malware?
It acts on reports. Logged-in users report through the project’s Security tab, and reports from trusted reporters can quarantine a project automatically. FutureSearch found the March versions; Socket and Upwind found October’s.
Can I tell who uploaded a file?
A file from a trusted publisher can carry an attestation naming the repo and workflow that built it (chapter 08). A token upload names nothing.
How do I report a malicious release?
Log in, open the project’s Security tab, choose “Report project as malware”, and link the file on inspector.pypi.io.

WHAT YOU NOW KNOW

  • PyPI accepts any upload from a credential the project allows; nobody reviews the code.
  • The March versions were a direct token upload with no GitHub tag; yank, quarantine and delete don’t clean caches or images.
  • The March versions were downloaded over 119,000 times in the 2 h 32 min before quarantine.
DEEP DIVE: owners, maintainers and 2FA, token scopes, quarantine in detail, the advisory database

Owners, maintainers and 2FA

A project has two roles. Owners manage the project: its people, its settings, its trusted publishers and its releases. Maintainers can upload releases, but they can’t invite anyone or delete files.

PyPI requires two-factor authentication on every account, so a stolen password alone can’t log in. An API token never asks for the second factor, which is why tokens are what attackers look for.

Account tokens and project tokens

An API token is scoped either to the whole account, uploading to every project the account can, or to one project. A project-scoped token limits a leak to that project. Either kind works from any machine until it’s revoked.

PyPI takes part in GitHub’s secret scanning, so a token pushed to a public GitHub repo is found and revoked. A token copied out of a CI runner’s memory never passes that check.

Quarantine in detail

PyPI can quarantine a whole project or a single release. A quarantined project disappears from the index, so resolvers find nothing, and its maintainers can’t upload, yank or delete until an admin acts. Reports from trusted reporters can start a quarantine automatically.

In March the whole litellm project was quarantined, and PyPI restored it once the bad versions were removed, so later releases installed as normal. PyPI’s report also covers Telnyx, a second package from the same campaign, which was exposed for 3 h 42 min.

The advisory database

The March versions have the advisory PYSEC-2026-2 (alias MAL-2026-2144) and GHSA-5mg7-485q-xm76, with no CVE. MAL-2026-2144 was extended on 29 October to list 1.107.0; October has no advisory of its own.

pip-audit reads advisories from PyPI or OSV and flags any installed version they list. Its README says not to rely on it against malicious packages: it finds only what has already been reported.

The CI job and the publish token

The March versions were uploaded straight to PyPI with a valid credential. PYSEC-2026-2, the advisory for them, gives the root cause in one line: an API token exposed through an exploited Trivy dependency. Trivy is a security scanner, the kind of tool that runs inside a CI job with everything the job can see. How does a token stored in a CI job end up with an attacker?

STEP 01 / 04

THE MAP SO FAR

Fig 6.1The accent path goes from a tool inside the job, through the credential, to PyPI.

The publisher’s CI job builds, tests and scans the code on every push. In March, LiteLLM’s job also held a that could upload to PyPI: the publish credential on the map.

This chapter adds two parts: the third-party tools that run inside that job, and the credential itself. The people and the release workflow around them are chapter 07.

STEP 02 / 04

EVERY STEP IN A JOB CAN READ THE JOB’S SECRETS

Fig 6.2Follow hops 1 to 4; the accent hops happen inside a step meant to scan for vulnerabilities.

A CI job’s secrets reach its steps as environment variables and files on the runner, and every step runs as the same user on the same machine. A , such as a scanner, can read anything the job can.

On 19 March TeamPCP, the group behind this campaign, published a malicious trivy 0.69.4, live from 18:22 to about 21:42 UTC. It also force-pushed 76 of 77 trivy-action tags and all 7 setup-trivy tags to new code.

That code read the memory of the runner’s Runner.Worker process through /proc/<pid>/mem, where the job’s secrets sit, and sent them to scan.aquasecurtiy.org. LiteLLM’s CircleCI project listed about 230 secrets, several of them PyPI publish credentials.

STEP 03 / 04

A TAG IS A POINTER THAT CAN MOVE

Fig 6.3The accent row is what each step runs after its tag is force-pushed to new code.

uses: aquasecurity/trivy-action@<tag> follows a tag, and whoever can push to that repo can move the tag to new code. A full 40-character commit SHA can’t move, because new code gets a new hash.

Since 28 October 2026 GitHub also offers : gh actions-lock writes .github/workflows/actions.lock, which pins every action, including the ones actions call, and rejects commits that exist only in a fork.

LiteLLM didn’t use the action. Its CircleCI job ran an unpinned apt-get install -y trivy, and Boost Security found that the runs it sampled got the clean 0.69.3. LiteLLM believes the token left through that scan job, but exactly how is still unconfirmed.

A STEP PINNED TO A COMMIT
steps:
  # A tag can be moved by anyone who can push to the action's repository:
  #   - uses: actions/checkout@v7
  # A full commit SHA can't. Keep the version in a comment for people:
  - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1

# Or let GitHub pin every action, including the ones actions call:
#   gh actions-lock    (writes .github/workflows/actions.lock)
STEP 04 / 04

A LONG-LIVED TOKEN WORKS FROM ANYWHERE

Fig 6.4Same rows for both; the accent row is the one the March upload depended on.

A PyPI API token works from any machine until someone revokes it. Whoever copies one can upload without going near CI, which is what happened in March.

replaces the stored token. On PyPI you name an owner, repo, workflow file and, optionally but strongly recommended, an environment. The job asks GitHub for an and swaps it for an upload token that lasts 15 minutes.

PyPI refuses OIDC tokens from pull_request_target runs, so a fork’s pull request can’t publish. Nothing long-lived is left to steal, and PyPI says both projects hit in March have adopted it since.

A PUBLISH JOB WITH NO STORED TOKEN
jobs:
  publish:
    runs-on: ubuntu-latest
    environment: pypi        # the trusted publisher on PyPI names this environment
    permissions:
      id-token: write        # lets the job ask GitHub for an OIDC token, and nothing more
    steps:
      - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333  # v8.0.2
        with:
          name: dist
          path: dist/
      - uses: pypa/gh-action-pypi-publish@7245906fb61fa33a13554f9d5fb856fe61758086  # v1.14.3
        # No password: the action swaps the OIDC token for a 15-minute PyPI token,
        # and uploads attestations with the files by default.

COMMON QUESTIONS

Is the scanner the villain?
No. Any third-party step had the same access. Trivy made a good target because its own repo had already been broken into, and because scanners run in many CI jobs.
Would rotating tokens monthly have helped?
Only if a rotation fell inside the gap. The token was used within days of the poisoned Trivy release, between 19 March and 24 March 2026.
Does Trusted Publishing stop every upload attack?
It stops a stolen-token upload. It trusts the workflow, so whoever can change the workflow can still publish (chapter 07).

WHAT YOU NOW KNOW

  • A CI step can read every secret in its job; the poisoned Trivy action read runner memory.
  • The March token worked outside CI, so the attacker uploaded directly, with no tag in the repo.
  • Trusted Publishing and SHA-pinned steps remove both: no lasting token, and no tag that moves.
DEEP DIVE: the TeamPCP campaign, pull_request_target, the OIDC claims PyPI checks, a separate publish job, CircleCI contexts

The TeamPCP campaign, in order

Trivy’s first compromise came on 28 February, through a pull_request_target workflow. On 19 March came trivy 0.69.4 and the moved action tags, tracked as CVE-2026-33634 (GHSA-69fq-xp46-6x23). Poisoned Trivy images followed on Docker Hub on 22–23 March.

Then came the KICS action, LiteLLM on 24 March 2026 and the Telnyx package on PyPI. Each step used credentials stolen in an earlier one, which is how one scanner’s compromise reached a Python library days later.

pull_request_target and pull_request

A pull_request run from a fork gets a read-only token and no secrets. A pull_request_target run executes the base repository’s workflow with its secrets and a write token, because it is meant for labelling and commenting. When that workflow checks out and runs the fork’s code, the fork’s code gets the secrets.

The OIDC claims PyPI checks

The OIDC token GitHub signs carries claims about the run. PyPI matches repository, repository_owner, repository_owner_id, job_workflow_ref and, when the publisher names one, environment. It doesn’t check ref or sha, so any branch that can run that workflow file can publish. An environment limited to main, with required reviewers, narrows that.

Workflow lockfiles pin actions, including the ones actions call, but not reusable workflows. Pin those by commit SHA in their own uses line.

A separate publish job

Build in one job that holds no secrets, and pass the files on as an artifact. Publish in a second job that has only id-token: write and the pypi environment, and runs no third-party code except the publish action. A poisoned test or scan step then has nothing to read.

CircleCI contexts

CircleCI keeps secrets in project settings or in contexts, which can be limited to some jobs and some groups. LiteLLM’s project listed about 230 secrets that every job in it could read. That is the real finding: one compromised step saw all of them.

The release pipeline and the people who can change it

After March, LiteLLM rebuilt its release process. Releases came from a private release repo, through Trusted Publishing, behind branch protection and an approval gate, with signed images. On 21 October 2026 that pipeline built and published 1.107.0, and the release held a payload that was in no reviewed commit. How did LiteLLM’s own release pipeline publish code that no reviewer saw?

STEP 01 / 04

THE MAP SO FAR

Fig 7.1The accent boxes decide what gets built; every later link trusts their output.

The top of the chain is people: maintainers, their GitHub accounts, and the tokens those accounts issue. They control the source repo, and the that turns it into files on PyPI.

This chapter adds all three. The October attack went through them, and through the same Trusted Publishing that chapter 06 recommends.

STEP 02 / 04

A KEY IN GIT HISTORY OUTLIVES ITS DELETION

Fig 7.2The accent slice is the 136 days the key stayed valid after it was committed.

On 27 May 2026 a config file holding an admin key for LiteLLM’s internal Agent Platform was committed to a public repo. It was removed about 3 hours later, but it stayed in the git history, and nobody revoked it.

On 10 October the attacker used the key. The platform’s agents API returned a GitHub org owner’s CLI , which carries that owner’s access to the BerriAI organisation.

LiteLLM’s first post said the token came from internal logs. Its root-cause post, which it calls the canonical account, traces it to the Agent Platform, and this chapter follows that post.

STEP 03 / 04

AN OWNER TOKEN CAN SWITCH OFF THE PROTECTIONS

Fig 7.3Follow hops 1 to 6 in order; the accent hops changed what the pipeline would build.

is a set of rules a push to a branch must pass, such as reviews and status checks. At 10:08:29 UTC on 21 October 2026, a push to main in LiteLLM’s release repo was rejected by those rules.

At 10:08:34 the owner’s token turned protection off and force-pushed commit 9cf97e9. It added one input to the release workflow, source_owner, so a release could build from another GitHub organisation instead of BerriAI.

It wasn’t the attacker’s first visit. Workflows run on 10 and 12 October had already posted the repo’s secrets to webhook.site, from short-lived branches deleted within minutes.

STEP 04 / 04

THE SOURCE WAS A LOOKALIKE FORK

Fig 7.4Two columns, the real repo and the fork; the accent row is one hyphen.

The release ran with source_owner set to Berri-AI, a organisation, so it built Berri-AI/litellm at commit 20535d8f instead of BerriAI/litellm. The v1.107.0 tag in the real repo has no payload.

At 11:11:18 a second maintainer’s token approved the production gate. That week the rule had been relaxed to any two engineers, and the attacker held both tokens.

7 wheels reached PyPI between 11:39:51 and 11:40:13. Then the runs were deleted and main was force-pushed back, so the release repo looked untouched.

COMMON QUESTIONS

Didn’t 2FA protect the owner account?
The token was a GitHub CLI OAuth token. Using one doesn’t ask for 2FA again, and it carries the owner’s access.
Did Trusted Publishing fail?
No. It confirmed the upload came from the real release workflow, which was true. The workflow had been changed to build from a fork.
Doesn’t a required reviewer stop this?
A GitHub environment needs only one of its listed reviewers to approve, and admins can bypass the rule unless that’s turned off. “Prevent self-review” stops the person who started the run, not a second stolen token.
How was it caught?
Socket’s scanner flagged the wheels 7 minutes after upload, and Upwind opened a GitHub issue at 12:06:41 UTC. LiteLLM began its response around 13:00.

WHAT YOU NOW KNOW

  • An admin key committed on 27 May 2026 and never revoked led to an org owner’s token on 10 October.
  • The owner token switched off branch protection and pointed the release at the Berri-AI fork.
  • A second stolen token passed the approval gate, so the trusted pipeline published 1.107.0.
DEEP DIVE: token types, secret scanning, approval rules that hold, the audit log, the exfiltration workflows

CLI tokens, fine-grained tokens and app tokens

gh auth login stores an OAuth token with broad scopes, such as repo and read:org, and it works until revoked. A fine-grained personal access token is limited to chosen repos and permissions, and it expires.

A GitHub App installation token lasts an hour and covers only the repos the app is installed on. Of the three, an org owner’s CLI token reaches the most.

Secret scanning and push protection

GitHub’s secret scanning runs on every public repo and alerts on known token formats. Push protection refuses a push that contains one, before it reaches history. A home-grown key format matches no pattern unless you register a custom one.

Neither helps once a key is public: anyone may have copied it already. Revoke it first, and only then rewrite history.

Approval rules that hold

An environment’s required reviewers list up to 6 people or teams, and one approval is enough. Turn on “prevent self-review” and turn off the admin bypass.

Then check that no single token, and no one person’s laptop, can both start a release and approve it. Changing the workflow file should need a review too.

The audit log

LiteLLM rebuilt the timeline from the GitHub organisation’s audit log. Each entry records the actor, the action and a hash of the token used, and those hashes tied activity on 10 October to the approval on release day. Stream the audit log somewhere the org owners can’t delete it.

The exfiltration workflows

On 10 and 12 October, workflows on short-lived ci-tmp-* branches ran with the repo’s secrets and posted them to webhook.site. The branches and their runs were deleted within three minutes.

A workflow on any branch can read secrets that aren’t tied to an environment. Keep release secrets in an environment that only main can deploy to.

Signatures and provenance: what they prove

Chapter 07 ended with 1.107.0 published by LiteLLM’s own release workflow. A pipeline like that can sign what it builds and publish a record of how it built it, so anyone downstream can check the file later. The 1.107.0 images carried both: a valid cosign signature and a build provenance record, made by LiteLLM’s real pipeline. Every check passed, and the release was malicious. What does a signature prove, and what check would have caught 1.107.0?

STEP 01 / 03

THE MAP SO FAR

Fig 8.1The accent chip is published beside the file, and anyone downstream can read it. Faint parts arrive in chapters 09 and 10.

When a pipeline publishes a file, it can also publish a signed statement about it: which workflow built it, from which repository and commit. That statement is , and its signature lets anyone check it later. On PyPI, a signed statement attached to a file is an .

This chapter adds that record beside the PyPI project. It changes nothing about what installs. It gives you something to check, and the rest of the chapter is about what that check proves.

STEP 02 / 03

A SIGNATURE PROVES WHICH WORKFLOW BUILT IT

Fig 8.2Rows are claims about a release; the accent row is the one no signature makes.

A keyless signature binds a file to an identity: a workflow file at a ref, plus the OIDC issuer that vouched for it. Its certificate also records the repository that ran the workflow. It has no field for what actions/checkout fetched.

A PyPI attestation (PEP 740) is signed the same way and names the trusted publisher that uploaded the file. Both answer one question: which pipeline built this? Neither says the source was reviewed, and a workflow that takes its source as an input can build anything and sign it.

STEP 03 / 03

THE PROVENANCE NAMED THE FORK

Fig 8.3The same fields for two releases; the accent row is the only one that differs in a way that matters.

1.107.0’s image provenance records run 39587059944 in BerriAI/project-releaser, the real release repository. Its source is https://github.com/Berri-AI/litellm at 20535d8f, and its inputs include source_owner: Berri-AI. The record for 1.106.2 names BerriAI/litellm, one hyphen away.

The evidence was in the signed record, and nothing that checked the release read that field. The check that catches it pins the source as well as the signer: verify the image’s signature, then require its provenance to name https://github.com/BerriAI/litellm. For PyPI files, require an attestation from that repository.

LiteLLM had published no PyPI attestations, so a policy requiring one would have rejected every version, 1.106.2 included. That is the price of the check, and why it works only once publishers attest.

CHECK THE SOURCE, NOT ONLY THE SIGNER
# PyPI attestation: was this file published by the repository you expect?
# (pypi-attestations says its CLI is not a stable interface yet)
pypi-attestations verify pypi --repository https://github.com/BerriAI/litellm \
  pypi:litellm-1.106.2-cp310-abi3-manylinux_2_28_x86_64.whl
# For litellm this fails: it has never published attestations, so there is nothing to check.

# Keyless cosign: which workflow, in which repository, signed this image?
cosign verify ghcr.io/berriai/litellm:main-stable \
  --certificate-identity https://github.com/BerriAI/project-releaser/.github/workflows/release.yml@refs/heads/main \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com
# 1.107.0 would pass this check too. The source it built is in its provenance,
# and that field is the one to pin:  vcs:source = https://github.com/BerriAI/litellm

COMMON QUESTIONS

Would the keyless signing LiteLLM adopted on 15 November have stopped it?
No. The signing identity would be the same: the release workflow on main in BerriAI/project-releaser, which is the pipeline the attacker used. Only a check on the source repository sees the difference.
What about reproducible builds?
Rebuilding from the release tag and comparing would have shown 3 extra files: cook.pth, cook.py, sitecustomize.py. Socket and Upwind found them by comparing the published wheel with the tagged source.
Does pip check attestations at install?
No. Neither pip 26.3 nor uv 0.13.9 verifies attestations when it installs. Checking them is a separate step, with pypi-attestations or PyPI’s integrity API.

WHAT YOU NOW KNOW

  • 1.107.0 had a valid signature and provenance from LiteLLM’s real release pipeline.
  • Its provenance named the source Berri-AI/litellm at 20535d8f; 1.106.2’s named BerriAI/litellm.
  • A check that pins the source repository as well as the signer would have rejected it.
DEEP DIVE: SLSA build levels, transparency logs, PEP 740 and PyPI’s integrity API, diffing a release against its tag

SLSA build levels

SLSA (Supply-chain Levels for Software Artifacts) grades how far you can trust a build’s provenance. Build level 1 means provenance exists. Level 2 means a hosted build platform generated and signed it. Level 3 adds a hardened platform: runs are isolated from each other, and the signing key is out of reach of the build’s own steps.

None of the levels says the source was reviewed. The spec says the provenance’s externalParameters, such as the source repository and ref, must be verified downstream, by you. That is the check 1.107.0 needed.

Transparency logs

Every keyless Sigstore signature is recorded in Rekor, a public append-only log. Anyone can search it for everything an identity signed, so a signature made by a hijacked pipeline leaves a public entry with a time on it.

With a stored key, as LiteLLM used in October, whoever holds the key or the pipeline can sign anything, and the fix is rotation. LiteLLM rotated its cosign key on 23 October. A log of every signature lets a team list what was signed during an incident window and check each entry.

PEP 740 and PyPI’s integrity API

PEP 740 lets PyPI host attestations next to each file. pypa/gh-action-pypi-publish makes them by default when it runs with Trusted Publishing (chapter 06). PyPI serves them at GET /integrity/<project>/<version>/<filename>/provenance with Accept: application/vnd.pypi.integrity.v1+json.

Each attestation carries SLSA provenance and a PyPI publish statement naming the trusted publisher. A file with no attestation has nothing to serve, so a verification policy has to say what a missing one means.

Diffing a release against its tag

The cheapest provenance check needs no signature. Unpack the wheel and compare it with git archive of the tag it claims to come from. Generated files differ, so the first comparison is noisy. A new top-level .pth file, or a sitecustomize.py the repository doesn’t have, is the signal.

For 1.107.0 that comparison shows 3 files the tag doesn’t have. It runs on any release, signed or not, and catches what a signature check can’t: code that the right pipeline built from the wrong source.

The payload: what it takes, where it sends it, how it stays

The first eight chapters traced how a bad release reaches your environment. Now it runs. On ask-docs’s CI run that started at 10:55 on 24 March 2026 (illustrative), pytest has just started, and litellm_init.pth has launched its background Python. The team’s code hasn’t run yet. The payload can already read everything the process can. What does the payload take, where does it send it, and how does it return?

STEP 01 / 04

THE MAP SO FAR

Fig 9.1The accent path is what the payload added: it reads the secrets, and the red leg sends them out.

The payload runs as your process: your user, your environment, your network. It reads the process’s secrets, packs them, and sends them out through the same network your app uses to call OpenAI. Sending stolen data out like this is .

This chapter adds the process’s secrets to the map, and the path out of it. Chapter 10 adds the box that can stop that path.

STEP 02 / 04

IT READS WHAT THE PROCESS CAN READ

Fig 9.2Each row is one place the harvester looked; the accent rows held something in ask-docs’s pods.

March’s harvester read environment variables, SSH keys, ~/.aws, the at 169.254.169.254, GCP and Azure config, ~/.kube/config and service-account tokens. It also read .env files up to 6 folders deep, shell histories, database configs and crypto wallets.

October’s had 16 modules, including AI tools’ credential files. When DATABASE_URL was set, it also read LiteLLM’s own provider-key tables from that database.

In ask-docs (illustrative) that meant two API keys, the database URL and the pod’s . On the CI runner it meant every secret the job had loaded.

STEP 03 / 04

ENCRYPTED, THEN ONE POST

Fig 9.3Follow hops 1 to 3; the accent hop is the only network call, to a domain that looks like LiteLLM’s.

The March payload made a random 32-byte AES-256 key and encrypted the archive with it (openssl enc -aes-256-cbc -pbkdf2). It wrapped that key with an RSA-4096 public key carried in the payload, and put both into tpcp.tar.gz.

One POST sent it to https://models.litellm.cloud/, with the header X-Filename: tpcp.tar.gz. October used the same scheme against http://www.litellm.dev/healthcheck. Both hosts are a of LiteLLM’s name. Only the attacker’s private key unwraps the AES key, so only the attacker can read the archive.

STEP 04 / 04

IT STAYS, AND IN A CLUSTER IT SPREADS

Fig 9.4Each node gets one privileged pod (accent). In ask-docs none started: its token can’t create pods.

March wrote ~/.config/sysmon/sysmon.py and a user systemd unit, sysmon.service, named “System Telemetry Service”. That is : the service waits 5 minutes, then polls checkmarx.zone/raw every 50 minutes for a binary to run.

With a service-account token that may create pods, it also started a privileged pod, node-setup-<node>, in kube-system on every node. Each mounted the host’s root folder and installed the same service on the node itself.

October’s sitecustomize.py polled a file on raw.githubusercontent.com every hour for a domain to fetch code from.

COMMON QUESTIONS

Would antivirus have caught it?
Probably not. The payload was Python started by Python, reading files that tools read every day. What stood out was the destination domain, and on one laptop a crash.
Why encrypt at all?
So that a network log, or a proxy that inspects TLS and captures the body, sees only random bytes, and nobody but the attacker can read the stolen keys.
Did the cluster spread work at ask-docs?
No (illustrative): its service account had default permissions, so the API refused the pod. The pod still lost every secret in its environment.

WHAT YOU NOW KNOW

  • The payload read every credential path a process can reach: environment, cloud config, the metadata service, Kubernetes tokens and .env files.
  • It encrypted the haul and POSTed it to models.litellm.cloud in March and www.litellm.dev in October.
  • It left a polling backdoor outside site-packages and, given a token that could, a privileged pod on every node.
DEEP DIVE: the metadata service and hop limits, reading other processes, the October loader, what network logs show

The metadata service and hop limits

On EC2, the instance metadata service at 169.254.169.254 hands out the node role’s temporary credentials. IMDSv2 requires a session token fetched with a PUT, and the PUT’s response carries a hop limit. With hop limit 1, the response can’t cross the extra network hop into a container behind a bridge, so pods can’t fetch the node’s credentials.

Set it with aws ec2 modify-instance-metadata-options and --http-tokens required --http-put-response-hop-limit 1. Pods that need AWS access then get it from a role of their own (IRSA or EKS Pod Identity), scoped to that pod.

Reading other processes

October’s collector also read the environment of every process it could, through /proc/<pid>/environ. On Linux a process can read that file for other processes of the same user. A CI runner or a container runs most of its work as one user, so one infected process sees its neighbours’ secrets too.

The October loader

October’s sitecustomize.py didn’t carry its next stage. Every hour it read a file on raw.githubusercontent.com for the domain to fetch code from. That is a dead drop: the attacker can change servers by editing one file, without shipping a new release.

Upwind reported the file empty. A dormant stage still counts: whoever controls that file can fill it, and every infected machine would fetch the new code within the hour.

What network logs show

Both attacks show up in the same two places: DNS queries and proxy logs. A machine that resolved models.litellm.cloud, checkmarx.zone or www.litellm.dev ran the payload, whatever its process list says now. The encrypted body tells you nothing; the host name and the time do.

Reports arrive hours or days after the fact, so keep DNS logs long enough to answer “did we ever resolve this name?” weeks later.

Containment and response

Every link so far either let the release through or didn’t notice it. The last link is the one you control at runtime: where your process may connect, and what you do once you learn a bad version ran. ask-docs ran 1.82.8 on 24 March 2026 and found out from the news (illustrative). If a bad version ran an hour ago, what do you do first, and what stops the next one?

STEP 01 / 03

THE MAP SO FAR

Fig 10.1The accent box stops the POST; the ✗ marks the leg that no longer connects.

An sits between your process and the internet and lets out only the hosts the process needs. The payload’s POST is the attack’s last step, so this is the last place to stop it.

This chapter adds egress control to the map, and the response, which has no box: find what ran, remove it, rebuild, then rotate.

STEP 02 / 03

AN ALLOW-LIST LETS ONLY KNOWN HOSTS OUT

Fig 10.2Two allowed calls go out; the red call to the lookalike domain is refused.

ask-docs needs api.openai.com, api.anthropic.com, its database and DNS. A default-deny egress policy, plus a proxy that allows only those names, drops the POST to models.litellm.cloud before anyone has recognised the attack.

Kubernetes NetworkPolicy matches IP addresses and ports, not host names. So the policy below lets pods reach only DNS and the proxy, and the proxy holds the list of names. A network plugin with name-based rules, such as Cilium’s FQDN policies, can replace the proxy. Either way, the cluster needs a network plugin that enforces NetworkPolicy.

DEFAULT-DENY EGRESS FOR THE NAMESPACE
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: ask-docs-egress
  namespace: ask-docs
spec:
  podSelector: {}            # every pod in the namespace
  policyTypes: [Egress]      # nothing leaves unless a rule below allows it
  egress:
    - to:                    # DNS, to the cluster's resolver only
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }
    - to:                    # the proxy, which allows only api.openai.com and api.anthropic.com
        - podSelector:
            matchLabels:
              app: egress-proxy
      ports:
        - { protocol: TCP, port: 3128 }
# (the database gets a rule of its own, by its private address range)
STEP 03 / 03

FIND IT FROM THE SHELL, THEN REPLACE EVERY SECRET

Fig 10.3One row per secret ask-docs held; the accent column is the action, and every row has one.

Every Python start runs a .pth payload again, pip show included, so look with the shell. Search site-packages and the pip and uv caches for litellm_init.pth, cook.pth and sitecustomize.py, then check for sysmon.service, pods named node-setup and the three domains in DNS logs. Each is an .

Then, in order: isolate the host or pod; delete the startup files and the backdoor, and stop its service; purge the pip and uv caches; rebuild images from a lock file with a known-good version, 1.82.6 for March or 1.106.2 for October.

Last, rotate every credential the process could read, whether or not you think it was used. In ask-docs (illustrative) that is both API keys, the database password, the pods’ tokens and the laptop’s keys. The CI job’s AWS role credentials had already expired.

LOOK FOR BOTH ATTACKS FROM THE SHELL
# Don't start Python, pip included: every start runs a .pth payload again.
find / -xdev \( -name 'litellm_init.pth' -o -name 'cook.pth' \) 2>/dev/null   # March, October
find / -xdev -path '*site-packages/sitecustomize.py' 2>/dev/null           # October (or yours)

ls -la ~/.config/sysmon/sysmon.py ~/.config/systemd/user/sysmon.service 2>/dev/null  # March
systemctl --user status sysmon.service --no-pager 2>/dev/null
ls -la /tmp/pglog /tmp/.pg_state 2>/dev/null                                     # March
ls -la /tmp/remote_code_loader.lock /tmp/.pc_once_* 2>/dev/null                  # October

kubectl get pods -n kube-system | grep node-setup                                # March, clusters
grep -rlE 'models\.litellm\.cloud|checkmarx\.zone|litellm\.dev' /var/log/ 2>/dev/null  # DNS, proxy logs

COMMON QUESTIONS

The CI job’s AWS role expired after an hour. Is it safe?
The temporary credentials are dead, but anything they could reach in that hour, such as ECR or an S3 bucket, needs checking in CloudTrail.
Do I need to rebuild images built during the window?
Yes. The .pth file is inside the image, so every container started from it runs the payload.
How do I hear about the next one fast?
OSV publishes malicious-package records (MAL-2026-2144 covers both LiteLLM attacks). uv’s preview malware check, UV_MALWARE_CHECK=1, stops a sync whose lock file matches one. pip-audit reads advisories, and its README says not to rely on it against malware.

WHAT YOU NOW KNOW

  • An egress allow-list of ask-docs’s two APIs, database and DNS would have dropped the POST to models.litellm.cloud.
  • Look for the files with the shell, because any Python start, pip included, runs the payload again.
  • Remove the backdoor first, then rebuild from a lock, then rotate every secret the process could read.
DEEP DIVE: choosing an egress mechanism, checking a cluster, finding affected images, rotation as a script

Choosing an egress mechanism

A forward proxy, such as Squid or Envoy, holds a list of host names. Clients must be configured to use it, and it sees the name in the CONNECT request or the TLS handshake without decrypting anything. A network plugin with FQDN rules, such as Cilium, watches DNS answers and allows the returned addresses for a while, with no proxy settings in the app.

An egress gateway gives outbound traffic one source address, which helps a partner’s firewall but filters nothing by itself. The CI runner needs the same treatment: a self-hosted runner in your own network can sit behind the same proxy, with PyPI, the registry and the model APIs on its list.

Checking a cluster

Beyond node-setup-<node> pods, look for any privileged pod or hostPath mount you didn’t create. On each node, list user systemd units and recent files under ~/.config/systemd. Then read the API audit log for pod creations by service accounts that never create pods.

Finding affected images

Any image built while a bad version was on PyPI may contain it: on 24 March 2026 from 10:39:24 UTC for 2 h 32 min, and on 21 October 2026 from 11:39:51 to 13:23 UTC. Compare build timestamps with those windows, then confirm by listing the image’s site-packages with a shell entrypoint, never Python.

A lock file recorded with each build answers the question without opening the image: it names the version and the hash of every file that went in.

Rotation as a script

Rotation kept on a wiki page takes a day and misses a key. Keep a script, or a runbook your tooling runs, that lists every secret each service holds, where it’s stored and how to reissue it. Short-lived credentials shrink that list: the CI job’s AWS role lasts an hour, and a Trusted Publishing upload token 15 minutes.

The whole picture

The whole chain, numbered. Every number has a row in the table below.

Numbers match the table below. A release travels clockwise, from the attacker on the top left to their server on the bottom left.
  1. 1maintainers

    change the repository and its settings

    trusts
    their accounts and the tokens those accounts issue
    in 2026
    October: an org owner’s token, reached through a leaked admin key
    guard
    scoped, short-lived tokens; revoke a leaked key at once
  2. 2source repo

    hold reviewed code

    trusts
    branch protection
    in 2026
    October: protection switched off, the release pointed at a fork
    guard
    protections no single token can lift
  3. 3release workflow

    build and publish a release

    trusts
    its inputs and its approvers
    in 2026
    October: source_owner set to Berri-AI, approved with a second stolen token
    guard
    a fixed source, approvers no one person can stand in for
  4. 4scan step

    check the code in CI

    trusts
    the tool’s own maintainers
    in 2026
    March: the campaign that poisoned Trivy 0.69.4
    guard
    pin by commit SHA; keep tools away from publish secrets
  5. 5publish credential

    prove an upload is allowed

    trusts
    whoever holds it
    in 2026
    March: an API token used from outside CI
    guard
    Trusted Publishing
  6. 6litellm on PyPI

    serve the files

    trusts
    any credential the project allows
    in 2026
    both: live 2 h 32 min in March, 1 h 43 min in October
    guard
    none on PyPI’s side; detection came from outside
  7. 7signature and provenance

    record who built a file, and from where

    trusts
    the workflow’s identity
    in 2026
    October: a valid signature, provenance naming Berri-AI/litellm
    guard
    check the source repository, not only the signer
  8. 8lock file

    fix every version and file hash

    trusts
    what you reviewed
    in 2026
    protects; the example team had none in March
    guard
    a hash-pinned lock, one index per name
  9. 9resolver

    pick and download versions

    trusts
    the newest allowed version, and any sdist’s build code
    in 2026
    March: litellm>=1.80 took 1.82.8, uploaded minutes earlier
    guard
    a cooldown of days
  10. 10site-packages

    hold the installed files

    trusts
    everything in the wheel
    in 2026
    both: extra files at the site-packages root
    guard
    look at new files in update pull requests
  11. 11interpreter startup

    prepare Python before your code

    trusts
    every .pth file and sitecustomize
    in 2026
    March: a .pth file; October: a .pth file and sitecustomize
    guard
    audit .pth files; Python 3.18 stops running import lines
  12. 12process secrets

    let the app reach its services

    trusts
    anything that runs as the process
    in 2026
    both: every credential the process could read
    guard
    fewer secrets in the environment, minimal service-account rights
  13. 13egress

    connect the app to the internet

    trusts
    nothing, once there is an allow-list
    in 2026
    both: one POST to a lookalike domain
    guard
    an allow-list of the hosts the app calls

Two attacks, the same last links

March entered through a CI tool and a stolen token; October entered through a leaked key and LiteLLM’s own release pipeline. From PyPI onwards they were the same attack: a resolver took the newest version, a startup hook ran it, and the payload sent what it found to a lookalike domain.

Different entry links, the same last four: both ended as a startup file that read secrets and made one POST.

Which defence stops which attack

Each defence guards one link, so it can only stop an attack that passes through that link. Two rows block both attacks, and you can turn both on today without asking anyone upstream.

The accent rows, the lock file and the cooldown, block both attacks; Trusted Publishing and the source check block one each.

THREE THINGS TO KEEP

  • Installing a package lets its authors run code as you. That code can start before yours does.
  • Both bad LiteLLM releases were caught within hours. Install versions you locked, and let new ones wait days.
  • When a bad version ran, assume it read every secret on that machine, and replace them all.
Read next. Prompt injection for another attack that rides on content you trust, and AWS access for the temporary credentials a payload looks for first.

lock it, let it wait, and keep your keys short-lived