Updated 2026-10-22 with BerriAI’s incident report (its 20:30 UTC update): exact times, how the release got out, more files to look for and more secrets to rotate. The investigation is still open, so check the report for later updates before acting.

What happened

Times are UTC, 2026-10-21, from BerriAI’s incident report and StepSecurity:

How it got out

According to BerriAI, the attacker held GitHub CLI credentials of three developer accounts: one changed and ran the release workflow, one approved the production release, and one deleted the run records. A token of a GitHub organization owner was used to turn off branch protection. The official v1.107.0 source tag does not contain the malicious files; the artifacts were built from the lookalike repository. How the attacker got the tokens in the first place is not yet confirmed.

What is affected

• PyPI: litellm==1.107.0, all seven wheels (macOS, Linux glibc and musl, Windows). The sdist has a tampered build config but no payload.
• Docker: the v1.107.0 tag of ghcr.io/berriai/litellm, litellm-database, litellm-non_root and docker.io/litellm/litellm. The floating tags latest, main-latest and main-stable pointed to the infected image from 11:46 to 14:20 UTC, about two and a half hours. The Docker Hub index digest is sha256:1bbcb1443bc962e45f5aad53fe3d2c30600e0379715fb6c664cdd3ac126c339e.
• Reported clean: BerriAI says every earlier version is safe; StepSecurity also checked 1.107.0rc1, 1.107.0rc2 and the 1.103.x–1.106.x releases.

What the malware does

It installs three files into site-packages: cook.pth, cook.py and sitecustomize.py. cook.pth imports cook, so the stealer runs on every Python interpreter start in that environment. Your code does not have to import litellm.

It collects browser data, SSH and GPG keys, cloud and CI credentials, .env files, the environment variables of readable processes and tokens of AI coding tools. If DATABASE_URL is set, it also dumps the LiteLLM proxy database, which holds upstream provider API keys. The bundle is encrypted and sent by HTTP POST to litellm.dev. On Linux and macOS a second component polls a dead-drop file every hour and executes whatever code it points to.

Are you affected?

Do not run Python in an environment you suspect. Every interpreter start re-runs the stealer, so pip show litellm is not a safe check. Use file-system and lockfile searches instead:

# Is the malicious version pinned anywhere in your repos?
grep -rn "litellm.*1\.107\.0" --include="requirements*.txt" --include="*.lock" --include="pyproject.toml" .

# Are the payload files on this host or in an extracted image layer?
find / \( -name cook.pth -o -name cook.py -o -name 'cook*.pyc' -o -name 'sitecustomize*.pyc' \
  -o -name "litellm-1.107.0*" \) 2>/dev/null

# Loader still running? (a live holder process means yes)
ls -l /tmp/remote_code_loader.lock /tmp/.pc_once_* 2>/dev/null

Finding any of these confirms exposure; not finding them does not prove a host is clean. A /tmp/.pc_once_* file is written only after the stealer got a successful reply from its server, so it means the data left. sitecustomize.py on its own is not proof: some environments ship one.

For containers, check which tags and digests you pulled, not what runs inside:

docker images --digests | grep -i litellm
# compromised Docker Hub index: sha256:1bbcb1443bc962e45f5aad53fe3d2c30600e0379715fb6c664cdd3ac126c339e

Treat as compromised:

• any host or CI runner that installed litellm==1.107.0, even briefly;
• any container started from a v1.107.0 image, or from latest, main-latest or main-stable pulled between 11:46 and 14:20 UTC on 2026-10-21;
• any machine where a pip install litellm without a pin ran on 2026-10-21 between 11:39 UTC and the PyPI removal.

Network indicators: requests to litellm.dev or www.litellm.dev. A request to the path /codepy means the second stage actually ran.

Marker file hashes (Linux and macOS, SHA-256), for matching:

cook.pth          7f191632e7320705d154b85204f1ae546b543acbd7e8861229b76f5042b92bdb
cook.py           9dd0a3926f96a37eb86cf1ee2310d9be93139cc4ebef3ef9e2efe9531390be25
sitecustomize.py  34bef04fe88ac9d0aae0475630b1738681d15b94f233ac8aa46eacf5968305fa

What to do

1. Isolate. Kill any process holding /tmp/remote_code_loader.lock. Block litellm.dev at DNS or egress.
2. Rebuild, don’t clean. Rebuild containers from a clean image. For hosts, reinstall the environment rather than deleting three files and moving on.
3. Pin. litellm==1.106.2, or the v1.106.2 image by digest. Purge cached 1.107.0 wheels and images from your mirrors and registries.
4. Rotate everything the host could reach. Assume the secrets were taken: the stealer output is encrypted, so you cannot confirm from the network side that nothing left.
   • the instance or task role of the host, cloud keys, SSH and GPG keys, registry and CI tokens, Kubernetes and Vault tokens, and any secret in the environment of any process on the host;
   • tokens of AI coding tools on developer machines;
   • on a proxy with a database: every provider API key stored in it and in the model configs, the master key, the salt key, the virtual keys and the DATABASE_URL password.
5. Look back. Search network logs for litellm.dev and /codepy. On AWS, look in CloudTrail for GetCallerIdentity from the instance role around the install time, then for that role’s credentials used from outside your infrastructure. Audit provider-side usage for keys that were on affected hosts.

Lessons that apply to any gateway

An LLM gateway holds the keys to all your model vendors, so it is a high-value target. A few habits reduce the blast radius, whichever gateway you run:

• Pin versions and digests. Never deploy latest. Update on purpose, after a delay.
• Install from a lockfile with hashes (pip install --require-hashes, or your language’s equivalent).
• Restrict egress from the gateway. A proxy that only needs to reach model vendors has no reason to reach arbitrary hosts.
• Keep vendor keys out of plaintext in env vars and databases where you can.
• Verify signatures and provenance of what you pull, when the project publishes them.

This is the second incident of this kind for LiteLLM this year; in March 2026 versions 1.82.7 and 1.82.8 were compromised. No ecosystem is immune: dependencies can be poisoned in any language.

Sources

• BerriAI: incident report for the v1.107.0 release
• StepSecurity: litellm 1.107.0 compromised on PyPI
• Wiz: LiteLLM compromised, steals credentials
• Socket: LiteLLM PyPI package and Docker images compromised
• BerriAI/litellm issue #47826

In development, it just works. But as soon as your prototype moves to production—serving thousands of concurrent users across multiple microservices—a familiar set of systemic problems begins to surface.

In this inaugural post of our dev blog, we examine the five most common challenges engineering teams encounter when integrating LLM APIs, the abstract architectural patterns used to solve them, and how our open-source tools—KTMS (formerly llm-proxy) and light-tokenmeter—implement those solutions.

1. Vendor Lock-In and Tight SDK Coupling

The Problem

Hardcoding vendor SDKs (OpenAI, Anthropic, Google Gemini, DeepSeek) directly into your application services tightly couples your code to proprietary request formats, error models, and parameter conventions. When a new model comes out or pricing changes, switching or testing alternatives requires code refactoring, testing cycles, and redeployments.

The Abstract Solution: A Unified Gateway Layer

Instead of having microservices communicate directly with external LLM APIs, place a gateway between your applications and the model providers. The gateway exposes a single protocol, normalizing requests and responses into a standard schema. Applications express what they need (e.g. reasoning quality or speed), and the gateway decides which model satisfies it.

How We Solve It: KTMS Dynamic Routing

• Unified Interface: Applications speak to KTMS via a high-performance gRPC endpoint (GatewayService) or standard OpenAI-compatible HTTP endpoints (/v1/chat/completions).
• Effort-Based Routing: Clients don’t need to hardcode model names. They can request an effort level (LOW, MEDIUM, or HIGH) along with custom routing attributes. An ordered rule table evaluates the request and maps it to the optimal model.
• Zero-Downtime Rule Updates: Rules can be altered at runtime over an admin RPC or web console without redeploying any application service.

2. Vendor Outages and Rate Limit Spikes (429 & 503 Errors)

The Problem

External AI APIs suffer from periodic degradations, sudden capacity shortages, and strict rate limits (Requests Per Minute and Tokens Per Minute). If your application relies on a single provider, an upstream HTTP 503 or 429 immediately cascades into a customer-facing outage.

The Abstract Solution: Automatic Fallback Chains & Local Throttling

1. Fallback Chains: When the primary model fails or returns a transient error, the gateway instantly attempts an equivalent fallback model from another provider without failing the client request.
2. Proactive Rate Limiting: Enforce token-bucket or sliding-window rate limits locally before traffic ever leaves your infrastructure, preventing provider bans and unexpected throttles.

How We Solve It: KTMS Fallback & Redis Quotas

• Seamless Fallbacks: If OpenAI returns a 5xx or rate limit error, KTMS immediately reroutes the request to Anthropic Claude or Gemini in the background according to your configured model chain.
• Local Throttling: Built-in Redis-backed limits enforce requests-per-minute and token budgets per API key and per end user before calling the vendor.

3. Secret Sprawl and Insecure Credential Management

The Problem

Distributing vendor API keys across dozens of developer machines, CI environments, and Kubernetes deployment manifests creates major security risks. If a key is compromised, revoking it breaks everything, and auditing which service made which call is nearly impossible.

The Abstract Solution: Centralized Credential Vaulting

Vendor credentials should live in a single encrypted vault accessible only to the proxy. Internal services receive scoped, internally minted tokens that authenticate only against the proxy.

How We Solve It: KTMS Encrypted Vendor Storage

• Encrypted at Rest: Vendor API keys are stored in PostgreSQL encrypted with AES-256-GCM. Client applications never see or touch the raw vendor credentials.
• Scoped Internal Keys: Admins mint client keys with specific permissions (such as generate or admin). If a client key is compromised, it can be revoked instantly without touching upstream vendor accounts.

4. “Token Shock” and Opaque Spend Attribution

The Problem

A runaway agent loop or an unoptimized prompt can burn through a budget before anyone notices. Upstream provider dashboards show aggregated spend hours after it occurred and provide zero insight into your internal business context: Which tenant spent this? Which feature caused the spike? Did that call succeed?

The Abstract Solution: Pre-Flight Guardrails + Asynchronous Telemetry

• Pre-flight verification: Check user or project quotas before dispatching expensive generations.
• Fine-grained telemetry sink: Fire an immutable event for every single model invocation, recording tokens, cost, latency, project ID, user ID, and action identifier for instant OLAP analysis.

How We Solve It: KTMS + light-tokenmeter

• Pre-call Token Budgets: KTMS checks daily token allocations per end user or client key before dispatching calls.
• Deep Usage Analytics with light-tokenmeter: A dedicated, lightweight Go service that ingests usage events asynchronously via gRPC or HTTP. It supports PostgreSQL and ClickHouse, inserting events via fast batch COPY.
• Actionable Insights: Query spend across dimensions to immediately discover who is greedy and who is worthy:

# Example: Summarize token spend by project and model
curl -s "http://tokenmeter:8080/v1/usage/summary?from=2026-10-01T00:00:00Z&to=2026-10-18T00:00:00Z&group_by_project_id=true&group_by_model=true"

5. Overpaying for Non-Realtime Background Workloads

The Problem

Sending batch tasks (such as offline document classification, embeddings, synthetic data generation, or nightly evaluations) through synchronous interactive endpoints is expensive and error-prone. Most major providers offer a 50% discount for asynchronous batch jobs, but managing file uploads, batch IDs, polling loops, and partial errors across multiple vendors is tedious.

The Abstract Solution: Unified Batch Abstraction

A single asynchronous batch interface that transparently uploads requests, tracks vendor batch jobs, polls for completion, and downloads results.

How We Solve It: KTMS Batch API

• Submit bulk jobs with SubmitBatch and retrieve status and itemized outputs with GetBatch and ListBatchResults.
• Works consistently across providers while securing the 50% batch discount automatically.

Summary Matrix

What’s Next?

We’ll write about architecture decisions, feature releases and production deep dives. Upcoming posts will include:

• Setting up multi-vendor fallback chains.
• Architecting ClickHouse-backed token telemetry with light-tokenmeter.
• Structured JSON generation guarantees across diverse providers.

Explore the projects and star them on GitHub:

• github.com/ai-process/ktms
• github.com/ai-process/light-tokenmeter
• github.com/ai-process/ktms-admin